Tidy Tip: Robots.txt

9^th March, 2025

Getting started with robots.txt

The robots.txt file is a simple but powerful tool that helps control how search engines crawl and index your website. By properly configuring it, you can guide search engine bots to prioritise the right pages while preventing them from accessing sensitive or duplicate content.

Why is robots.txt important?

Search engines use bots (also known as crawlers) to scan websites and index their content. Without guidance, these bots may crawl unnecessary pages, wasting crawl budget and potentially exposing content that shouldn’t be indexed. A well-structured robots.txt file can:

• Prevent search engines from crawling admin or backend areas
• Stop duplicate pages from being indexed
• Manage server load by restricting bot activity
• Direct bots towards priority pages for better indexing

How to create a robots.txt file

Setting up a robots.txt file is straightforward. Simply create a plain text file named “robots.txt” and upload it to the root directory of your website. The basic structure of a robots.txt file consists of directives that tell search engines what they can and cannot access.

Robot.txt example:

User-agent: *  
Disallow: /private/  
Allow: /public/

In this case, all search engine bots (`User-agent: *`) are blocked from accessing the `/private/` directory but allowed to crawl `/public/`.

Best practices for using robots.txt

1. Be specific – Avoid blocking important pages that should be indexed
2. Use ‘Disallow’ carefully – Blocking a page in robots.txt does not remove it from search results if it’s already indexed
3. Allow essential resources – Allow important resources such as PDFs, images, and other media that Google may find valuable and display in search results. Ensuring these assets are accessible can improve visibility and enhance user engagement
4. Regularly audit your file – As your site evolves, review your robots.txt to keep it relevant

Testing your robots.txt

Google Search Console offers a robots.txt tester that allows you to check if your rules are correctly implemented. This helps ensure that search engine bots are crawling your site as intended.

If you’re unsure about setting up or optimising your robots.txt file, Tidy Design can help! A poorly configured file could block important pages from search engines or leave sensitive content exposed. Our team can audit your robots.txt setup, check for errors, and ensure your site is structured correctly for maximum visibility. Get in touch today to optimise your website, we’d be happy to help!

Until next time, keep it Tidy!

Mike