Robots.txt Generator

The Crawl Budget Problem That robots.txt Actually Solves

Most guides explain robots.txt as a tool for "blocking pages from Google." That framing misses the real use case for established sites. The core problem robots.txt solves is crawl budget allocation.

Google's crawlers spend a finite amount of time and resources on each site — the crawl budget. For small sites (under a few thousand pages), crawl budget is rarely a concern. For large sites — e-commerce stores with faceted navigation, news sites with infinite scroll archives, SaaS platforms with parametric search URLs — crawl budget is a critical operational constraint. Googlebot visiting 100,000 faceted search result URLs (like /products?color=red&size=M&sort=price-asc) wastes crawl budget that could have been spent crawling actual product pages.

robots.txt is the primary mechanism for directing crawlers away from these low-value URL patterns and toward the pages you actually want indexed.

The Crawl-Block vs Index-Block Distinction: A Common and Costly Mistake

This is the most frequently misunderstood aspect of robots.txt, and getting it wrong can have serious SEO consequences. There are two completely separate operations:

• Blocking crawling (robots.txt Disallow) — prevents Googlebot from fetching and reading the page
• Blocking indexing (noindex meta tag or X-Robots-Tag header) — prevents Google from including the page in its index

The critical edge case: if a URL is disallowed in robots.txt but is linked to from many external sites, Google can still index that URL as a "URL-only" entry in the index — it knows the page exists and may rank it, but it has never read the content. This means Google might rank an empty snippet for your /admin/login page, your staging environment, or your internal search results.

Furthermore, if you use Disallow to block a page that you want to noindex, the noindex tag never gets delivered to Googlebot (because the crawler cannot access the page to read the tag). The result is paradoxical: blocking a page with robots.txt while trying to noindex it with a meta tag produces a page that Google cannot noindex, because it cannot see the noindex instruction. The correct approach for pages you want removed from the index: allow crawling, use noindex.

Syntax Reference for Common Use Cases

Block all crawlers from a directory:

User-agent: *
Disallow: /admin/

Block a specific crawler (e.g., AI training bots):

User-agent: GPTBot
Disallow: /

Block crawlers from parameterized URLs (faceted navigation):

User-agent: *
Disallow: /*?*sort=
Disallow: /*?*filter=
Disallow: /*?*page=

Point Googlebot to your sitemap (best practice for all sites):

Sitemap: https://yourdomain.com/sitemap.xml

What robots.txt Cannot Do

• It cannot protect sensitive content — authenticated routes require server-side access controls
• It cannot prevent all bots from accessing blocked pages — only bots that respect the REP standard (Googlebot, Bingbot, and most legitimate crawlers do; scrapers and malicious bots typically do not)
• It cannot remove a URL from Google's index — use the noindex tag or the URL Removal Tool in Google Search Console for that

How to Use This Tool

1. Select which crawlers you want to configure rules for.
2. Add Disallow paths for directories and URL patterns you want to exclude.
3. Enter your sitemap URL to include the Sitemap directive.
4. Download the generated robots.txt and upload it to your site's root directory.