Robots.txt
Also known as: robots exclusion standard, crawl control, robot exclusion file
Robots.txt is a text file placed at the root of a website (domain.com/robots.txt) that provides instructions to search engine crawlers about which parts of the site they are permitted to access.
It uses the Robots Exclusion Standard to specify rules for different crawlers (user-agents). Robots.txt does not prevent pages from being indexed if other sites link to them: it only controls crawling. It is commonly used to prevent crawlers from accessing admin areas, staging environments, search result pages, and other content that shouldn't appear in search engines.
Robots.txt, explained at three levels
At home
Robots.txt is a text file that tells Google and other search engine bots where they can and cannot go on your website. If you have an admin login page or a staging version of your site, you don't want Google to show those in search results. Robots.txt lets you say 'don't crawl these pages,' which usually keeps low-value content out of search results. It is not a lock, though: a blocked page can still be listed if other sites link to it.
At work
Every website has a robots.txt file (or should have one). It controls which pages search engines can access. Important uses: blocking admin and login pages, blocking duplicate content pages (like search result pages within your site), blocking staging or development environments, and controlling crawl budget on very large sites by directing crawlers to focus on important pages.
At home
Basic robots.txt syntax: 'User-agent: *' applies rules to all crawlers. 'Disallow: /admin/' blocks all crawlers from the /admin/ section. 'Allow: /' permits all crawlers to access everything. A sitemap declaration at the bottom of robots.txt helps crawlers find your sitemap for more efficient crawling. Robots.txt is publicly visible at domain.com/robots.txt: anyone can read it, so don't use it to 'hide' sensitive information by blocking it; that information may still be accessible.
At work
Crawl budget is the reason large sites care deeply about robots.txt: search engines have limited resources for crawling each site. Blocking low-value URLs (internal search results, faceted navigation duplicates, admin pages, session-parameter URLs) directs crawl budget toward high-value pages. A site with 100,000 low-value URLs and 10,000 important URLs may find that Google rarely crawls the important pages because it's spending crawl budget on the noise.
At home
A critical distinction: robots.txt blocks crawling, not indexing. If another site links to a URL that your robots.txt blocks, Google may still index that URL (it can infer content exists even without crawling the page). To prevent indexing of specific pages, use a meta robots noindex tag on the page itself: this requires the page to be crawlable so Google can read the tag. The combination of blocking in robots.txt and adding noindex is contradictory: Google can't read the noindex tag if it can't crawl the page.
At work
Robots.txt misconfiguration is one of the highest-impact SEO accidents: a single incorrect 'Disallow: /' in robots.txt blocks all crawlers from the entire site, causing all pages to fall out of Google's index within weeks. This mistake has happened to major websites during development migrations when a staging environment's robots.txt was accidentally deployed to production. Regular robots.txt auditing and monitoring: checking that production robots.txt hasn't changed unexpectedly: is a basic SEO hygiene practice for any site that depends on organic search traffic.
Real-world examples
At home
Lena launched a new version of her portfolio website. During development, the staging site had 'Disallow: /' in robots.txt to prevent Google from indexing work in progress. When she launched the live version, she forgot to update robots.txt. A month later, she noticed her site had disappeared from Google. The misconfigured robots.txt had blocked Google from crawling anything. She fixed it, and rankings recovered over several weeks.
At work
Marcus managed SEO for a large e-commerce site. Analysis showed Google was spending 80% of its crawl on faceted navigation URLs that generated no organic traffic. He added rules to robots.txt blocking URL patterns with multiple filter parameters, reducing crawled low-value URLs. Crawl frequency on product and category pages increased, and new products appeared in Google search much faster.
Common confusions
- Many people think adding a URL to robots.txt Disallow hides it from Google entirely — Robots.txt disallow only instructs crawlers not to visit the URL: it does not prevent the URL from appearing in search results. If another website links to a blocked URL, Google can infer the page exists and index it without crawling the content. To prevent a page from appearing in search results, use a meta robots noindex tag on the page itself, which requires the page to be crawlable.
- Some assume all web crawlers respect robots.txt — Robots.txt is a convention, not a technical enforcement. Reputable crawlers (Google, Bing, legitimate SEO tools) respect it as an industry standard. Malicious crawlers, scrapers, and bots frequently ignore robots.txt entirely: it provides no protection against determined bad actors. For actual access control, authentication (login requirements) or server-level IP blocking is required.
For analysts
At the start of any SEO engagement, check robots.txt first: it takes 30 seconds and can reveal catastrophic misconfigurations blocking crawlers from the entire site or important sections. Type domain.com/robots.txt in the browser. If you see 'Disallow: /' with no Allow rules, the site has been accidentally blocked from Google. This is the highest-impact single file in SEO and the most common source of sudden ranking drops after site migrations.
History & origin
The Robots Exclusion Protocol was created in 1994 by Martijn Koster during the early World Wide Web era, to address the problem of web crawlers consuming excessive server resources and indexing content that site owners didn't want indexed. The standard was published informally at first and later formalized. Robots.txt remains one of the oldest web protocols still in active use. Google, Bing, and most reputable crawlers respect robots.txt instructions as an industry convention (not a technical enforcement). In 2019, Google retired support for the 'noindex' directive in robots.txt, directing users to use meta robots tags instead. In 2022 the protocol became an official internet standard, RFC 9309.