Featured · client work
Lunarlink Solutions, Toronto
See it live
A robots.txt file is a plain text file at the root of a domain that tells automated crawlers which parts of the site they may request. It controls crawling, not privacy and not indexing: anything a robots.txt file blocks is still public to anyone who has the URL.
There is exactly one robots.txt per hostname, and it has to sit at the root: example.com/robots.txt, never example.com/pages/robots.txt. A subdomain is a separate hostname, so shop.example.com needs its own file and does not inherit the main one.
The contents are short. A User-agent line names which crawler the rules underneath apply to, Disallow and Allow lines name paths, and a Sitemap line points at the XML sitemap. Most small business sites need four or five lines in total.
A disallowed URL can still appear in search results. If another site links to it, a search engine can list the URL with no title and no description, because it knows the page exists but was never allowed to read it. Removing a page from results takes a noindex tag or a password, not a robots.txt rule.
The two mechanisms also conflict in a way that catches people out. A noindex tag only works if the crawler is allowed to fetch the page and read it. Blocking a page in robots.txt guarantees the noindex tag is never seen, so the page stays in the index indefinitely.
The line Disallow: / blocks the entire site. It belongs on a staging server and nowhere else, and it moves to production every time a site is copied across without anyone checking the file. Traffic does not fall immediately, which is what makes it expensive: pages drop out over several weeks, long after the deploy that caused it.
The second most common mistake is blocking the CSS and JavaScript folders. A crawler that cannot load the stylesheet sees an unstyled page, cannot confirm the mobile layout works, and scores the page accordingly.
Very little. Point at the sitemap, block the URLs that generate infinite variations such as internal search results and filtered listings, and leave everything else open. A brochure site with twenty pages has nothing worth blocking.
Crawlers are also free to ignore the file. Well-behaved ones honour it; scrapers and email harvesters do not, and never have. Anything that genuinely must stay private needs authentication.
| Line | What it does | What it does not do |
|---|---|---|
| User-agent | Names which crawler the rules below apply to | Confirm the crawler is who it claims to be |
| Disallow | Asks a crawler not to request a path | Remove a URL from search results |
| Allow | Carves an exception out of a broader Disallow | Override a noindex tag on the page itself |
| Sitemap | Points crawlers at the XML sitemap | Guarantee that the listed URLs get indexed |
Send the link. We read the site by hand and write back inside five business days.
No call, no commitment
