robots.txt is the first file a crawler reads on your site. Most are copied from a template years ago and never looked at again.
That is a problem, because the mistakes are silent. Nothing errors, nothing appears in a build log, and you only find out when a page you care about is missing from search results.
This is the misconception that causes the most damage. Google is explicit that robots.txt "is not a mechanism for keeping a web page out of Google".
Disallow stops a crawler fetching a URL. It does not stop that URL being indexed. If another site links to it, Google can still list it, showing the URL and the anchor text with no description.
Worse, the two directives fight each other. If you Disallow a page and put noindex on it, the crawler never fetches the page, so it never sees the noindex. The block defeats the very thing meant to remove the page.
User-agent: *Disallow: /old-campaign/
The page carries a noindex tag, but Googlebot is not allowed to fetch it, so the tag is never read. The URL can stay in the index indefinitely.
❌ Figure: Bad example - Bad example - Blocking a page you are trying to deindex
Instead, leave the page crawlable and let the noindex do its job:
<meta name="robots" content="noindex">
The page stays fetchable, Googlebot reads the noindex and drops it from the index. Once it is gone you can add a Disallow as well, if you also want to save crawl budget.
✅ Figure: Good example - Good example - noindex removes a page, robots.txt only manages crawling
To keep something genuinely private, use authentication. robots.txt is a public file, so listing a secret path there advertises it to anyone who reads it.
https://www.ssw.com.au/robots.txthttps://www.example.com and https://shop.example.com need their own filesDisallow: /file.asp does not cover /FILE.aspPrecedence surprises people, and a rule that looks right can do nothing at all.
User-agent match. A named group replaces the * group rather than adding to it, so anything you still want enforced has to be repeated inside itAllow beats Disallow* matches any sequence and $ anchors the end of a pathUser-agent: *Disallow: /admin/Disallow: /private/Allow: /private/public-report.pdfUser-agent: GooglebotDisallow: /admin/Disallow: /private/Allow: /private/public-report.pdf
/private/public-report.pdf is allowed because it is a longer match than /private/. The Googlebot group repeats the rules, because it would otherwise ignore the * group entirely.
✅ Figure: Good example - Good example - Specific rules repeated in the named group
robots.txt is where crawlers look for your sitemap. The URL must be fully qualified, as Google does not guess between http and https or www and non-www.
Sitemap: https://www.ssw.com.au/sitemap.xml
See Do you have a crawler-friendly sitemap.xml? for what belongs in it.
Disallow: /. Deploy that to production and you remove the entire site from search. Check this after any migrationrobots.txt, so it is a map of what you wanted hiddenrobots.txt is now also how you decide whether AI answer engines can reach your content, and blocking the wrong one quietly removes you from ChatGPT, Claude or Perplexity answers. That is covered in Do you let AI answer engines crawl your site?.
Test your file with Google Search Console's robots.txt report rather than assuming it does what you intended. The full specification is RFC 9309, and Google documents its own behaviour in Introduction to robots.txt.