How to use robots.txt
- 1. Fill allow/disallow. One path per line.
- 2. Add a sitemap URL. Absolute.
- 3. List page URLs for the sitemap. No live fetch.
- 4. security.txt contact. mailto: is conventional.
About this tool
robots.txt is advice to crawlers, not a security boundary. sitemap.xml is a hint. security.txt is how researchers find you. Generate the stubs locally and commit them.
Disallow is not auth
Anything still fetchable by URL is fetchable. Use auth for secrets.
security.txt
Place it at /.well-known/security.txt as well as /security.txt if you can.
Code examples
robots
User-agent: *
Disallow: /admin