How to use the robots.txt generator
- Start with Allow everything, Block everything or Block two folders, or edit the first group directly.
- In each group, enter the user agents it applies to (use * for all crawlers), the paths to disallow, and any paths to allow. Add a crawl delay only if you need one.
- Press Add a group to give one crawler its own rules, and list your sitemap addresses at the bottom.
- Check the result, then Copy robots.txt or Download robots.txt and upload it to the top folder of your site so it loads at /robots.txt.
How a robots.txt file is read
A robots.txt file is a plain text list of rules that well behaved crawlers read before they visit your site. Rules come in groups. A group starts with one or more User-agent lines naming the crawlers it covers, followed by Disallow lines for paths they should not fetch and Allow lines for paths that are fine inside a disallowed folder. An empty Disallow means nothing is blocked. The format is defined in RFC 9309, which also supports * as a wildcard and $ for the end of a path.
A crawler follows the group that best matches its name, and a group with the name * applies to all crawlers that have no group of their own. Paths must start with a slash. Sitemap lines can appear anywhere and take a full address.
What it can and cannot do
The robots exclusion protocol is a request, not a lock. RFC 9309 itself says these rules are not a form of access authorization. Crawlers that ignore the rules can still visit, anyone can read the file, and a blocked address can still appear in search results if other sites link to it, because the crawler never read the page to see a noindex instruction. To keep a page private, put it behind a login. To keep a page out of search results, let crawlers read it and add a noindex robots tag, which you can generate with the meta tag generator.
Block folders with care. Blocking your whole site with Disallow: / is correct for a site that is not ready to be found and a serious mistake on a live one. Blocking style or script folders can also keep search engines from seeing the page as visitors do.
Crawl delay and testing
The crawl delay line is not part of RFC 9309, and many crawlers ignore it. It is offered because some still read it. After you upload the file, open yourdomain/robots.txt in a browser to see it, and use each search engine's own testing tools to check how it reads the rules. Everything here runs in your browser and nothing you enter is sent anywhere. Use the URL encoder if a path contains special characters.
Assumptions and limits
- The file follows RFC 9309. Individual crawlers may support more, fewer or different rules.
- You can add up to 8 groups. Paths must start with / or *, and sitemap addresses must be full http or https addresses.
- The tool checks the form of each line. It cannot tell whether a path exists on your site.
- Crawl-delay is included as an option but is not part of the standard.
Frequently asked questions
Where does the robots.txt file go?
It must be at the top level of your site so it loads at yourdomain/robots.txt. A file placed in a subfolder is not read by crawlers.
Does robots.txt keep a page out of Google?
Not reliably. It asks crawlers not to fetch the page, but the address can still be listed if other pages link to it. Use a noindex tag on a page that crawlers can read, or a password for private content.
What does Disallow with nothing after it mean?
An empty Disallow line means no path is blocked for that group, so the crawler may fetch everything. A single slash after Disallow blocks the whole site.
Should I list my sitemap in robots.txt?
It is a convenient place for crawlers to find it. Add each sitemap as a full address on its own Sitemap line.