How to use the robots.txt url tester
- Paste the whole robots.txt file in the first box.
- Enter the crawler name, such as Googlebot. If the file has no group for that name, the group for * is used.
- List the URLs or paths to test, one per line. Full http or https URLs and paths that start with / both work.
- Read the result and the winning rule with its line number for each URL.
Formula
group: the one naming the crawler (not case sensitive), otherwise the * group
a rule matches when its pattern matches the start of the path plus query; * matches any run and a final $ ends the path
the longest matching pattern wins; Allow wins a tie; no match means allowed
Only the path and query are tested, never the host or the fragment. A Disallow with no value blocks nothing.
A small file
The sample file disallows /private/ and any path ending in .pdf for all crawlers, but allows /private/public-info.html. For Googlebot, which has no group of its own in the file, the * group applies. /private/report.html is blocked by Disallow: /private/, /private/public-info.html is allowed because Allow: /private/public-info.html is longer, and /files/guide.pdf is blocked by Disallow: /*.pdf$.
/files/guide.pdf?download=1 is allowed, since the $ means the path must end in .pdf and this one continues with a query. Googlebot-Image has its own group, so the rule Disallow: /photos/ does not apply to Googlebot.
How rules are chosen
When several rules match, the one with the longest pattern wins, so a specific Allow can open one page inside a blocked folder. If an Allow and a Disallow match with the same length, Allow wins. Percent-encoded characters match whether written in upper or lower case.
The crawler name is matched against the user-agent lines without case. A crawler uses only the most specific group for its name. Groups with the same name are combined.
What robots.txt does not do
It asks well behaved crawlers to stay away. It does not make a page private, because anyone can still open a URL, and a blocked page can still appear in listings if other pages link to it. Use real access controls for anything sensitive.
This page follows RFC 9309 only. A particular crawler may add its own rules, such as limits on file size or extra fields, so confirm with its documentation. To write a new file, use the robots.txt Generator.
Assumptions and limits
- The standard matching rules of RFC 9309 are used. Crawler specific extensions are not.
- The crawler name is the product token only, without version or spaces.
- Only the path and the query are tested.
- Files up to 500 KiB and up to 200 URLs are read.
- Nothing you type is stored or sent anywhere, and no URL is fetched.
Frequently asked questions
Why is a longer Allow able to override a Disallow?
The longest matching pattern is the most specific one, so the rule that names a page more exactly beats the one that names its whole folder.
What does the dollar sign mean?
A dollar sign at the end of a pattern means the path must end there. Disallow: /*.pdf$ blocks paths ending in .pdf but not /file.pdf?download=1.
Which group applies to my crawler?
The group that names it. If none does, the * group applies. A crawler with its own group ignores the * group completely, so copy any rules you still want.
Why does the page say no group applies?
The file has no group for the crawler and no * group, so nothing restricts it and everything is allowed.