René's URL Explorer Experiment

Title: RFC 9309: Robots Exclusion Protocol

Description: This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Specifically, it adds definition language for the protocol, instructions for handling errors, and instructions for caching.

Mail addresses
m.koster@greenhills.co.uk
garyillyes@google.com
henner@google.com
lizzi@google.com

Generator: xml2rfc 3.14.2

direct link

Domain: www.rfc-editor.org

Hey, it has address:

Martijn Koster

Stalworthy Manor Farm

Suton Lane

Wymondham, Norfolk

NR18 9JG

United Kingdom

Email: m.koster@greenhills.co.uk

scripts	Common,Latin
author	Lizzi Sassman
keyword	robots.txt
rfc.number	9309

Links:

9309	https://www.rfc-editor.org/rfc/rfc9309
Abstract	https://www.rfc-editor.org/rfc/rfc9309.html#abstract
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-abstract-1
Status of This Memo	https://www.rfc-editor.org/rfc/rfc9309.html#name-status-of-this-memo
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-boilerplate.1-1
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-boilerplate.1-2
https://www.rfc-editor.org/info/rfc9309	https://www.rfc-editor.org/info/rfc9309
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-boilerplate.1-3
Copyright Notice	https://www.rfc-editor.org/rfc/rfc9309.html#name-copyright-notice
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-boilerplate.2-1
https://trustee.ietf.org/license-info	https://trustee.ietf.org/license-info
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-boilerplate.2-2
▲	https://www.rfc-editor.org/rfc/rfc9309.html
Table of Contents	https://www.rfc-editor.org/rfc/rfc9309.html#name-table-of-contents
1	https://www.rfc-editor.org/rfc/rfc9309.html#section-1
Introduction	https://www.rfc-editor.org/rfc/rfc9309.html#name-introduction
1.1	https://www.rfc-editor.org/rfc/rfc9309.html#section-1.1
Requirements Language	https://www.rfc-editor.org/rfc/rfc9309.html#name-requirements-language
2	https://www.rfc-editor.org/rfc/rfc9309.html#section-2
Specification	https://www.rfc-editor.org/rfc/rfc9309.html#name-specification
2.1	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.1
Protocol Definition	https://www.rfc-editor.org/rfc/rfc9309.html#name-protocol-definition
2.2	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2
Formal Syntax	https://www.rfc-editor.org/rfc/rfc9309.html#name-formal-syntax
2.2.1	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.1
The User-Agent Line	https://www.rfc-editor.org/rfc/rfc9309.html#name-the-user-agent-line
2.2.2	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.2
The "Allow" and "Disallow" Lines	https://www.rfc-editor.org/rfc/rfc9309.html#name-the-allow-and-disallow-line
2.2.3	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.3
Special Characters	https://www.rfc-editor.org/rfc/rfc9309.html#name-special-characters
2.2.4	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.4
Other Records	https://www.rfc-editor.org/rfc/rfc9309.html#name-other-records
2.3	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3
Access Method	https://www.rfc-editor.org/rfc/rfc9309.html#name-access-method
2.3.1	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3.1
Access Results	https://www.rfc-editor.org/rfc/rfc9309.html#name-access-results
2.3.1.1	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3.1.1
Successful Access	https://www.rfc-editor.org/rfc/rfc9309.html#name-successful-access
2.3.1.2	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3.1.2
Redirects	https://www.rfc-editor.org/rfc/rfc9309.html#name-redirects
2.3.1.3	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3.1.3
"Unavailable" Status	https://www.rfc-editor.org/rfc/rfc9309.html#name-unavailable-status
2.3.1.4	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3.1.4
"Unreachable" Status	https://www.rfc-editor.org/rfc/rfc9309.html#name-unreachable-status
2.3.1.5	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3.1.5
Parsing Errors	https://www.rfc-editor.org/rfc/rfc9309.html#name-parsing-errors
2.4	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.4
Caching	https://www.rfc-editor.org/rfc/rfc9309.html#name-caching
2.5	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.5
Limits	https://www.rfc-editor.org/rfc/rfc9309.html#name-limits
3	https://www.rfc-editor.org/rfc/rfc9309.html#section-3
Security Considerations	https://www.rfc-editor.org/rfc/rfc9309.html#name-security-considerations
4	https://www.rfc-editor.org/rfc/rfc9309.html#section-4
IANA Considerations	https://www.rfc-editor.org/rfc/rfc9309.html#name-iana-considerations
5	https://www.rfc-editor.org/rfc/rfc9309.html#section-5
Examples	https://www.rfc-editor.org/rfc/rfc9309.html#name-examples
5.1	https://www.rfc-editor.org/rfc/rfc9309.html#section-5.1
Simple Example	https://www.rfc-editor.org/rfc/rfc9309.html#name-simple-example
5.2	https://www.rfc-editor.org/rfc/rfc9309.html#section-5.2
Longest Match	https://www.rfc-editor.org/rfc/rfc9309.html#name-longest-match
6	https://www.rfc-editor.org/rfc/rfc9309.html#section-6
References	https://www.rfc-editor.org/rfc/rfc9309.html#name-references
6.1	https://www.rfc-editor.org/rfc/rfc9309.html#section-6.1
Normative References	https://www.rfc-editor.org/rfc/rfc9309.html#name-normative-references
6.2	https://www.rfc-editor.org/rfc/rfc9309.html#section-6.2
Informative References	https://www.rfc-editor.org/rfc/rfc9309.html#name-informative-references
	https://www.rfc-editor.org/rfc/rfc9309.html#appendix-A
Authors' Addresses	https://www.rfc-editor.org/rfc/rfc9309.html#name-authors-addresses
1.	https://www.rfc-editor.org/rfc/rfc9309.html#section-1
Introduction	https://www.rfc-editor.org/rfc/rfc9309.html#name-introduction
RFC3986	https://www.rfc-editor.org/rfc/rfc9309.html#RFC3986
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-1-1
RFC8288	https://www.rfc-editor.org/rfc/rfc9309.html#RFC8288
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-1-2
ROBOTSTXT	https://www.rfc-editor.org/rfc/rfc9309.html#ROBOTSTXT
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-1-3
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-1-4
1.1.	https://www.rfc-editor.org/rfc/rfc9309.html#section-1.1
Requirements Language	https://www.rfc-editor.org/rfc/rfc9309.html#name-requirements-language
RFC2119	https://www.rfc-editor.org/rfc/rfc9309.html#RFC2119
RFC8174	https://www.rfc-editor.org/rfc/rfc9309.html#RFC8174
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-1.1-1
2.	https://www.rfc-editor.org/rfc/rfc9309.html#section-2
Specification	https://www.rfc-editor.org/rfc/rfc9309.html#name-specification
2.1.	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.1
Protocol Definition	https://www.rfc-editor.org/rfc/rfc9309.html#name-protocol-definition
Section 2.3	https://www.rfc-editor.org/rfc/rfc9309.html#access-method
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.1-1
Section 2.2.2	https://www.rfc-editor.org/rfc/rfc9309.html#the-allow-and-disallow-lines
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.1-2.2
Section 2.2.1	https://www.rfc-editor.org/rfc/rfc9309.html#the-user-agent-line
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.1-2.4
2.2.	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2
Formal Syntax	https://www.rfc-editor.org/rfc/rfc9309.html#name-formal-syntax
RFC5234	https://www.rfc-editor.org/rfc/rfc9309.html#RFC5234
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2-1
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2-2
2.2.1.	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.1
The User-Agent Line	https://www.rfc-editor.org/rfc/rfc9309.html#name-the-user-agent-line
RFC9110	https://www.rfc-editor.org/rfc/rfc9309.html#RFC9110
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.1-1
Figure 1	https://www.rfc-editor.org/rfc/rfc9309.html#figure-1
Example of a User-Agent HTTP header and robots.txt user-agent line for the ExampleBot product token	https://www.rfc-editor.org/rfc/rfc9309.html#name-example-of-a-user-agent-htt
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.1-3
Section 2.2.2	https://www.rfc-editor.org/rfc/rfc9309.html#the-allow-and-disallow-lines
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.1-4
Figure 2	https://www.rfc-editor.org/rfc/rfc9309.html#figure-2
Example of how to merge two robots.txt groups that match the same product token	https://www.rfc-editor.org/rfc/rfc9309.html#name-example-of-how-to-merge-two
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.1-6
Figure 3	https://www.rfc-editor.org/rfc/rfc9309.html#figure-3
Example of no matching groups other than the "*" for the ExampleBot product token	https://www.rfc-editor.org/rfc/rfc9309.html#name-example-of-no-matching-grou
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.1-8
2.2.2.	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.2
The "Allow" and "Disallow" Lines	https://www.rfc-editor.org/rfc/rfc9309.html#name-the-allow-and-disallow-line
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.2-1
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.2-2
RFC3986	https://www.rfc-editor.org/rfc/rfc9309.html#RFC3986
RFC3986	https://www.rfc-editor.org/rfc/rfc9309.html#RFC3986
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.2-3
RFC3986	https://www.rfc-editor.org/rfc/rfc9309.html#RFC3986
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.2-4
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.2-5
Figure 4	https://www.rfc-editor.org/rfc/rfc9309.html#figure-4
Examples of matching percent-encoded URI components	https://www.rfc-editor.org/rfc/rfc9309.html#name-examples-of-matching-percen
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.2-7
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.2-8
2.2.3.	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.3
Special Characters	https://www.rfc-editor.org/rfc/rfc9309.html#name-special-characters
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.3-1
Figure 5	https://www.rfc-editor.org/rfc/rfc9309.html#figure-5
List of special characters in robots.txt files	https://www.rfc-editor.org/rfc/rfc9309.html#name-list-of-special-characters-
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.3-3
Figure 6	https://www.rfc-editor.org/rfc/rfc9309.html#figure-6
Example of percent-encoding	https://www.rfc-editor.org/rfc/rfc9309.html#name-example-of-percent-encoding
2.2.4.	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.4
Other Records	https://www.rfc-editor.org/rfc/rfc9309.html#name-other-records
SITEMAPS	https://www.rfc-editor.org/rfc/rfc9309.html#SITEMAPS
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.4-1
Section 2	https://www.rfc-editor.org/rfc/rfc9309.html#specification
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.4-2
2.3.	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3
Access Method	https://www.rfc-editor.org/rfc/rfc9309.html#name-access-method
RFC3629	https://www.rfc-editor.org/rfc/rfc9309.html#RFC3629
RFC2046	https://www.rfc-editor.org/rfc/rfc9309.html#RFC2046
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3-1
RFC3986	https://www.rfc-editor.org/rfc/rfc9309.html#RFC3986
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3-2
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3-3
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3-4
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3-5
2.3.1.	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3.1
Access Results	https://www.rfc-editor.org/rfc/rfc9309.html#name-access-results
2.3.1.1.	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3.1.1
Successful Access	https://www.rfc-editor.org/rfc/rfc9309.html#name-successful-access
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3.1.1-1
2.3.1.2.	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3.1.2
Redirects	https://www.rfc-editor.org/rfc/rfc9309.html#name-redirects
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3.1.2-1
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3.1.2-2
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3.1.2-3
2.3.1.3.	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3.1.3
"Unavailable" Status	https://www.rfc-editor.org/rfc/rfc9309.html#name-unavailable-status
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3.1.3-1
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3.1.3-2
2.3.1.4.	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3.1.4
"Unreachable" Status	https://www.rfc-editor.org/rfc/rfc9309.html#name-unreachable-status
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3.1.4-1
Section 2.3.1.3	https://www.rfc-editor.org/rfc/rfc9309.html#unavailable-status
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3.1.4-2
2.3.1.5.	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3.1.5
Parsing Errors	https://www.rfc-editor.org/rfc/rfc9309.html#name-parsing-errors
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3.1.5-1
2.4.	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.4
Caching	https://www.rfc-editor.org/rfc/rfc9309.html#name-caching
RFC9111	https://www.rfc-editor.org/rfc/rfc9309.html#RFC9111
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.4-1
2.5.	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.5
Limits	https://www.rfc-editor.org/rfc/rfc9309.html#name-limits
Section 3	https://www.rfc-editor.org/rfc/rfc9309.html#security
KiB	https://www.rfc-editor.org/rfc/rfc9309.html#KiB
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-2.5-1
3.	https://www.rfc-editor.org/rfc/rfc9309.html#section-3
Security Considerations	https://www.rfc-editor.org/rfc/rfc9309.html#name-security-considerations
RFC9110	https://www.rfc-editor.org/rfc/rfc9309.html#RFC9110
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-3-1
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-3-2
Section 2.5	https://www.rfc-editor.org/rfc/rfc9309.html#limits
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-3-3.2
Section 2.2	https://www.rfc-editor.org/rfc/rfc9309.html#formal-syntax
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-3-3.4
RFC9110	https://www.rfc-editor.org/rfc/rfc9309.html#RFC9110
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-3-3.6
4.	https://www.rfc-editor.org/rfc/rfc9309.html#section-4
IANA Considerations	https://www.rfc-editor.org/rfc/rfc9309.html#name-iana-considerations
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-4-1
5.	https://www.rfc-editor.org/rfc/rfc9309.html#section-5
Examples	https://www.rfc-editor.org/rfc/rfc9309.html#name-examples
5.1.	https://www.rfc-editor.org/rfc/rfc9309.html#section-5.1
Simple Example	https://www.rfc-editor.org/rfc/rfc9309.html#name-simple-example
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-5.1-1
Section 2.2	https://www.rfc-editor.org/rfc/rfc9309.html#formal-syntax
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-5.1-2.2
Section 2.2	https://www.rfc-editor.org/rfc/rfc9309.html#formal-syntax
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-5.1-2.4
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-5.1-2.6
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-5.1-2.8
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-5.1-3
5.2.	https://www.rfc-editor.org/rfc/rfc9309.html#section-5.2
Longest Match	https://www.rfc-editor.org/rfc/rfc9309.html#name-longest-match
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-5.2-1
¶	https://www.rfc-editor.org/rfc/rfc9309.html#section-5.2-2
6.	https://www.rfc-editor.org/rfc/rfc9309.html#section-6
References	https://www.rfc-editor.org/rfc/rfc9309.html#name-references
6.1.	https://www.rfc-editor.org/rfc/rfc9309.html#section-6.1
Normative References	https://www.rfc-editor.org/rfc/rfc9309.html#name-normative-references
https://www.rfc-editor.org/info/rfc2046	https://www.rfc-editor.org/info/rfc2046
https://www.rfc-editor.org/info/rfc2119	https://www.rfc-editor.org/info/rfc2119
https://www.rfc-editor.org/info/rfc3629	https://www.rfc-editor.org/info/rfc3629
https://www.rfc-editor.org/info/rfc3986	https://www.rfc-editor.org/info/rfc3986
https://www.rfc-editor.org/info/rfc5234	https://www.rfc-editor.org/info/rfc5234
https://www.rfc-editor.org/info/rfc8174	https://www.rfc-editor.org/info/rfc8174
https://www.rfc-editor.org/info/rfc8288	https://www.rfc-editor.org/info/rfc8288
https://www.rfc-editor.org/info/rfc9110	https://www.rfc-editor.org/info/rfc9110
https://www.rfc-editor.org/info/rfc9111	https://www.rfc-editor.org/info/rfc9111
6.2.	https://www.rfc-editor.org/rfc/rfc9309.html#section-6.2
Informative References	https://www.rfc-editor.org/rfc/rfc9309.html#name-informative-references
https://simple.wikipedia.org/wiki/Kibibyte	https://simple.wikipedia.org/wiki/Kibibyte
https://www.robotstxt.org/	https://www.robotstxt.org/
https://www.sitemaps.org/index.html	https://www.sitemaps.org/index.html
Authors' Addresses	https://www.rfc-editor.org/rfc/rfc9309.html#name-authors-addresses

Viewport: initial-scale=1.0

URLs of crawlers that visited me.