René's URL Explorer Experiment


Title: Issue 35457: robotparser reads empty robots.txt file as "all denied" - Python tracker

direct link

Domain: bugs.python.org

Nonetext/html; charset=utf-8

Links:

https://bugs.python.org/
Python Homehttps://www.python.org/
Abouthttps://www.python.org/about/
Newshttps://www.python.org/blogs/
Documentationhttps://www.python.org/doc/
Downloadshttps://www.python.org/downloads/
Communityhttps://www.python.org/community/
Foundationhttps://www.python.org/psf/
Developer's Guidehttps://devguide.python.org/
Issue Trackerhttps://bugs.python.org/
Searchhttps://bugs.python.org/issue?@template=search&status=1
Random Issuehttps://bugs.python.org/issue?@action=random
Issues with patchhttps://bugs.python.org/issue?status=1&@sort=-activity&@columns=id%2Cgithub%2Cactivity%2Ctitle%2Ccreator%2Cstatus&@dispname=Issues%20with%20patch&@startwith=0&@group=priority&keywords=2&@action=search&@filter=&@pagesize=50
Easy issueshttps://bugs.python.org/issue?status=1&@sort=-activity&@columns=id%2Cgithub%2Cactivity%2Ctitle%2Ccreator%2Cstatus&@dispname=Easy%20issues&@startwith=0&@group=priority&keywords=6&@action=search&@filter=&@pagesize=50
Statshttps://bugs.python.org/issue?@template=stats
Lost your login?https://bugs.python.org/user?@template=forgotten
User Listhttps://bugs.python.org/user?@sort=username
Committer Listhttps://bugs.python.org/user?iscommitter=1&@action=search&@sort=username&@pagesize=300
Tracker Documentationhttp://docs.python.org/devguide/triaging.html
Tracker Developmenthttp://wiki.python.org/moin/TrackerDevelopment
Report Tracker Problemhttps://github.com/python/psf-infra-meta/issues
https://github.com/python/cpython/issues
GitHubhttps://github.com/python/cpython/issues
see the GitHub FAQs in the Python's Developer Guide.https://devguide.python.org/gh-faq/
This issue has been migrated to GitHub: https://github.com/python/cpython/issues/79638 https://github.com/python/cpython/issues/79638
Titlehttp://docs.python.org/devguide/triaging.html#title
Typehttp://docs.python.org/devguide/triaging.html#type
Stagehttp://docs.python.org/devguide/triaging.html#stage
Componentshttp://docs.python.org/devguide/triaging.html#components
Versionshttp://docs.python.org/devguide/triaging.html#versions
Statushttp://docs.python.org/devguide/triaging.html#status
Resolutionhttp://docs.python.org/devguide/triaging.html#resolution
Dependencieshttp://docs.python.org/devguide/triaging.html#dependencies
Supersederhttp://docs.python.org/devguide/triaging.html#superseder
Assigned Tohttp://docs.python.org/devguide/triaging.html#assigned-to
Nosy Listhttp://docs.python.org/devguide/triaging.html#nosy-list
Priorityhttp://docs.python.org/devguide/triaging.html#priority
Keywordshttp://docs.python.org/devguide/triaging.html#keywords
msg331595https://bugs.python.org/issue35457#msg331595
(view)https://bugs.python.org/msg331595
http://www.robotstxt.org/robotstxt.htmlhttp://www.robotstxt.org/robotstxt.html
http://10.223.68.186/robots.txthttp://10.223.68.186/robots.txt
http://10.223.68.186/robots.txthttp://10.223.68.186/robots.txt
msg331870https://bugs.python.org/issue35457#msg331870
(view)https://bugs.python.org/msg331870
https://docs.python.org/2.7/library/robotparser.html#module-robotparserhttps://docs.python.org/2.7/library/robotparser.html#module-robotparser
https://docs.python.org/3/library/urllib.robotparser.html#module-urllib.robotparserhttps://docs.python.org/3/library/urllib.robotparser.html#module-urllib.robotparser
http://www.robotstxt.org/orig.htmlhttp://www.robotstxt.org/orig.html
msg331963https://bugs.python.org/issue35457#msg331963
(view)https://bugs.python.org/msg331963
http://www.robotstxt.org/orig.htmlhttp://www.robotstxt.org/orig.html
msg359180https://bugs.python.org/issue35457#msg359180
(view)https://bugs.python.org/msg359180
https://tools.ietf.org/html/draft-koster-rep-00#section-2.2.1https://tools.ietf.org/html/draft-koster-rep-00#section-2.2.1
msg359185https://bugs.python.org/issue35457#msg359185
(view)https://bugs.python.org/msg359185
1afc1696167547a5fa101c53e5a3ab4717f8852chttps://hg.python.org/lookup/1afc1696167547a5fa101c53e5a3ab4717f8852c
122541beceeccce4ef8a9bf739c727ccdcbf2f28https://hg.python.org/lookup/122541beceeccce4ef8a9bf739c727ccdcbf2f28
https://github.com/python/cpython/blob/f82e59ac4020a64c262a925230a8eb190b652e87/Lib/robotparser.py#L66-L67https://github.com/python/cpython/blob/f82e59ac4020a64c262a925230a8eb190b652e87/Lib/robotparser.py#L66-L67
https://github.com/python/cpython/blob/149175c6dfc8455023e4335575f3fe3d606729f9/Lib/urllib/robotparser.py#L69-L70https://github.com/python/cpython/blob/149175c6dfc8455023e4335575f3fe3d606729f9/Lib/urllib/robotparser.py#L69-L70
msg359202https://bugs.python.org/issue35457#msg359202
(view)https://bugs.python.org/msg359202
https://www.robotstxt.org/norobots-rfc.txthttps://www.robotstxt.org/norobots-rfc.txt
https://tools.ietf.org/html/draft-koster-rep-00#section-2.2.1https://tools.ietf.org/html/draft-koster-rep-00#section-2.2.1
https://www.robotstxt.org/robotstxt.htmlhttps://www.robotstxt.org/robotstxt.html
msg359202https://bugs.python.org/msg359202
berker.peksaghttps://bugs.python.org/user14674
xtreakhttps://bugs.python.org/user28746
msg359185https://bugs.python.org/msg359185
gallicroosterhttps://bugs.python.org/user33358
msg359180https://bugs.python.org/msg359180
msg331963https://bugs.python.org/msg331963
terry.reedyhttps://bugs.python.org/user155
msg331870https://bugs.python.org/msg331870
The Python Software Foundationhttps://python.org/psf-landing/
Rounduphttp://roundup.sourceforge.net
Python Software Foundationhttp://python.org/psf
Legal Statementshttp://python.org/about/legal

URLs of crawlers that visited me.