René's URL Explorer Experiment


Title: bpo-39280: Don't allow datetime parsing to accept non-Ascii digits by cool-RR · Pull Request #17931 · python/cpython · GitHub

Open Graph Title: bpo-39280: Don't allow datetime parsing to accept non-Ascii digits by cool-RR · Pull Request #17931 · python/cpython

X Title: bpo-39280: Don't allow datetime parsing to accept non-Ascii digits by cool-RR · Pull Request #17931 · python/cpython

Description: I've been doing some research into the use of \d in regular expressions in CPython, and any security vulnerabilities that might happen as a result of the fact that it accepts non-Ascii digits like ٢ and 5. In most places in the CPython codebase, the re.ASCII flag is used for such cases, thus ensuring the re module prohibits these non-Ascii digits. Personally, my preference is to never use \d and always use [0-9]. I think that it's rule that's more easy to enforce and less likely to result in a slipup, but that's a matter of personal taste. I found a few places where we don't use the re.ASCII flag and we do accept non-Ascii digits. The first and less interesting place is platform.py, where we define patterns used for detecting versions of PyPy and IronPython. I don't know how anyone would exploit that, but personally I'd change that to a [0-9] just to be safe. I've opened bpo-39279 for that. The more sensitive place is the datetime module. Happily, the datetime.datetime.fromisoformat function rejects non-Ascii digits. But the datetime.datetime.strptime function does not: from datetime import datetime time_format = '%Y-%m-%d' parse = lambda s: datetime.strptime(s, time_format) x = '٢019-12-22' y = '2019-12-22' assert x != y assert parse(x) == parse(y) print(parse(x)) # Output: 2019-12-22 00:00:00 If user code were to check for uniqueness of a datetime by comparing it as a string, this is where an attacker could fool this logic, by using a non-Ascii digit. Two more interesting points about this: If you'd try the same trick, but you'd insert ٢ in the day section instead of the year section, Python would reject that. So we definitely have inconsistent behavior. In the documentation for strptime, we're referencing the 1989 C standard. Since the first version of Unicode was published in 1991, it's reasonable not to expect the standard to support digits that were introduced in Unicode. If you'd scroll down in that documentation, you'll see that we also implement the less-known ISO 8601 standard, where %G-%V-%u represents a year, week number, and day of week. The %G is vulnerable: from datetime import datetime time_format = '%G-%V-%u' parse = lambda s: datetime.strptime(s, time_format) x = '٢019-53-4' y = '2019-53-4' assert x != y assert parse(x) == parse(y) print(parse(x)) # Output: 2020-01-02 00:00:00 I looked at the ISO 8601:2004 document, and under the "Fundamental principles" chapter, it says: This International Standard gives a set of rules for the representation of time points time intervals recurring time intervals. Both accurate and approximate representations can be identified by means of unique and unambiguous expressions specifying the relevant dates, times of day and durations. Note the "unique and unambiguous". By accepting non-Ascii digits, we're breaking the uniqueness requirement of ISO 8601. https://bugs.python.org/issue39280

Open Graph Description: I've been doing some research into the use of \d in regular expressions in CPython, and any security vulnerabilities that might happen as a result of the fact that it accepts non-Ascii digits l...

X Description: I've been doing some research into the use of \d in regular expressions in CPython, and any security vulnerabilities that might happen as a result of the fact that it accepts non-Ascii digi...

Opengraph URL: https://github.com/python/cpython/pull/17931

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:a041c76e-1d91-11a5-52d2-923a91690acd
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-id978A:23E48:CBD6F3:FF6D56:6B023C7D
html-safe-nonce1c542e75a713d1c799725392d4036a5155c5b5564eff786730f1ad1322d6b792
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5NzhBOjIzRTQ4OkNCRDZGMzpGRjZENTY6NkIwMjNDN0QiLCJ2aXNpdG9yX2lkIjoiMjUzMjA1MzQ0ODQ5OTAyNzE3IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0=
visitor-hmac3cb8f1f475f59cf2cadf9af887b168023508f6b70844eaed399b4c593965a5b0
hovercard-subject-tagpull_request:361181171
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/python/cpython/pull/17931/files
twitter:imagehttps://avatars.githubusercontent.com/u/56778?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/56778?s=400&v=4
og:image:altI've been doing some research into the use of \d in regular expressions in CPython, and any security vulnerabilities that might happen as a result of the fact that it accepts non-Ascii digits l...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/python/cpython git https://github.com/python/cpython.git
octolytics-dimension-user_id1525981
octolytics-dimension-user_loginpython
octolytics-dimension-repository_id81598961
octolytics-dimension-repository_nwopython/cpython
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id81598961
octolytics-dimension-repository_network_root_nwopython/cpython
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release423dbb54c5e7bfecc838eb387c1435eb3d0e760d
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/python/cpython/pull/17931/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fpull%2F17931%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fpull%2F17931%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=python%2Fcpython
Reloadhttps://github.com/python/cpython/pull/17931/files
Reloadhttps://github.com/python/cpython/pull/17931/files
Reloadhttps://github.com/python/cpython/pull/17931/files
pythonhttps://github.com/python
cpythonhttps://github.com/python/cpython
Codehttps://github.com/python/cpython
Issues5k+ (5k+)https://github.com/python/cpython/issues
Pull requests2.4k (2.4k)https://github.com/python/cpython/pulls
Actionshttps://github.com/python/cpython/actions
Projectshttps://github.com/python/cpython/projects
Security and qualityhttps://github.com/python/cpython/security
Insightshttps://github.com/python/cpython/insights
Sign up for GitHub https://github.com/signup?return_to=%2Fpython%2Fcpython%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fpython%2Fcpython%2Fissues%2Fnew%2Fchoose
cool-RRhttps://github.com/cool-RR
python:masterhttps://github.com/python/cpython/tree/master
cool-RR:2020-01-09-datetimehttps://github.com/cool-RR/cpython/tree/2020-01-09-datetime
Conversation 0 https://github.com/python/cpython/pull/17931
Commits 2 https://github.com/python/cpython/pull/17931/commits
Checks 0 https://github.com/python/cpython/pull/17931/checks
Files changed https://github.com/python/cpython/pull/17931/files
Please reload this pagehttps://github.com/python/cpython/pull/17931/files
bpo-39280: Don't allow datetime parsing to accept non-Ascii digits https://github.com/python/cpython/pull/17931/files#top
Show all changes 2 commits https://github.com/python/cpython/pull/17931/files
864c031 Fix outdated comment in _strptime.py cool-RR Jan 9, 2020 https://github.com/python/cpython/pull/17931/commits/864c031e4cd9b34feba121222aabafd16750ee6d
75bc88f bpo-39280: Don't allow datetime parsing to accept non-Ascii digits cool-RR Jan 9, 2020 https://github.com/python/cpython/pull/17931/commits/75bc88f2c23e199c07e7105b08b481031a1352f5
Clear filters https://github.com/python/cpython/pull/17931/files
Please reload this pagehttps://github.com/python/cpython/pull/17931/files
Please reload this pagehttps://github.com/python/cpython/pull/17931/files
_strptime.py https://github.com/python/cpython/pull/17931/files#diff-c214ba7b6692519554f750422c7c285579aa9e5399b71957a23d8f368be778b7
datetimetester.py https://github.com/python/cpython/pull/17931/files#diff-917d5d1d1423993f37a643afa9f2a2f9caa3c0f91e06c0d7ebea1b879d53b4ef
2020-01-10-00-16-46.bpo-39280.VA476D.rst https://github.com/python/cpython/pull/17931/files#diff-e3b47b5a4229aab0a35b3a2775e37ec33f54df49906a795e47e9e0325f949940
Lib/_strptime.pyhttps://github.com/python/cpython/pull/17931/files#diff-c214ba7b6692519554f750422c7c285579aa9e5399b71957a23d8f368be778b7
View file https://github.com/cool-RR/cpython/blob/75bc88f2c23e199c07e7105b08b481031a1352f5/Lib/_strptime.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/17931/{{ revealButtonHref }}
https://github.com/python/cpython/pull/17931/files#diff-c214ba7b6692519554f750422c7c285579aa9e5399b71957a23d8f368be778b7
https://github.com/python/cpython/pull/17931/files#diff-c214ba7b6692519554f750422c7c285579aa9e5399b71957a23d8f368be778b7
Lib/test/datetimetester.pyhttps://github.com/python/cpython/pull/17931/files#diff-917d5d1d1423993f37a643afa9f2a2f9caa3c0f91e06c0d7ebea1b879d53b4ef
View file https://github.com/cool-RR/cpython/blob/75bc88f2c23e199c07e7105b08b481031a1352f5/Lib/test/datetimetester.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/17931/{{ revealButtonHref }}
https://github.com/python/cpython/pull/17931/files#diff-917d5d1d1423993f37a643afa9f2a2f9caa3c0f91e06c0d7ebea1b879d53b4ef
https://github.com/python/cpython/pull/17931/files#diff-917d5d1d1423993f37a643afa9f2a2f9caa3c0f91e06c0d7ebea1b879d53b4ef
Misc/NEWS.d/next/Library/2020-01-10-00-16-46.bpo-39280.VA476D.rsthttps://github.com/python/cpython/pull/17931/files#diff-e3b47b5a4229aab0a35b3a2775e37ec33f54df49906a795e47e9e0325f949940
View file https://github.com/cool-RR/cpython/blob/75bc88f2c23e199c07e7105b08b481031a1352f5/Misc/NEWS.d/next/Library/2020-01-10-00-16-46.bpo-39280.VA476D.rst
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/17931/{{ revealButtonHref }}
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.