René's URL Explorer Experiment


Title: `tokenize.generate_tokens()` performance regression in 3.12 · Issue #119118 · python/cpython · GitHub

Open Graph Title: `tokenize.generate_tokens()` performance regression in 3.12 · Issue #119118 · python/cpython

X Title: `tokenize.generate_tokens()` performance regression in 3.12 · Issue #119118 · python/cpython

Description: Bug report Bug description: There seems to be a significant performance regression in tokenize.generate_tokens() between 3.11 and 3.12 when tokenizing a (very) large dict on a single line. I searched the existing issues but couldn't find...

Open Graph Description: Bug report Bug description: There seems to be a significant performance regression in tokenize.generate_tokens() between 3.11 and 3.12 when tokenizing a (very) large dict on a single line. I search...

X Description: Bug report Bug description: There seems to be a significant performance regression in tokenize.generate_tokens() between 3.11 and 3.12 when tokenizing a (very) large dict on a single line. I search...

Opengraph URL: https://github.com/python/cpython/issues/119118

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"`tokenize.generate_tokens()` performance regression in 3.12","articleBody":"# Bug report\r\n\r\n### Bug description:\r\n\r\nThere seems to be a significant performance regression in `tokenize.generate_tokens()` between 3.11 and 3.12 when tokenizing a (very) large dict on a single line. I searched the existing issues but couldn't find anything about this.\r\n\r\nTo reproduce, rename the file [largedict.py.txt](https://github.com/python/cpython/files/15348502/largedict.py.txt) to `largedict.py` in the same directory as the script below, then run the script. That file comes from https://github.com/nedbat/coveragepy/issues/1785.\r\n\r\n```python\r\nimport io, time, sys, tokenize\r\nimport largedict\r\n\r\ntext = largedict.d\r\nreadline = io.StringIO(text).readline\r\nglob_start = start = time.time()\r\nprint(f\"{sys.implementation.name} {sys.platform} {sys.version}\")\r\n\r\nfor i, (ttype, ttext, (sline, scol), (_, ecol), _) in enumerate(tokenize.generate_tokens(readline)):\r\n    if i % 500 == 0:\r\n        print(i, ttype, ttext, sline, scol, time.time() - start)\r\n        start = time.time()\r\n    if i % 5000 == 0:\r\n        print(time.time() - glob_start)\r\n\r\nprint(f\"Time taken: {time.time() - glob_start}\")\r\n```\r\n\r\nFor Python 3.12, this results in:\r\n```\r\ncpython linux 3.12.3 (main, May 17 2024, 07:19:22) [GCC 11.4.0]\r\n0 1 a_large_dict_literal 1 0 0.04641866683959961\r\n0.046633005142211914\r\n500 3 ':tombol_a_(golongan_darah):' 1 2675 9.689745903015137\r\n1000 3 ':flagge_anguilla:' 1 5261 9.767053604125977\r\n1500 3 ':флаг_Армения:' 1 7879 9.258271932601929\r\n[...]\r\n```\r\n\r\nFor Python 3.11, this results in:\r\n```\r\ncpython linux 3.11.0rc1 (main, Aug 12 2022, 10:02:14) [GCC 11.2.0]\r\n0 1 a_large_dict_literal 1 0 0.013637304306030273\r\n0.013663768768310547\r\n500 3 ':tombol_a_(golongan_darah):' 1 2675 0.002939462661743164\r\n1000 3 ':flagge_anguilla:' 1 5261 0.0028715133666992188\r\n1500 3 ':флаг_Армения:' 1 7879 0.002806425094604492\r\n[...]\r\n352500 3 'pt' 1 2589077 0.003370046615600586\r\nTime taken: 2.1244866847991943\r\n```\r\n\r\nThat is, each 500 tokens in Python 3.12 is taking over 9 seconds to process, while the 352500 tokens in Python 3.11 is taking a bit over 2 seconds to process. \r\n\r\nI can reproduce this on Linux (WSL) and Windows. Also seems to affect 3.13.\r\n\r\n### CPython versions tested on:\r\n\r\n3.9, 3.10, 3.11, 3.12\r\n\r\n### Operating systems tested on:\r\n\r\nLinux, Windows\n\n\u003c!-- gh-linked-prs -->\n### Linked PRs\n* gh-119615\n* gh-119682\n* gh-119683\n\u003c!-- /gh-linked-prs -->\n","author":{"url":"https://github.com/devdanzin","@type":"Person","name":"devdanzin"},"datePublished":"2024-05-17T11:49:37Z","url":"https://github.com/python/cpython/issues/119118"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:13162630-a942-f6df-dcc7-8e2455dcdd12
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-idAA58:1F2B:1489FD0:1B589E0:6B020244
html-safe-nonce75305edb7d7d0ccb07c3a09064145ad94eea9a3b627e33c7de165a46f540cd13
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBQTU4OjFGMkI6MTQ4OUZEMDoxQjU4OUUwOjZCMDIwMjQ0IiwidmlzaXRvcl9pZCI6IjI0MDAxMzU1MzQ3NzQxOTA2NjAiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmac80f6c7fbc1e5d3a6982184f5cdca828274d28bd17b999401db21ac21f14c643b
hovercard-subject-tagissue:2302529303
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/119118/issue_layout
twitter:imagehttps://opengraph.githubassets.com/1839b84d3db155238b0a872a5f5ab9bbcf26565a7540bd66b44601383ff2d5d0/python/cpython/issues/119118
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/1839b84d3db155238b0a872a5f5ab9bbcf26565a7540bd66b44601383ff2d5d0/python/cpython/issues/119118
og:image:altBug report Bug description: There seems to be a significant performance regression in tokenize.generate_tokens() between 3.11 and 3.12 when tokenizing a (very) large dict on a single line. I search...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernamedevdanzin
hostnamegithub.com
expected-hostnamegithub.com
None87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456
turbo-cache-controlno-preview
go-importgithub.com/python/cpython git https://github.com/python/cpython.git
octolytics-dimension-user_id1525981
octolytics-dimension-user_loginpython
octolytics-dimension-repository_id81598961
octolytics-dimension-repository_nwopython/cpython
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id81598961
octolytics-dimension-repository_network_root_nwopython/cpython
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release423dbb54c5e7bfecc838eb387c1435eb3d0e760d
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/python/cpython/issues/119118#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fissues%2F119118
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fissues%2F119118
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=python%2Fcpython
Reloadhttps://github.com/python/cpython/issues/119118
Reloadhttps://github.com/python/cpython/issues/119118
Reloadhttps://github.com/python/cpython/issues/119118
pythonhttps://github.com/python
cpythonhttps://github.com/python/cpython
Codehttps://github.com/python/cpython
Issues5k+ (5k+)https://github.com/python/cpython/issues
Pull requests2.4k (2.4k)https://github.com/python/cpython/pulls
Actionshttps://github.com/python/cpython/actions
Projectshttps://github.com/python/cpython/projects
Security and qualityhttps://github.com/python/cpython/security
Insightshttps://github.com/python/cpython/insights
https://github.com/python/cpython
New issuehttps://github.com/login?return_to=https://github.com/python/cpython/issues/119118
New issuehttps://github.com/login?return_to=https://github.com/python/cpython/issues/119118
tokenize.generate_tokens() performance regression in 3.12https://github.com/python/cpython/issues/119118#top
https://github.com/pablogsal
https://github.com/lysnikolaou
performancePerformance or resource usagehttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22performance%22
type-bugAn unexpected behavior, bug, or errorhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22type-bug%22
https://github.com/devdanzin
devdanzinhttps://github.com/devdanzin
on May 17, 2024https://github.com/python/cpython/issues/119118#issue-2302529303
largedict.py.txthttps://github.com/python/cpython/files/15348502/largedict.py.txt
coveragepy/coveragepy#1785https://github.com/coveragepy/coveragepy/issues/1785
gh-119118: Fix performance regression in tokenize module #119615https://github.com/python/cpython/pull/119615
[3.13] gh-119118: Fix performance regression in tokenize module (GH-119615) #119682https://github.com/python/cpython/pull/119682
[3.12] gh-119118: Fix performance regression in tokenize module (GH-119615) #119683https://github.com/python/cpython/pull/119683
devdanzinhttps://github.com/devdanzin
type-bugAn unexpected behavior, bug, or errorhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22type-bug%22
on May 17, 2024https://github.com/python/cpython/issues/119118#event-12847086098
AlexWaygoodhttps://github.com/AlexWaygood
performancePerformance or resource usagehttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22performance%22
on May 17, 2024https://github.com/python/cpython/issues/119118#event-12847242640
AlexWaygoodhttps://github.com/AlexWaygood
pablogsalhttps://github.com/pablogsal
lysnikolaouhttps://github.com/lysnikolaou
on May 17, 2024https://github.com/python/cpython/issues/119118#event-12847249646
https://github.com/erlend-aasland
erlend-aaslandhttps://github.com/erlend-aasland
on May 17, 2024https://github.com/python/cpython/issues/119118#issuecomment-2117668633
PEP-701https://peps.python.org/701
https://github.com/pablogsal
pablogsalhttps://github.com/pablogsal
on May 17, 2024https://github.com/python/cpython/issues/119118#issuecomment-2117687187
https://github.com/devdanzin
devdanzinhttps://github.com/devdanzin
on May 17, 2024https://github.com/python/cpython/issues/119118#issuecomment-2118000210
https://github.com/pablogsal
pablogsalhttps://github.com/pablogsal
on May 17, 2024https://github.com/python/cpython/issues/119118#issuecomment-2118007553
https://github.com/pablogsal
pablogsalhttps://github.com/pablogsal
on May 17, 2024https://github.com/python/cpython/issues/119118#issuecomment-2118187864
cpython/Python/Python-tokenize.chttps://github.com/python/cpython/blob/31a28cbae0989f57ad01b428c007dade24d9593a/Python/Python-tokenize.c#L213
31a28cbhttps://github.com/python/cpython/commit/31a28cbae0989f57ad01b428c007dade24d9593a
https://github.com/erlend-aasland
erlend-aaslandhttps://github.com/erlend-aasland
on May 17, 2024https://github.com/python/cpython/issues/119118#issuecomment-2118216631
1https://github.com/python/cpython/issues/119118#user-content-fn-1-c19b93c963ddc494de7b7c7ec4ff05cd
↩https://github.com/python/cpython/issues/119118#user-content-fnref-1-c19b93c963ddc494de7b7c7ec4ff05cd
https://github.com/lysnikolaou
lysnikolaouhttps://github.com/lysnikolaou
on May 17, 2024https://github.com/python/cpython/issues/119118#issuecomment-2118346688
https://github.com/lysnikolaou
lysnikolaouhttps://github.com/lysnikolaou
on May 17, 2024https://github.com/python/cpython/issues/119118#issuecomment-2118355032
https://github.com/nedbat
nedbathttps://github.com/nedbat
on May 25, 2024https://github.com/python/cpython/issues/119118#issuecomment-2131414377
@lysnikolaouhttps://github.com/lysnikolaou
https://github.com/lysnikolaou
lysnikolaouhttps://github.com/lysnikolaou
on May 25, 2024https://github.com/python/cpython/issues/119118#issuecomment-2131415507
@nedbathttps://github.com/nedbat
#119118 (comment)https://github.com/python/cpython/issues/119118#issuecomment-2118355032
https://github.com/pablogsal
pablogsalhttps://github.com/pablogsal
on May 25, 2024https://github.com/python/cpython/issues/119118#issuecomment-2131417726
@lysnikolaouhttps://github.com/lysnikolaou
lysnikolaouhttps://github.com/lysnikolaou
on May 27, 2024https://github.com/python/cpython/issues/119118#event-12947905639
pythongh-119118: Fix performance regression in tokenize modulehttps://github.com/lysnikolaou/cpython/commit/66f9385f692bbd75ceb35e6278b906123aa4e3ba
pythongh-119118https://github.com/python/cpython/issues/119118
66f9385https://github.com/lysnikolaou/cpython/commit/66f9385f692bbd75ceb35e6278b906123aa4e3ba
pablogsalhttps://github.com/pablogsal
on May 28, 2024https://github.com/python/cpython/issues/119118#event-12962190454
gh-119118: Fix performance regression in tokenize module (#119615)https://github.com/python/cpython/commit/d87b0151062e36e67f9e42e1595fba5bf23a485c
gh-119118https://github.com/python/cpython/issues/119118
#119615https://github.com/python/cpython/pull/119615
d87b015https://github.com/python/cpython/commit/d87b0151062e36e67f9e42e1595fba5bf23a485c
miss-islingtonhttps://github.com/miss-islington
on May 28, 2024https://github.com/python/cpython/issues/119118#event-12962191482
pythongh-119118: Fix performance regression in tokenize module (pytho…https://github.com/miss-islington/cpython/commit/64be1b7aee2d42e44b568a3ee93260e3ddca9651
pythongh-119118https://github.com/python/cpython/issues/119118
pytho…https://github.com/python/cpython/pull/119615
64be1b7https://github.com/miss-islington/cpython/commit/64be1b7aee2d42e44b568a3ee93260e3ddca9651
pythongh-119118: Fix performance regression in tokenize module (pytho…https://github.com/miss-islington/cpython/commit/8e935c9e388c8dbd53f8e9e4ad3189b9fdf890eb
pythongh-119118https://github.com/python/cpython/issues/119118
pytho…https://github.com/python/cpython/pull/119615
8e935c9https://github.com/miss-islington/cpython/commit/8e935c9e388c8dbd53f8e9e4ad3189b9fdf890eb
lysnikolaouhttps://github.com/lysnikolaou
on May 28, 2024https://github.com/python/cpython/issues/119118#event-12963048930
[3.13] gh-119118: Fix performance regression in tokenize module (GH-1…https://github.com/python/cpython/commit/0d0be6b3efeace4743329f81c08f9720cc221207
gh-119118https://github.com/python/cpython/issues/119118
GH-1…https://github.com/python/cpython/pull/119615
0d0be6bhttps://github.com/python/cpython/commit/0d0be6b3efeace4743329f81c08f9720cc221207
[3.12] gh-119118: Fix performance regression in tokenize module (GH-1…https://github.com/python/cpython/commit/4a0af0cfdcc0b81da5d78dc219df4985c4403f9c
gh-119118https://github.com/python/cpython/issues/119118
GH-1…https://github.com/python/cpython/pull/119615
4a0af0chttps://github.com/python/cpython/commit/4a0af0cfdcc0b81da5d78dc219df4985c4403f9c
https://github.com/lysnikolaou
lysnikolaouhttps://github.com/lysnikolaou
on May 28, 2024https://github.com/python/cpython/issues/119118#issuecomment-2136081660
#119615https://github.com/python/cpython/pull/119615
lysnikolaouhttps://github.com/lysnikolaou
completedhttps://github.com/python/cpython/issues?q=is%3Aissue%20state%3Aclosed%20archived%3Afalse%20reason%3Acompleted
on May 28, 2024https://github.com/python/cpython/issues/119118#event-12963066310
estyxxhttps://github.com/estyxx
on Jul 17, 2024https://github.com/python/cpython/issues/119118#event-13533252818
pythongh-119118: Fix performance regression in tokenize module (pytho…https://github.com/estyxx/cpython/commit/2b2bd1146f2e18ae67e863cf824aa71fc6336dd5
pythongh-119118https://github.com/python/cpython/issues/119118
pytho…https://github.com/python/cpython/pull/119615
2b2bd11https://github.com/estyxx/cpython/commit/2b2bd1146f2e18ae67e863cf824aa71fc6336dd5
Sign up for freehttps://github.com/signup?return_to=https://github.com/python/cpython/issues/119118
Sign in to commenthttps://github.com/login?return_to=https://github.com/python/cpython/issues/119118
lysnikolaouhttps://github.com/lysnikolaou
pablogsalhttps://github.com/pablogsal
performancePerformance or resource usagehttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22performance%22
type-bugAn unexpected behavior, bug, or errorhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22type-bug%22
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.