René's URL Explorer Experiment


Title: gh-148706: Pure-Python pickle._Pickler can be significantly optimized by mjbommar · Pull Request #148707 · python/cpython · GitHub

Open Graph Title: gh-148706: Pure-Python pickle._Pickler can be significantly optimized by mjbommar · Pull Request #148707 · python/cpython

X Title: gh-148706: Pure-Python pickle._Pickler can be significantly optimized by mjbommar · Pull Request #148707 · python/cpython

Description: As discussed in #148706 , packages like dill can be made ~20-40% faster by optimizing various paths in the pure Python pickler that they subclass. Note: I included a number of experiment logs and microbenchmark data for reviewers to examine; should obviously be removed before merging, but I didn't see any documentation on an alternative for sharing this kind of info. A few key points, most of which are provided in more detail in the Misc/pickle-perf-diary.md doc and Misc/pickle-perf-data/ results: No new failures with downstream packages like attrs, dill, cloudpickle, joblib, etc. (3.15 already has some regressions) We could cherrypick smaller improvements one at a time; see the diary file to see how they individually contribute Some representative benchmarks: Main workload (pickle_pure_bench.py — in Misc/pickle-perf-data/): ┌──────────────────────┬────────┬───────────┬────────┐ │ Bench │ main │ F6 branch │ Δ │ ├──────────────────────┼────────┼───────────┼────────┤ │ list_of_ints_10k │ 0.7647 │ 0.4721 │ −38.3% │ ├──────────────────────┼────────┼───────────┼────────┤ │ list_of_strs_1k │ 0.3220 │ 0.2574 │ −20.1% │ ├──────────────────────┼────────┼───────────┼────────┤ │ dict_str_int_5k │ 1.0872 │ 0.7857 │ −27.7% │ ├──────────────────────┼────────┼───────────┼────────┤ │ deep_list │ 2.7034 │ 1.3783 │ −49.0% │ ├──────────────────────┼────────┼───────────┼────────┤ │ nested_list_of_dicts │ 2.0137 │ 1.2735 │ −36.8% │ └──────────────────────┴────────┴───────────┴────────┘ Bytes workload (pickle_pure_bench_bytes.py): ┌─────────────────────────┬────────┬────────┬────────────────────────────────────┐ │ Bench │ pre-F6 │ F6 │ Δ │ ├─────────────────────────┼────────┼────────┼────────────────────────────────────┤ │ list_of_short_bytes_5k │ 1.5257 │ 1.3631 │ −10.7% │ ├─────────────────────────┼────────┼────────┼────────────────────────────────────┤ │ list_of_medium_bytes_1k │ 0.2943 │ 0.2597 │ −11.7% │ ├─────────────────────────┼────────┼────────┼────────────────────────────────────┤ │ dict_bytes_to_int_2k │ 0.8423 │ 0.7658 │ −9.1% │ ├─────────────────────────┼────────┼────────┼────────────────────────────────────┤ │ list_of_bytearrays_1k │ 0.3094 │ 0.3107 │ +0.4% (bytearray not in fast path) │ └─────────────────────────┴────────┴────────┴────────────────────────────────────┘ dill dumps bench (5 shapes): ┌──────────────────────┬─────────┬─────────┬────────┐ │ Shape │ main │ F6 │ Δ │ ├──────────────────────┼─────────┼─────────┼────────┤ │ flat_list_of_ints_5k │ 0.217 s │ 0.136 s │ −37.3% │ ├──────────────────────┼─────────┼─────────┼────────┤ │ list_of_strs_1k │ 0.070 s │ 0.057 s │ −18.8% │ ├──────────────────────┼─────────┼─────────┼────────┤ │ dict_str_int_3k │ 0.358 s │ 0.261 s │ −27.0% │ ├──────────────────────┼─────────┼─────────┼────────┤ │ list_of_tuples_2k │ 0.491 s │ 0.391 s │ −20.3% │ ├──────────────────────┼─────────┼─────────┼────────┤ │ mixed_records_500 │ 0.413 s │ 0.269 s │ −34.8% │ └──────────────────────┴─────────┴─────────┴────────┘ Assisted-by: gpt-5-4-xhigh Assisted-by: opus-4-7 Issue: gh-148706

Open Graph Description: As discussed in #148706 , packages like dill can be made ~20-40% faster by optimizing various paths in the pure Python pickler that they subclass. Note: I included a number of experiment logs and m...

X Description: As discussed in #148706 , packages like dill can be made ~20-40% faster by optimizing various paths in the pure Python pickler that they subclass. Note: I included a number of experiment logs and m...

Opengraph URL: https://github.com/python/cpython/pull/148707

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:2fe61270-27cb-465f-3c27-845abca9e52e
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-id8FB2:2D40DE:374C301:465BE26:6B022CDC
html-safe-noncedc6eb7f56411d9554975a7e571a0168a90089c773e975e1a1af94e645c05d90b
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI4RkIyOjJENDBERTozNzRDMzAxOjQ2NUJFMjY6NkIwMjJDREMiLCJ2aXNpdG9yX2lkIjoiNjUzMTIyNTg5Njg3NDAyODI1MiIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmac5f309f7b304604b58164a39d49dd7fd75b30626d0b31e93964184a6ed077edf2
hovercard-subject-tagpull_request:3548234581
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/python/cpython/pull/148707/files
twitter:imagehttps://avatars.githubusercontent.com/u/85053?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/85053?s=400&v=4
og:image:altAs discussed in #148706 , packages like dill can be made ~20-40% faster by optimizing various paths in the pure Python pickler that they subclass. Note: I included a number of experiment logs and m...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/python/cpython git https://github.com/python/cpython.git
octolytics-dimension-user_id1525981
octolytics-dimension-user_loginpython
octolytics-dimension-repository_id81598961
octolytics-dimension-repository_nwopython/cpython
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id81598961
octolytics-dimension-repository_network_root_nwopython/cpython
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release423dbb54c5e7bfecc838eb387c1435eb3d0e760d
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/python/cpython/pull/148707/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fpull%2F148707%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fpull%2F148707%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=python%2Fcpython
Reloadhttps://github.com/python/cpython/pull/148707/files
Reloadhttps://github.com/python/cpython/pull/148707/files
Reloadhttps://github.com/python/cpython/pull/148707/files
pythonhttps://github.com/python
cpythonhttps://github.com/python/cpython
Codehttps://github.com/python/cpython
Issues5k+ (5k+)https://github.com/python/cpython/issues
Pull requests2.4k (2.4k)https://github.com/python/cpython/pulls
Actionshttps://github.com/python/cpython/actions
Projectshttps://github.com/python/cpython/projects
Security and qualityhttps://github.com/python/cpython/security
Insightshttps://github.com/python/cpython/insights
Sign up for GitHub https://github.com/signup?return_to=%2Fpython%2Fcpython%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fpython%2Fcpython%2Fissues%2Fnew%2Fchoose
mjbommarhttps://github.com/mjbommar
python:mainhttps://github.com/python/cpython/tree/main
mjbommar:exp-pickle/4-pure-python-exact-containershttps://github.com/mjbommar/cpython/tree/exp-pickle/4-pure-python-exact-containers
Conversation 2 https://github.com/python/cpython/pull/148707
Commits 9 https://github.com/python/cpython/pull/148707/commits
Checks 51 https://github.com/python/cpython/pull/148707/checks
Files changed https://github.com/python/cpython/pull/148707/files
Please reload this pagehttps://github.com/python/cpython/pull/148707/files
gh-148706: Pure-Python pickle._Pickler can be significantly optimized https://github.com/python/cpython/pull/148707/files#top
Show all changes 9 commits https://github.com/python/cpython/pull/148707/files
94b53eb pickle: add exact-container fast paths to pure-Python _batch_appends … mjbommar Apr 17, 2026 https://github.com/python/cpython/pull/148707/commits/94b53eb9186cd06ecd111c81ed90dbf038d20e0d
bb9d721 pickle: inline frame-check and short-circuit common atomic types in s… mjbommar Apr 17, 2026 https://github.com/python/cpython/pull/148707/commits/bb9d72199059850b3d5ef84f1a2e67865031b67c
69eed04 pickle: add Misc/pickle-perf-diary.md + raw bench data mjbommar Apr 17, 2026 https://github.com/python/cpython/pull/148707/commits/69eed0405f8f6ec09dfcb51ee9833ae891803ea8
285fcae pickle: fix large-dict mutation semantics; reorder save() fast paths … mjbommar Apr 17, 2026 https://github.com/python/cpython/pull/148707/commits/285fcae806098fae487209af60d41d4c937de714
7c6af84 pickle: cache BININT1 opcode bytes for n in 0..255 (F4) mjbommar Apr 17, 2026 https://github.com/python/cpython/pull/148707/commits/7c6af84626238d257914ff90b65c8923f6c6e33b
2f1d38b pickle: inline the MEMOIZE proto-4+ path in memoize() (F2) mjbommar Apr 17, 2026 https://github.com/python/cpython/pull/148707/commits/2f1d38b12c01cb2067b7662745afde33a686db11
e917108 pickle: extend save() fast path to bytes (F6) mjbommar Apr 17, 2026 https://github.com/python/cpython/pull/148707/commits/e917108ace45f92c3ce80d0ad4907bbd22f56fe7
024a57c pickle: update perf diary for round 2 (F1-F6) + raw data mjbommar Apr 17, 2026 https://github.com/python/cpython/pull/148707/commits/024a57c44d702d2bd86081a988420f053feb3d41
4f0b43c gh-148706: add NEWS entry for pickle._Pickler optimization mjbommar Apr 18, 2026 https://github.com/python/cpython/pull/148707/commits/4f0b43c2b6b47317cba7cf86f492791727031b72
Clear filters https://github.com/python/cpython/pull/148707/files
Please reload this pagehttps://github.com/python/cpython/pull/148707/files
Please reload this pagehttps://github.com/python/cpython/pull/148707/files
pickle.py https://github.com/python/cpython/pull/148707/files#diff-0e6904a03cced1cde99717e889a6a20715fd002d7186a64e77477c2d99c32a47
2026-04-18-00-28-38.gh-issue-148706.mvlqm5.rst https://github.com/python/cpython/pull/148707/files#diff-a04099b03bef28d5ff5de87f70df06d07db4bed80cb01471979f837b26421fcc
README.md https://github.com/python/cpython/pull/148707/files#diff-d009bbe04d378e7ef179459760277243f44f0b4f96bd5a6362b0a70021a533aa
pickle-F1v2.json https://github.com/python/cpython/pull/148707/files#diff-5e405aa7ae46fc2623e5d96d488dfa827ef4d704cfd346b838fa9344a7b11a4b
pickle-F2.json https://github.com/python/cpython/pull/148707/files#diff-f7f45632633bb82fa0a7b67f2cf433980d463aef18856a7832849f7ffdce1e31
pickle-F3.json https://github.com/python/cpython/pull/148707/files#diff-1f80dca7da5f01aad1477ca693f42a75c2953aba680199493251d6281860e7ab
pickle-F4.json https://github.com/python/cpython/pull/148707/files#diff-cb134e02bf3e2f30066488aa5c1a312c1f9c4ef698a92fb2be82a0cb352d4e30
pickle-F5.json https://github.com/python/cpython/pull/148707/files#diff-a299ccc09115f3a422f618d10ad9e0221405b92db925588f9eaa30be3efaca38
pickle-F6v2.json https://github.com/python/cpython/pull/148707/files#diff-bd28d069d71513c70d8f45353bdcc426b672a3f10fa11c25efeb30c82bd5562f
pickle-bytes-F6v2.json https://github.com/python/cpython/pull/148707/files#diff-0ac2b6ff67f16f6e98b95a0b5fc4cf1acdca484efb57e451948adc11dcfe3e74
pickle-bytes-pre.json https://github.com/python/cpython/pull/148707/files#diff-c31e4154ad86b8a4952abeb69e4a39ead493df00087e177670f2e057ec878ae4
pickle-post-fix.json https://github.com/python/cpython/pull/148707/files#diff-b7560d797daaaa6b0f99c54e850ab211c83278e8b499580db92ae606466ff545
pickle-pure-BD.json https://github.com/python/cpython/pull/148707/files#diff-d5289670b264cdbe84e8813b6245db49c66bb31ebe4f64cea3b4bc547f4c4ef5
pickle-pure-DE-v2.json https://github.com/python/cpython/pull/148707/files#diff-732e9f46293828de47c19468644ecd52229587f601620cbfc21830cd6490de75
pickle-pure-DE.json https://github.com/python/cpython/pull/148707/files#diff-e2c4a22acbb4c0efcd4e0e4a5262d0248c3849f9eb4b10d9a2d57a81f1549768
pickle-pure-Donly-verify.json https://github.com/python/cpython/pull/148707/files#diff-caacc53afdb64e5d1c6b81c4b80cb9513885778bec6aaabc6de201532bb8c114
pickle-pure-baseline.json https://github.com/python/cpython/pull/148707/files#diff-7699797f0f8b35f9ae77e44c397b358b408ddafbd24242532d655667e4ac250c
pickle-pure-exp4d.json https://github.com/python/cpython/pull/148707/files#diff-d9098a91c8e500adc1f516e8e79e191901525f2cd2a8616286b932d4e960e84c
pickle_pure_bench.py https://github.com/python/cpython/pull/148707/files#diff-137cfdbf3a011a2684402a77ab114d755c8b6e847cd2b6cd2d100e22c4d4c35b
pickle_pure_bench_bytes.py https://github.com/python/cpython/pull/148707/files#diff-6c4d8afb76b64062c5a7b78d19d8a7127944f49a9b0a837e95d9cd362d3a4947
pickle_save_profile.py https://github.com/python/cpython/pull/148707/files#diff-db928fba49b0b2b38e772071882db4e71eb45306ae9f61e00cdb2a3d1fca3380
pickle-perf-diary.md https://github.com/python/cpython/pull/148707/files#diff-d4b1d6bf3081a77619f97e3e1808fafa602a611e3178f2ccd1c9222f2ca26155
Lib/pickle.pyhttps://github.com/python/cpython/pull/148707/files#diff-0e6904a03cced1cde99717e889a6a20715fd002d7186a64e77477c2d99c32a47
View file https://github.com/mjbommar/cpython/blob/4f0b43c2b6b47317cba7cf86f492791727031b72/Lib/pickle.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/148707/{{ revealButtonHref }}
https://github.com/python/cpython/pull/148707/files#diff-0e6904a03cced1cde99717e889a6a20715fd002d7186a64e77477c2d99c32a47
https://github.com/python/cpython/pull/148707/files#diff-0e6904a03cced1cde99717e889a6a20715fd002d7186a64e77477c2d99c32a47
https://github.com/python/cpython/pull/148707/files#diff-0e6904a03cced1cde99717e889a6a20715fd002d7186a64e77477c2d99c32a47
https://github.com/python/cpython/pull/148707/files#diff-0e6904a03cced1cde99717e889a6a20715fd002d7186a64e77477c2d99c32a47
https://github.com/python/cpython/pull/148707/files#diff-0e6904a03cced1cde99717e889a6a20715fd002d7186a64e77477c2d99c32a47
https://github.com/python/cpython/pull/148707/files#diff-0e6904a03cced1cde99717e889a6a20715fd002d7186a64e77477c2d99c32a47
https://github.com/python/cpython/pull/148707/files#diff-0e6904a03cced1cde99717e889a6a20715fd002d7186a64e77477c2d99c32a47
https://github.com/python/cpython/pull/148707/files#diff-0e6904a03cced1cde99717e889a6a20715fd002d7186a64e77477c2d99c32a47
https://github.com/python/cpython/pull/148707/files#diff-0e6904a03cced1cde99717e889a6a20715fd002d7186a64e77477c2d99c32a47
https://github.com/python/cpython/pull/148707/files#diff-0e6904a03cced1cde99717e889a6a20715fd002d7186a64e77477c2d99c32a47
https://github.com/python/cpython/pull/148707/files#diff-0e6904a03cced1cde99717e889a6a20715fd002d7186a64e77477c2d99c32a47
https://github.com/python/cpython/pull/148707/files#diff-0e6904a03cced1cde99717e889a6a20715fd002d7186a64e77477c2d99c32a47
https://github.com/python/cpython/pull/148707/files#diff-0e6904a03cced1cde99717e889a6a20715fd002d7186a64e77477c2d99c32a47
Misc/NEWS.d/next/Library/2026-04-18-00-28-38.gh-issue-148706.mvlqm5.rsthttps://github.com/python/cpython/pull/148707/files#diff-a04099b03bef28d5ff5de87f70df06d07db4bed80cb01471979f837b26421fcc
View file https://github.com/mjbommar/cpython/blob/4f0b43c2b6b47317cba7cf86f492791727031b72/Misc/NEWS.d/next/Library/2026-04-18-00-28-38.gh-issue-148706.mvlqm5.rst
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/148707/{{ revealButtonHref }}
Misc/pickle-perf-data/README.mdhttps://github.com/python/cpython/pull/148707/files#diff-d009bbe04d378e7ef179459760277243f44f0b4f96bd5a6362b0a70021a533aa
View file https://github.com/mjbommar/cpython/blob/4f0b43c2b6b47317cba7cf86f492791727031b72/Misc/pickle-perf-data/README.md
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/148707/{{ revealButtonHref }}
Please reload this pagehttps://github.com/python/cpython/pull/148707/files
Please reload this pagehttps://github.com/python/cpython/pull/148707/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.