René's URL Explorer Experiment


Title: gh-155460: Pre-allocate the output buffer in compression.zstd decompression by ThomasWaldmann · Pull Request #155464 · python/cpython · GitHub

Open Graph Title: gh-155460: Pre-allocate the output buffer in compression.zstd decompression by ThomasWaldmann · Pull Request #155464 · python/cpython

X Title: gh-155460: Pre-allocate the output buffer in compression.zstd decompression by ThomasWaldmann · Pull Request #155464 · python/cpython

Description: Issue: gh-155460 compression.zstd decompression grows its output buffer step by step, even though most frames record their decompressed size in the frame header (zstd.compress() writes it by default). With this PR, when the first input of a frame holds the whole frame and the frame records its content size, the output buffer is allocated once, at that size. libzstd then decodes straight into it in a single pass, instead of through its internal window buffer. The recorded size is untrusted, so it is used only if: it is at most 1 GiB (OUTPUT_PREALLOC_MAX); the frame is at least 1/32768 of it (OUTPUT_MAX_EXPANSION): a block decodes to at most 128 KiB and takes at least 4 bytes, so no valid frame expands more; the frame's window size fits window_log_max, which libzstd's single-pass mode doesn't check. Otherwise the buffer grows as before. libzstd fails a frame that decodes to another size than recorded, whether the output was pre-allocated or not, and max_length still limits the allocation. zstd.decompress() feeds each frame of multi-frame input whole (sized with get_frame_size()), so each frame is pre-allocated, not only the first. Throughput at level 3, measured with pyperf (table): on Linux/x86-64 about 2x for 256 KiB to 16 MiB frames and unchanged from 32 MiB up; on macOS/arm64 7-29% higher at all sizes. For 1000 frames of 512 KiB to 8 MiB it is 14% higher on Linux and 18% higher on macOS. Co-authored-by: Maik Schulze maik.schulze.cv@googlemail.com 🤖 Generated with Claude Code

Open Graph Description: Issue: gh-155460 compression.zstd decompression grows its output buffer step by step, even though most frames record their decompressed size in the frame header (zstd.compress() writes it by defa...

X Description: Issue: gh-155460 compression.zstd decompression grows its output buffer step by step, even though most frames record their decompressed size in the frame header (zstd.compress() writes it by defa...

Opengraph URL: https://github.com/python/cpython/pull/155464

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:45f7ce0d-39cb-f539-b1a1-19872dd3d8f0
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idB6D6:19B6A2:14238B8:1AA34EB:6B01F485
html-safe-noncea647826ee62f929015aff01fabcc977014b3c653bc10967aa335a4592bfb72b7
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCNkQ2OjE5QjZBMjoxNDIzOEI4OjFBQTM0RUI6NkIwMUY0ODUiLCJ2aXNpdG9yX2lkIjoiMjM3MzU1ODY3NjU3ODQ5NzY2OSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmace45b58711c4363534c07b2f69b49201a10d45bf08993f98de2fc661d57191c13
hovercard-subject-tagpull_request:4242426397
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/python/cpython/pull/155464/files
twitter:imagehttps://avatars.githubusercontent.com/u/356103?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/356103?s=400&v=4
og:image:altIssue: gh-155460 compression.zstd decompression grows its output buffer step by step, even though most frames record their decompressed size in the frame header (zstd.compress() writes it by defa...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/python/cpython git https://github.com/python/cpython.git
octolytics-dimension-user_id1525981
octolytics-dimension-user_loginpython
octolytics-dimension-repository_id81598961
octolytics-dimension-repository_nwopython/cpython
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id81598961
octolytics-dimension-repository_network_root_nwopython/cpython
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release423dbb54c5e7bfecc838eb387c1435eb3d0e760d
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/python/cpython/pull/155464/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fpull%2F155464%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fpull%2F155464%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=python%2Fcpython
Reloadhttps://github.com/python/cpython/pull/155464/files
Reloadhttps://github.com/python/cpython/pull/155464/files
Reloadhttps://github.com/python/cpython/pull/155464/files
pythonhttps://github.com/python
cpythonhttps://github.com/python/cpython
Codehttps://github.com/python/cpython
Issues5k+ (5k+)https://github.com/python/cpython/issues
Pull requests2.4k (2.4k)https://github.com/python/cpython/pulls
Actionshttps://github.com/python/cpython/actions
Projectshttps://github.com/python/cpython/projects
Security and qualityhttps://github.com/python/cpython/security
Insightshttps://github.com/python/cpython/insights
Sign up for GitHub https://github.com/signup?return_to=%2Fpython%2Fcpython%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fpython%2Fcpython%2Fissues%2Fnew%2Fchoose
ThomasWaldmannhttps://github.com/ThomasWaldmann
python:mainhttps://github.com/python/cpython/tree/main
ThomasWaldmann:zstd-decompress-preallochttps://github.com/ThomasWaldmann/cpython/tree/zstd-decompress-prealloc
Conversation 1 https://github.com/python/cpython/pull/155464
Commits 3 https://github.com/python/cpython/pull/155464/commits
Checks 54 https://github.com/python/cpython/pull/155464/checks
Files changed https://github.com/python/cpython/pull/155464/files
Please reload this pagehttps://github.com/python/cpython/pull/155464/files
gh-155460: Pre-allocate the output buffer in compression.zstd decompression https://github.com/python/cpython/pull/155464/files#top
Show all changes 3 commits https://github.com/python/cpython/pull/155464/files
0814951 gh-155460: Pre-allocate the output buffer in compression.zstd decompr… ThomasWaldmann Aug 10, 2026 https://github.com/python/cpython/pull/155464/commits/0814951d81cdf6eae01ada14ecf366b5d0109604
431f0f1 gh-155460: Pre-size each whole zstd frame, respect window_log_max ThomasWaldmann Nov 6, 2026 https://github.com/python/cpython/pull/155464/commits/431f0f1255f6afb352e047f1da344d5eee7753b7
d81eed6 gh-155460: Make the multi-frame zstd test depend on whole-frame feeding ThomasWaldmann Nov 6, 2026 https://github.com/python/cpython/pull/155464/commits/d81eed6fe3efa8fdf6973895de9631c892a59422
Clear filters https://github.com/python/cpython/pull/155464/files
Please reload this pagehttps://github.com/python/cpython/pull/155464/files
Please reload this pagehttps://github.com/python/cpython/pull/155464/files
_streams.py https://github.com/python/cpython/pull/155464/files#diff-f9d600f1d9215778b168f18fa2b16d83dd6981dbdee25153d648c1c773015a56
__init__.py https://github.com/python/cpython/pull/155464/files#diff-5c2100c5a4e2ff7a1b201c86b489ad2425a74477e54d2c73af63972fc3907259
test_zstd.py https://github.com/python/cpython/pull/155464/files#diff-d347ebfcbcfd73ae33135b08623594cc45342d42b774fbbc80e508176f860d5d
2026-08-09-21-45-00.gh-issue-155460.zstdsz.rst https://github.com/python/cpython/pull/155464/files#diff-7ed92e470b4b6d81aeea3217a68531390497b5738317e418517f454fba074a7b
decompressor.c https://github.com/python/cpython/pull/155464/files#diff-5b717a4e417388ae310008129390ff165e9273f8183293709512e4c4feaa4889
Lib/compression/_common/_streams.pyhttps://github.com/python/cpython/pull/155464/files#diff-f9d600f1d9215778b168f18fa2b16d83dd6981dbdee25153d648c1c773015a56
View file https://github.com/python/cpython/blob/d81eed6fe3efa8fdf6973895de9631c892a59422/Lib/compression/_common/_streams.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/155464/{{ revealButtonHref }}
https://github.com/python/cpython/pull/155464/files#diff-f9d600f1d9215778b168f18fa2b16d83dd6981dbdee25153d648c1c773015a56
https://github.com/python/cpython/pull/155464/files#diff-f9d600f1d9215778b168f18fa2b16d83dd6981dbdee25153d648c1c773015a56
https://github.com/python/cpython/pull/155464/files#diff-f9d600f1d9215778b168f18fa2b16d83dd6981dbdee25153d648c1c773015a56
https://github.com/python/cpython/blob/main/.github/CODEOWNERS#L663
Lib/compression/zstd/__init__.pyhttps://github.com/python/cpython/pull/155464/files#diff-5c2100c5a4e2ff7a1b201c86b489ad2425a74477e54d2c73af63972fc3907259
View file https://github.com/python/cpython/blob/d81eed6fe3efa8fdf6973895de9631c892a59422/Lib/compression/zstd/__init__.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/155464/{{ revealButtonHref }}
https://github.com/python/cpython/pull/155464/files#diff-5c2100c5a4e2ff7a1b201c86b489ad2425a74477e54d2c73af63972fc3907259
https://github.com/python/cpython/pull/155464/files#diff-5c2100c5a4e2ff7a1b201c86b489ad2425a74477e54d2c73af63972fc3907259
https://github.com/python/cpython/blob/main/.github/CODEOWNERS#L664
Lib/test/test_zstd.pyhttps://github.com/python/cpython/pull/155464/files#diff-d347ebfcbcfd73ae33135b08623594cc45342d42b774fbbc80e508176f860d5d
View file https://github.com/python/cpython/blob/d81eed6fe3efa8fdf6973895de9631c892a59422/Lib/test/test_zstd.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/155464/{{ revealButtonHref }}
https://github.com/python/cpython/pull/155464/files#diff-d347ebfcbcfd73ae33135b08623594cc45342d42b774fbbc80e508176f860d5d
https://github.com/python/cpython/pull/155464/files#diff-d347ebfcbcfd73ae33135b08623594cc45342d42b774fbbc80e508176f860d5d
Misc/NEWS.d/next/Library/2026-08-09-21-45-00.gh-issue-155460.zstdsz.rsthttps://github.com/python/cpython/pull/155464/files#diff-7ed92e470b4b6d81aeea3217a68531390497b5738317e418517f454fba074a7b
View file https://github.com/python/cpython/blob/d81eed6fe3efa8fdf6973895de9631c892a59422/Misc/NEWS.d/next/Library/2026-08-09-21-45-00.gh-issue-155460.zstdsz.rst
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/155464/{{ revealButtonHref }}
Please reload this pagehttps://github.com/python/cpython/pull/155464/files
Please reload this pagehttps://github.com/python/cpython/pull/155464/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.