Title: gh-154904: Speed up import of shutil by probing compression extensions by maxday · Pull Request #154908 · python/cpython · GitHub
Open Graph Title: gh-154904: Speed up import of shutil by probing compression extensions by maxday · Pull Request #154908 · python/cpython
X Title: gh-154904: Speed up import of shutil by probing compression extensions by maxday · Pull Request #154908 · python/cpython
Description: Speed up import shutil by probing compression extensions instead of importing the wrappers shutil imports bz2, lzma and compression.zstd at module scope only to set the _*_SUPPORTED booleans, then discards them, also dragging incompression._common[._streams] and compression.zstd._zstdfile. Every program importing shutil pays for it even if it never touches an archive. They are pure Python wrappers whose only dependency that can be missing is the extension they wrap, so probe that directly: - import bz2 - del bz2 + import _bz2 + del _bz2 Same for lzma→_lzma, compression.zstd→_zstd. zlib is left alone: it is the extension. tarfile/zipfile still import the wrappers when an archive is created or extracted. Net import cost (median of 40 fresh interpreters, minus -c pass, one core): before after delta import shutil 12.75 ms 10.25 ms −2.50 ms (−19.6%) import urllib.request 36.64 ms 34.30 ms −2.34 ms (−6.4%) urllib.request benefits via tempfile→shutil. Motivation is short-lived processes on constrained hardware (CLI tools, serverless cold starts). No API change: flags, both registries and get_{archive,unpack}_formats() are unchanged, including on builds missing a compression library. The probe is a real import, so ImportError semantics are exact, unlike find_spec('bz2'), which resolves the always-present wrapper and reports success where _bz2 failed to compile (the case on my machine). PEP 810 lazy import also won't do: the flags are read at module scope (shutil.py:1145, :1354), so must be final at import time. Tests Added to TestArchives: test_compression_supported_flags pins the flags to real import semantics, test_compression_wrappers_not_imported_by_shutil prevents the eager imports returning. Existing test_unpack_archive_* coverage unchanged; patchcheck clean. $ ./python -m test test_shutil -v -m "test_compression*" ok, ok $ ./python -m test test_shutil test_zipfile test_tarfile test_importlib \ test_site test_venv test_zipimport test_bz2 test_lzma test_zstd \ test_gzip test_urllib2 test_logging -j6 SUCCESS — 14 OK, run=3,959 skipped=314 (test_bz2 skips on main too) negative controls, each test fails without the half it covers: find_spec variant: ..._supported_flags FAIL: True != False unpatched main: ..._wrappers_not_imported FAIL: "['lzma', 'compression', 'compression.zstd']" != '[]' fixes: #154904 Issue: gh-154904
Open Graph Description: Speed up import shutil by probing compression extensions instead of importing the wrappers shutil imports bz2, lzma and compression.zstd at module scope only to set the _*_SUPPORTED booleans, then ...
X Description: Speed up import shutil by probing compression extensions instead of importing the wrappers shutil imports bz2, lzma and compression.zstd at module scope only to set the _*_SUPPORTED booleans, then ...
Opengraph URL: https://github.com/python/cpython/pull/154908
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:4bb2ffd9-2e33-f8a3-e700-c0f98d00c59b |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | CAEE:17FB7:34840A2:4298378:6B021263 |
| html-safe-nonce | a91072297bf04736d30cd869b702997c3f51d8629cbc81ff68920d6c84788383 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDQUVFOjE3RkI3OjM0ODQwQTI6NDI5ODM3ODo2QjAyMTI2MyIsInZpc2l0b3JfaWQiOiIyNjg4MDAxNzQwNjQ4ODc4NjkxIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | 9b406d78c4c0ff9a7921c13b96f9ebc37b2e0f46d2e26493c3b106e10a3317e0 |
| hovercard-subject-tag | pull_request:4165747394 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/python/cpython/pull/154908/files |
| twitter:image | https://avatars.githubusercontent.com/u/864493?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/864493?s=400&v=4 |
| og:image:alt | Speed up import shutil by probing compression extensions instead of importing the wrappers shutil imports bz2, lzma and compression.zstd at module scope only to set the _*_SUPPORTED booleans, then ... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width