René's URL Explorer Experiment


Title: zlib, gzip, lzma and zstd in 3.15: decompressing large outputs is slower on macOS and Windows, and the process footprint grows on macOS (bisected to f262297) · Issue #161478 · python/cpython · GitHub

Open Graph Title: zlib, gzip, lzma and zstd in 3.15: decompressing large outputs is slower on macOS and Windows, and the process footprint grows on macOS (bisected to f262297) · Issue #161478 · python/cpython

X Title: zlib, gzip, lzma and zstd in 3.15: decompressing large outputs is slower on macOS and Windows, and the process footprint grows on macOS (bisected to f262297) · Issue #161478 · python/cpython

Description: Bug report Since 3.15, decompressing a few MiB or more is markedly slower on macOS and Windows than in 3.14: 1.5 to 3 times for zstd, up to 1.5 times for zlib, gzip and lzma, and not measurably for bz2. Linux is unaffected. On macOS, the...

Open Graph Description: Bug report Since 3.15, decompressing a few MiB or more is markedly slower on macOS and Windows than in 3.14: 1.5 to 3 times for zstd, up to 1.5 times for zlib, gzip and lzma, and not measurably for...

X Description: Bug report Since 3.15, decompressing a few MiB or more is markedly slower on macOS and Windows than in 3.14: 1.5 to 3 times for zstd, up to 1.5 times for zlib, gzip and lzma, and not measurably for...

Opengraph URL: https://github.com/python/cpython/issues/161478

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"zlib, gzip, lzma and zstd in 3.15: decompressing large outputs is slower on macOS and Windows, and the process footprint grows on macOS (bisected to f262297)","articleBody":"### Bug report\n\nSince 3.15, decompressing a few MiB or more is markedly slower on macOS and Windows than in 3.14: 1.5 to 3 times for zstd, up to 1.5 times for zlib, gzip and lzma, and not measurably for bz2. Linux is unaffected. On macOS, the process also accumulates memory call after call: after a loop of 64 MiB decompressions, its footprint has grown by 3.4 GiB, against 134 MiB in 3.14. Both effects bisect to [f262297](https://github.com/python/cpython/commit/f262297d525e87906c5e4ab28e80284189641c9e) ([gh-139877](https://github.com/python/cpython/issues/139877)), which moved the output buffer that zlib, bz2, lzma and zstd share to `PyBytesWriter`. The tables below have the numbers.\n\nWe found it while moving a package manager from Python 3.9 with [python-zstandard](https://github.com/indygreg/python-zstandard) to the built-in [`compression.zstd`](https://docs.python.org/3.14/library/compression.zstd.html).\n\n**Reproducer.** The script decompresses 1, 4, 16 and 64 MiB with `zstd.decompress()` and `zlib.decompress()` and prints the best time per call. The data is 1/4 random bytes and 3/4 zeros, so it compresses about 4:1. Only the standard library is needed (3.14 or later):\n\n```python\nimport random\nimport sys\nimport timeit\nimport zlib\nfrom compression import zstd\n\nMiB = 1 \u003c\u003c 20\nchunk = random.Random(0).randbytes(MiB) + bytes(3 * MiB)     # compresses about 4:1\n\nfor name, module in ((\"zstd\", zstd), (\"zlib\", zlib)):\n    for size in (1, 4, 16, 64):\n        blob = module.compress((chunk * 16)[:size * MiB])\n        best = min(timeit.repeat(lambda: module.decompress(blob), number=1, repeat=max(5, 512 // size)))\n        print(f\"{sys.version.split()[0]} {name}: {size:2} MiB in {best * 1e3:7.3f} ms, {size * MiB / best / 1e9:5.1f} GB/s\")\n```\n\n**Setup.** 3.14.10 and 3.15.2 as distributed, the bisect below from source:\n- **Windows 11 x64:** Ryzen 9 7950X, 2 × 32 GB DDR5-5600; python-build-standalone builds (via uv).\n- **macOS 26.7 arm64:** Apple M4 Pro, 48 GB; python-build-standalone builds (via uv); both link zlib 1.2.12.\n- **Linux arm64:** the official `python:3.14-slim-trixie` and `python:3.15-slim-trixie` images, in Docker Desktop's VM on the same Mac.\n\nlibzstd is 1.5.7 everywhere.\n\n**Results.** The tables come from `bench_compression.py` (in the collapsed block at the end of this report), which runs the reproducer's measurement for every module, one-shot and with the decompressor objects, and also counts page faults and peak memory. Each host ran it three times per interpreter, alternating, best of three. Best time per call, 3.14.10 → 3.15.2:\n\n| `zstd.decompress()` | Windows x64 | macOS arm64 | Linux arm64 |\n|---|---:|---:|---:|\n| 1 MiB | 0.393 → 0.290 ms | 0.036 → 0.030 ms | 0.050 → 0.043 ms |\n| 4 MiB | 1.25 → 1.11 ms | 0.125 → **0.288 ms** | 0.156 → 0.122 ms |\n| 16 MiB | 4.34 → **6.51 ms** | 0.618 → **1.78 ms** | 1.82 → 0.867 ms |\n| 64 MiB | 16.7 → **27.2 ms** | 2.80 → **8.69 ms** | 4.95 → 3.65 ms |\n\n| `zlib.decompress()` | Windows x64 | macOS arm64 | Linux arm64 |\n|---|---:|---:|---:|\n| 1 MiB | 0.283 → 0.165 ms | 0.069 → 0.059 ms | 0.271 → 0.264 ms |\n| 4 MiB | 1.02 → 0.840 ms | 0.666 → **0.891 ms** | 1.54 → 1.58 ms |\n| 16 MiB | 4.25 → **6.24 ms** | 3.00 → **3.58 ms** | 10.8 → 9.50 ms |\n| 64 MiB | 17.6 → **26.4 ms** | 12.4 → **14.3 ms** | 40.6 → 42.5 ms |\n\n| time in 3.15.2 relative to 3.14.10 (+ is slower; bold: 15% or more) | 1 MiB | 4 MiB | 16 MiB | 64 MiB |\n|---|---:|---:|---:|---:|\n| Windows x64, `zstd` | -26% | -11% | **+50%** | **+62%** |\n| Windows x64, `gzip` | -24% | -18% | **+40%** | **+48%** |\n| Windows x64, `zlib` | -42% | -17% | **+47%** | **+50%** |\n| Windows x64, `lzma` | -20% | +1% | **+18%** | **+39%** |\n| Windows x64, `bz2` | +2% | +1% | +3% | +4% |\n| macOS arm64, `zstd` | -16% | **+130%** | **+188%** | **+210%** |\n| macOS arm64, `gzip` | -18% | **+26%** | **+42%** | **+44%** |\n| macOS arm64, `zlib` | -14% | **+34%** | **+19%** | **+16%** |\n| macOS arm64, `lzma` | -1% | +1% | -1% | +7% |\n| macOS arm64, `bz2` | 0% | 0% | +1% | +1% |\n| Linux arm64, `zstd` | -14% | -22% | -52% | -26% |\n| Linux arm64, `gzip` | -7% | -4% | -11% | +1% |\n| Linux arm64, `zlib` | -2% | +3% | -12% | +5% |\n| Linux arm64, `lzma` | -2% | 0% | -10% | -1% |\n| Linux arm64, `bz2` | +3% | +3% | +5% | +4% |\n\n- **The pattern is the same on macOS and Windows:** the slowdown starts between 1 and 4 MiB of output on macOS, between 4 and 16 MiB on Windows, and grows with the size. It is largest for zstd, whose decoder is the fastest, so the buffer is the largest share of its time; bz2's decoder is slow enough to hide it. The decompressor objects (`zlib.decompressobj()`, `ZstdDecompressor`, …) show the same as the one-shot functions.\n- **On macOS, how much slower 3.15 is depends on what the process did before.** The same 64 MiB zstd call takes 5.4 ms in a fresh process (2.0× 3.14's), 6.9 ms after the reproducer's smaller cases (2.5×, what the reproducer above prints), and 8.7 ms after the script's zlib and gzip cases (3.1×), with one page fault per page in all three; 3.14.10 takes 2.7–2.8 ms in all three.\n- **At 1 MiB,** 3.15 is faster than 3.14 for zstd, gzip and zlib on all three hosts, and for lzma on Windows; bz2 is unchanged.\n- **Compression** with zlib, gzip, lzma and bz2 stays within 5% on every host. `zstd.compress()` is 1.2–1.4× faster on Windows, and 1.3× slower on macOS for incompressible data.\n- **Other zstd paths** go through the same buffer and regress the same way: `ZstdFile.read()` is 2.0× slower at 64 MiB on macOS.\n- **Still present on `main`:** a PGO build of [7054b71](https://github.com/python/cpython/commit/7054b71e1547fa575fd243916dad5a5fbc65b238) (2026-11-05) behaves like 3.15.2 in the same run of the script: zstd 64 MiB on macOS in 8.17 against 8.69 ms, zlib 14.5 against 14.3 ms.\n\n**Memory.** One fresh process per case calls `decompress()` repeatedly on the reproducer's data and discards the result (`footprint.py`, in the second collapsed block at the end). The tables show how much the process's memory has grown since before the first call: on macOS the physical footprint (`task_info(TASK_VM_INFO).phys_footprint`, what Activity Monitor shows), on Linux `RssAnon` from `/proc/self/status`, on Windows the private commit (`PrivateUsage` from `GetProcessMemoryInfo`, measured with a port of the same script; that run used 40 calls where the others use 50). 3.14.10's growth doesn't change with the number of calls, so it has one column; where 3.15.2's exceeds five outputs it's bold.\n\nmacOS arm64 (`phys_footprint`):\n\n| | 3.14.10, any number of calls | 3.15.2, 10 calls | 50 calls | 100 calls | 200 calls | 400 calls |\n|---|---:|---:|---:|---:|---:|---:|\n| `zstd.decompress()`, 1 MiB | 3.0 MiB | 2.1 MiB | 2.1 MiB | 4.2 MiB | 4.2 MiB | 4.2 MiB |\n| `zstd.decompress()`, 4 MiB | 12 MiB | **45 MiB** | **208 MiB** | **260 MiB** | **260 MiB** | **260 MiB** |\n| `zstd.decompress()`, 16 MiB | 50 MiB | **165 MiB** | **518 MiB** | **518 MiB** | **504 MiB** | **507 MiB** |\n| `zstd.decompress()`, 64 MiB | 134 MiB | **646 MiB** | **1.2 GiB** | **2.2 GiB** | **2.4 GiB** | **3.4 GiB** |\n| `zlib.decompress()`, 1 MiB | 2.0 MiB | 1.5 MiB | 1.5 MiB | 1.5 MiB | 1.5 MiB | 2.6 MiB |\n| `zlib.decompress()`, 4 MiB | 11 MiB | **42 MiB** | **206 MiB** | **262 MiB** | **261 MiB** | **261 MiB** |\n| `zlib.decompress()`, 16 MiB | 46 MiB | **163 MiB** | **645 MiB** | **979 MiB** | **965 MiB** | **920 MiB** |\n| `zlib.decompress()`, 64 MiB | 128 MiB | **640 MiB** | **800 MiB** | **1.2 GiB** | **1.2 GiB** | **1.2 GiB** |\n\nWindows x64 (private commit retained):\n\n| | 3.14.10, any number of calls | 3.15.2, 10 calls | 40 calls | 100 calls | 200 calls | 400 calls |\n|---|---:|---:|---:|---:|---:|---:|\n| `zstd.decompress()`, 1 MiB | 0.3 MiB | 0.7 MiB | 0.7 MiB | 0.7 MiB | 0.7 MiB | 0.7 MiB |\n| `zstd.decompress()`, 4 MiB | 0.3 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB |\n| `zstd.decompress()`, 16 MiB | 0.5 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB |\n| `zstd.decompress()`, 64 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB |\n| `zlib.decompress()`, 1 MiB | 0.0 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB |\n| `zlib.decompress()`, 4 MiB | 0.2 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB |\n| `zlib.decompress()`, 16 MiB | 0.2 MiB | 0.5 MiB | 0.5 MiB | 0.5 MiB | 0.5 MiB | 0.5 MiB |\n| `zlib.decompress()`, 64 MiB | 0.0 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB |\n\nLinux arm64 (`RssAnon`):\n\n| | 3.14.10, any number of calls | 3.15.2, 10 calls | 50 calls | 100 calls | 200 calls | 400 calls |\n|---|---:|---:|---:|---:|---:|---:|\n| `zstd.decompress()`, 1 MiB | 0.0 MiB | 1.0 MiB | 1.0 MiB | 1.0 MiB | 1.0 MiB | 1.0 MiB |\n| `zstd.decompress()`, 4 MiB | 0.0 MiB | 3.6 MiB | 3.6 MiB | 3.6 MiB | 3.6 MiB | 3.6 MiB |\n| `zstd.decompress()`, 16 MiB | 0.0 MiB | 11 MiB | 11 MiB | 11 MiB | 11 MiB | 11 MiB |\n| `zstd.decompress()`, 64 MiB | 13 MiB | 6.4 MiB | 6.4 MiB | 6.4 MiB | 6.4 MiB | 6.4 MiB |\n| `zlib.decompress()`, 1 MiB | 0.0 MiB | 0.0 MiB | 0.0 MiB | 0.0 MiB | 0.0 MiB | 0.0 MiB |\n| `zlib.decompress()`, 4 MiB | 0.0 MiB | 1.2 MiB | 1.2 MiB | 1.2 MiB | 1.2 MiB | 1.2 MiB |\n| `zlib.decompress()`, 16 MiB | 0.0 MiB | 6.4 MiB | 6.4 MiB | 6.4 MiB | 6.4 MiB | 6.4 MiB |\n| `zlib.decompress()`, 64 MiB | 13 MiB | 0.0 MiB | 0.0 MiB | 0.0 MiB | 0.0 MiB | 0.0 MiB |\n\n- **On macOS, 3.15 grows with every call,** by about one output per call at first, until the allocator trims, at a bound that depends on the size: about 260 MiB for 4 MiB outputs, 0.5–1 GiB for 16 MiB, 1.2–3.4 GiB for 64 MiB. 3.14 stays at 2–3× the output. On Linux and Windows, neither version grows: glibc and the UCRT give large blocks back.\n- **Python doesn't leak:** tracemalloc's current traced memory is unchanged after the calls. The allocator keeps the freed memory, and macOS counts it against the process: it isn't marked reusable, and it isn't compressed. No other process was short of memory (65% of 48 GB free).\n- **Windows:** nothing accumulates, but the peak commit during a call is higher in 3.15 from 4 MiB (2.6–4.2× the output against 2.2–3.0× in 3.14), because `realloc` holds the old and the new block at once, plus the over-allocation; tracemalloc doesn't see that.\n\n### Bisected to f262297\n\n[f262297](https://github.com/python/cpython/commit/f262297d525e87906c5e4ab28e80284189641c9e) ([gh-139877](https://github.com/python/cpython/issues/139877), [PR #139976](https://github.com/python/cpython/pull/139976), \"Use PyBytesWriter in pycore_blocks_output_buffer.h\") against its parent [4044255](https://github.com/python/cpython/commit/404425575c68bef9d2f042710fc713134d04c23f), both built from source with PGO (and LTO on macOS), run alternately three times, best of three. \"Page faults per output page\" is the process's page faults during one call, divided by the output size in pages (16 KiB on macOS, 4 KiB elsewhere): 1.00 means the whole output was written into memory the process had never touched.\n\n| macOS arm64 | 4044255 | f262297 | page faults per output page |\n|---|---:|---:|---:|\n| `zstd.decompress()`, 1 MiB | 0.036 ms | 0.030 ms | 0.00 → 0.00 |\n| `zstd.decompress()`, 4 MiB | 0.127 ms | **0.279 ms** | 0.00 → 1.01 |\n| `zstd.decompress()`, 16 MiB | 0.624 ms | **1.81 ms** | 0.00 → 1.00 |\n| `zstd.decompress()`, 64 MiB | 2.72 ms | **8.93 ms** | 0.00 → 1.00 |\n| `zlib.decompress()`, 1 MiB | 0.069 ms | 0.064 ms | 0.00 → 0.00 |\n| `zlib.decompress()`, 4 MiB | 0.745 ms | **0.874 ms** | 0.00 → 1.01 |\n| `zlib.decompress()`, 16 MiB | 2.99 ms | **3.66 ms** | 0.00 → 1.00 |\n| `zlib.decompress()`, 64 MiB | 12.3 ms | **14.5 ms** | 0.00 → 1.00 |\n\nlzma gets 4–13% slower and bz2 about 1%, with the same rise in page faults. Across all modules, one-shot functions and decompressor objects alike, f262297 against its parent reproduces the 3.14.10 → 3.15.2 ratios on macOS within about 10% (zstd 64 MiB: 3.3× against 3.1×, zlib 1.18× against 1.15×).\n\n| Windows x64 | 4044255 | f262297 | page faults per output page |\n|---|---:|---:|---:|\n| `zstd.decompress()`, 1 MiB | 0.380 ms | 0.278 ms | 2.71 → 2.02 |\n| `zstd.decompress()`, 4 MiB | 1.27 ms | 1.10 ms | 2.50 → 2.08 |\n| `zstd.decompress()`, 16 MiB | 4.38 ms | **6.56 ms** | 2.13 → 3.28 |\n| `zstd.decompress()`, 64 MiB | 16.8 ms | **27.1 ms** | 2.03 → 3.33 |\n| `zlib.decompress()`, 1 MiB | 0.265 ms | 0.150 ms | 1.71 → 1.01 |\n| `zlib.decompress()`, 4 MiB | 1.01 ms | 0.832 ms | 1.94 → 1.51 |\n| `zlib.decompress()`, 16 MiB | 4.05 ms | **6.11 ms** | 1.99 → 3.13 |\n| `zlib.decompress()`, 64 MiB | 17.2 ms | **25.9 ms** | 2.00 → 3.29 |\n\nOn Windows too, the pair reproduces the 3.14.10 → 3.15.2 ratios for zlib, gzip and zstd: faster up to 4 MiB, 1.4–1.6× slower from 16 MiB. lzma gets up to 1.17× slower and bz2 stays within 2%, with the same rise in page faults.\n\n### Cause\n\nf262297 replaced the list of fixed-size blocks, joined at the end, with one `bytes` object grown with `PyBytesWriter_Grow()`. That saves the final copy, and its own benchmarks found decompression 10–30% faster. But each growth step is a `realloc`, and what that costs depends on the allocator:\n\n- **Linux (glibc):** `realloc` grows a large block in place (`mremap`). Page faults stay at 0.0–0.5 per output page on both versions (the VM uses transparent huge pages); zstd is faster in 3.15 and the other modules are within about 12%.\n- **macOS:** 3.14's blocks have fixed sizes, and `malloc` hands the previous call's blocks back, so the output is written into memory that's already mapped: 0.00 page faults per output page. In 3.15, `realloc` grows the buffer mostly in place, but into address space the process has never touched, so the decoder faults in every output page on every call: 1.00 per page from 4 MiB, in every module. (Measured by interposing `realloc` for a 64 MiB `zstd.decompress()`: seven large `realloc` calls per call, six in place, one move that copies 1.7 MiB.) The next call starts small and grows into new space again, so the freed block isn't reused, and the allocator keeps it: that's the footprint growth above. 3.15 copies less than 3.14, which joins its blocks with one full copy, and is still slower: faulting in fresh pages costs more than copying into mapped memory.\n- **Windows:** the process heap behind the UCRT's `malloc` returns blocks larger than about 512 KiB to the OS when they're freed, so 3.14 faults in fresh memory too, 2.0 per output page. 3.15 adds the `realloc` copies and [over-allocates by 50%](https://github.com/python/cpython/blob/7054b71e1547fa575fd243916dad5a5fbc65b238/Objects/bytesobject.c#L3823-L3829): an instrumented build showed every `realloc` from 1.34 MiB on moving the block and copying its whole capacity, including the tail that was never written (2, 8 and 20 MiB as the output passes 1.34, 5.34 and 13.34 MiB), for 3.3 faults per output page.\n\n### Fix\n\nCollect large outputs in blocks again and join them once, as 3.14 did, everywhere but on Linux, where `realloc` grows a block in place. I'll open a PR for this. With it, decompression is back at 3.14's speed within about 10% from 4 MiB on macOS and from 16 MiB on Windows, macOS sees no page faults, and its footprint over repeated calls stays at 2–3× the output. Two costs: 3.14's peak within one call, 2.2–2.9× the output instead of 1.5–2.3×, because the blocks and the joined result exist together at the end; and on Windows at 4 MiB, 3.14's speed too, where 3.15 had been about 20% faster.\n\n### Related\n\n- [gh-139877](https://github.com/python/cpython/issues/139877) and [PR #139976](https://github.com/python/cpython/pull/139976): the move to `PyBytesWriter`.\n- [gh-155460](https://github.com/python/cpython/issues/155460) and [PR #155464](https://github.com/python/cpython/pull/155464): allocating the zstd output from the recorded size. My earlier comment there has timings for multi-threaded zstd on Windows.\n- pyperformance seems to have no benchmark that exercises the compression modules; I'd be happy to try to contribute one, as a separate issue there.\n\n\u003cdetails>\n\u003csummary>bench_compression.py\u003c/summary>\n\n```python\n\"\"\"Output-buffer cost across the compression modules: zlib, gzip, bz2, lzma and zstd, one-shot\nfunctions and (de)compressor objects, decompression and compression, 1 to 64 MiB.\n\nFor each case: the best time per call, page faults per output page (Windows:\nGetProcessMemoryInfo(); elsewhere: getrusage()), and tracemalloc's peak during one call divided by\nthe output size. Standard library only (3.14 or later).\n\n    python bench_compression.py [filter]     # filter: substring of the case name\n\"\"\"\nimport bz2, gzip, lzma, mmap, platform, random, sys, time, tracemalloc, zlib\nfrom compression import zstd\n\nif sys.platform == \"win32\":  # the process's page fault counter, from GetProcessMemoryInfo()\n    import ctypes\n    from ctypes import wintypes\n\n    class Counters(ctypes.Structure):\n        _fields_ = [(\"cb\", wintypes.DWORD), (\"PageFaultCount\", wintypes.DWORD),\n                    (\"rest\", ctypes.c_size_t * 8)]\n\n    _k32 = ctypes.WinDLL(\"kernel32\")\n    _k32.GetCurrentProcess.restype = wintypes.HANDLE\n    _k32.K32GetProcessMemoryInfo.argtypes = [wintypes.HANDLE, ctypes.POINTER(Counters),\n                                             wintypes.DWORD]\n\n    def faults():\n        c = Counters(cb=ctypes.sizeof(Counters))\n        _k32.K32GetProcessMemoryInfo(_k32.GetCurrentProcess(), ctypes.byref(c), c.cb)\n        return c.PageFaultCount\nelse:  # minor and major page faults, from getrusage()\n    import resource\n\n    def faults():\n        r = resource.getrusage(resource.RUSAGE_SELF)\n        return r.ru_minflt + r.ru_majflt\n\nMiB = 1 \u003c\u003c 20\nRNG = random.Random(0)\nCHUNK = RNG.randbytes(MiB) + bytes(3 * MiB)          # \"mixed\": about 4:1, as in bench_presize.py\nRANDOM = RNG.randbytes(64 * MiB)                    # \"random\": incompressible, output ≈ input\n\ndef mixed(size):\n    return (CHUNK * (size // len(CHUNK) + 1))[:size]\n\n# (module, one-shot compress, one-shot decompress, decompressor object factory, slow codec)\nMODULES = [\n    (\"zlib\", zlib.compress, zlib.decompress, zlib.decompressobj, False),\n    (\"gzip\", gzip.compress, gzip.decompress, None, False),\n    (\"zstd\", zstd.compress, zstd.decompress, zstd.ZstdDecompressor, False),\n    (\"lzma\", lzma.compress, lzma.decompress, lzma.LZMADecompressor, True),\n    (\"bz2\", bz2.compress, bz2.decompress, bz2.BZ2Decompressor, True),\n]\n# Fast settings for compressing, so the codec takes less of the time than the output buffer.\nFAST = {\"zlib\": lambda d: zlib.compress(d, 1), \"gzip\": lambda d: gzip.compress(d, 1),\n        \"zstd\": lambda d: zstd.compress(d, 1), \"lzma\": lambda d: lzma.compress(d, preset=0),\n        \"bz2\": lambda d: bz2.compress(d, 1)}\n\ndef cases():\n    for name, compress, decompress, factory, slow in MODULES:\n        for m in (1, 4, 16, 64):\n            blob = compress(mixed(m * MiB))\n            yield f\"{name}.decompress(), mixed\", m, decompress, blob, slow\n            if factory is not None:\n                yield (f\"{name} decompressor object, mixed\", m,\n                       lambda b, f=factory: f().decompress(b), blob, slow)\n        for m in ((4, 16) if slow else (4, 16, 64)):\n            yield f\"{name}.compress(), random\", m, FAST[name], RANDOM[:m * MiB], True\n            if not slow:\n                yield f\"{name}.compress(), mixed\", m, FAST[name], mixed(m * MiB), True\n\ndef main():\n    flt = sys.argv[1] if len(sys.argv) > 1 else \"\"\n    print(sys.version.split()[0], \"|\", platform.platform(), \"| page\", mmap.PAGESIZE,\n          \"| zlib\", zlib.ZLIB_RUNTIME_VERSION, \"| libzstd\", zstd.zstd_version)\n    print(\"| case | MiB in | best ms | output MiB | page faults per output page | peak / output |\")\n    print(\"|---|---|---|---|---|---|\")\n    for label, m, func, arg, slow in cases():\n        if flt not in label:\n            continue\n        out = len(func(arg))\n        repeats = max(3, (64 if slow else 512) // m)\n        best = (float(\"inf\"), 0)\n        for _ in range(repeats):\n            f = faults()\n            t = time.perf_counter()\n            func(arg)\n            t = time.perf_counter() - t\n            best = min(best, (t, faults() - f))\n        tracemalloc.start()\n        func(arg)\n        peak = tracemalloc.get_traced_memory()[1]\n        tracemalloc.stop()\n        t, f = best\n        print(f\"| {label} | {m} | {t * 1e3:.4f} | {out / MiB:.2f} | {f * mmap.PAGESIZE / out:.2f} \"\n              f\"| {peak / out:.2f} |\", flush=True)\n\nmain()\n```\n\n\u003c/details>\n\n\u003cdetails>\n\u003csummary>footprint.py\u003c/summary>\n\n```python\n\"\"\"Process footprint over repeated decompression calls, macOS and Linux. One process per case.\n\n    python footprint.py make \u003cdir>                  # write the compressed inputs once\n    python footprint.py run \u003cdir> \u003cmodule> \u003cMiB>    # one case, prints one line\n\nSame data as bench_compression.py (\"mixed\"), compressed beforehand, so the measuring process\nholds only the compressed blob. After 10/50/100/200/400 calls, prints the growth since before\nthe first call, in multiples of the output size, of:\n- macOS: phys_footprint (task_info TASK_VM_INFO, what Activity Monitor shows as memory);\n- Linux: RssAnon (/proc/self/status, the anonymous memory the process holds);\nand tracemalloc's current traced memory (0.00 = no Python-level leak).\nCases: zstd and zlib at 1/4/16/64 MiB, as every other table. Windows: a port of this script\nreading PrivateUsage from GetProcessMemoryInfo().\n\"\"\"\nimport os, random, sys, tracemalloc, zlib\nfrom compression import zstd\n\nMiB = 1 \u003c\u003c 20\nMODS = {\"zlib\": zlib, \"zstd\": zstd}\nCASES = tuple((m, s) for m in (\"zstd\", \"zlib\") for s in (1, 4, 16, 64))\nCALLS = (10, 50, 100, 200, 400)\n\nif sys.platform == \"darwin\":\n    import ctypes\n    _libc = ctypes.CDLL(None)\n    _libc.mach_task_self.restype = ctypes.c_uint\n    METRIC = \"phys_footprint\"\n\n    def mem():\n        buf = (ctypes.c_uint8 * 1024)()\n        cnt = ctypes.c_uint(256)\n        if _libc.task_info(_libc.mach_task_self(), 22, buf, ctypes.byref(cnt)) != 0:\n            raise OSError(\"task_info failed\")\n        return int.from_bytes(bytes(buf[144:152]), \"little\")   # phys_footprint\nelse:\n    METRIC = \"RssAnon\"\n\n    def mem():\n        with open(\"/proc/self/status\") as f:\n            for line in f:\n                if line.startswith(\"RssAnon:\"):\n                    return int(line.split()[1]) * 1024\n        raise OSError(\"no RssAnon in /proc/self/status\")\n\nif sys.argv[1] == \"make\":\n    d = sys.argv[2]\n    os.makedirs(d, exist_ok=True)\n    chunk = random.Random(0).randbytes(MiB) + bytes(3 * MiB)\n    for name, m in CASES:\n        data = (chunk * (m // 4 + 1))[:m * MiB]\n        with open(os.path.join(d, f\"{name}-{m}.bin\"), \"wb\") as f:\n            f.write(MODS[name].compress(data))\nelse:\n    d, name, m = sys.argv[2], sys.argv[3], int(sys.argv[4])\n    with open(os.path.join(d, f\"{name}-{m}.bin\"), \"rb\") as f:\n        blob = f.read()\n    dec = MODS[name].decompress\n    n = m * MiB\n    tracemalloc.start()\n    t0 = tracemalloc.get_traced_memory()[0]\n    base = mem()\n    done, row = 0, []\n    for c in CALLS:\n        while done \u003c c:\n            out = dec(blob)\n            assert len(out) == n\n            del out\n            done += 1\n        traced = tracemalloc.get_traced_memory()[0] - t0\n        row.append(f\"{c}: {(mem() - base) / n:.1f} (traced {traced / n:.2f})\")\n    print(f\"| {sys.version.split()[0]} | {name} {m} MiB | {METRIC} | \" + \" | \".join(row) + \" |\",\n          flush=True)\n```\n\n\u003c/details>\n\n\n### CPython versions tested on:\n\n3.14, 3.15, CPython main branch\n\n### Operating systems tested on:\n\nWindows, macOS, Linux\n\n\u003c!-- gh-linked-prs -->\n### Linked PRs\n* gh-161482\n\u003c!-- /gh-linked-prs -->\n","author":{"url":"https://github.com/maikschulze","@type":"Person","name":"maikschulze"},"datePublished":"2026-11-06T11:17:19Z","url":"https://github.com/python/cpython/issues/161478"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:0f50a5a8-7a3f-b2be-10fa-97b16bec7bb4
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-idDBF0:158DF1:13643B3:199FDF3:6B01EE12
html-safe-nonce7722cd7aea6576fcd789bb36b1bb3d63756763395a383a48efc84e4a3e3259fe
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJEQkYwOjE1OERGMToxMzY0M0IzOjE5OUZERjM6NkIwMUVFMTIiLCJ2aXNpdG9yX2lkIjoiNjk1NTEyNDM3MjgyNjg3NzQ1OCIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmac25d405429d3a918e366eeb8cd1793166bd91cbcfdc8f6c86c159f30ccd50ee1b
hovercard-subject-tagissue:6241686425
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/161478/issue_layout
twitter:imagehttps://opengraph.githubassets.com/a3d93605dddf54339a2c23805198c038551f1b3daa0dc4692fb2e516d108b18a/python/cpython/issues/161478
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/a3d93605dddf54339a2c23805198c038551f1b3daa0dc4692fb2e516d108b18a/python/cpython/issues/161478
og:image:altBug report Since 3.15, decompressing a few MiB or more is markedly slower on macOS and Windows than in 3.14: 1.5 to 3 times for zstd, up to 1.5 times for zlib, gzip and lzma, and not measurably for...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernamemaikschulze
hostnamegithub.com
expected-hostnamegithub.com
None87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456
turbo-cache-controlno-preview
go-importgithub.com/python/cpython git https://github.com/python/cpython.git
octolytics-dimension-user_id1525981
octolytics-dimension-user_loginpython
octolytics-dimension-repository_id81598961
octolytics-dimension-repository_nwopython/cpython
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id81598961
octolytics-dimension-repository_network_root_nwopython/cpython
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release423dbb54c5e7bfecc838eb387c1435eb3d0e760d
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/python/cpython/issues/161478#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fissues%2F161478
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fissues%2F161478
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=python%2Fcpython
Reloadhttps://github.com/python/cpython/issues/161478
Reloadhttps://github.com/python/cpython/issues/161478
Reloadhttps://github.com/python/cpython/issues/161478
pythonhttps://github.com/python
cpythonhttps://github.com/python/cpython
Codehttps://github.com/python/cpython
Issues5k+ (5k+)https://github.com/python/cpython/issues
Pull requests2.4k (2.4k)https://github.com/python/cpython/pulls
Actionshttps://github.com/python/cpython/actions
Projectshttps://github.com/python/cpython/projects
Security and qualityhttps://github.com/python/cpython/security
Insightshttps://github.com/python/cpython/insights
https://github.com/python/cpython
New issuehttps://github.com/login?return_to=https://github.com/python/cpython/issues/161478
New issuehttps://github.com/login?return_to=https://github.com/python/cpython/issues/161478
#161482https://github.com/python/cpython/pull/161482
zlib, gzip, lzma and zstd in 3.15: decompressing large outputs is slower on macOS and Windows, and the process footprint grows on macOS (bisected to f262297)https://github.com/python/cpython/issues/161478#top
#161482https://github.com/python/cpython/pull/161482
extension-modulesC modules in the Modules dirhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22extension-modules%22
performancePerformance or resource usagehttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22performance%22
type-bugAn unexpected behavior, bug, or errorhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22type-bug%22
https://github.com/maikschulze
maikschulzehttps://github.com/maikschulze
on Nov 6, 2026https://github.com/python/cpython/issues/161478#issue-6241686425
f262297https://github.com/python/cpython/commit/f262297d525e87906c5e4ab28e80284189641c9e
gh-139877https://github.com/python/cpython/issues/139877
python-zstandardhttps://github.com/indygreg/python-zstandard
compression.zstdhttps://docs.python.org/3.14/library/compression.zstd.html
7054b71https://github.com/python/cpython/commit/7054b71e1547fa575fd243916dad5a5fbc65b238
f262297https://github.com/python/cpython/commit/f262297d525e87906c5e4ab28e80284189641c9e
f262297https://github.com/python/cpython/commit/f262297d525e87906c5e4ab28e80284189641c9e
gh-139877https://github.com/python/cpython/issues/139877
PR #139976https://github.com/python/cpython/pull/139976
4044255https://github.com/python/cpython/commit/404425575c68bef9d2f042710fc713134d04c23f
f262297https://github.com/python/cpython/commit/f262297d525e87906c5e4ab28e80284189641c9e
f262297https://github.com/python/cpython/commit/f262297d525e87906c5e4ab28e80284189641c9e
f262297https://github.com/python/cpython/commit/f262297d525e87906c5e4ab28e80284189641c9e
f262297https://github.com/python/cpython/commit/f262297d525e87906c5e4ab28e80284189641c9e
over-allocates by 50%https://github.com/python/cpython/blob/7054b71e1547fa575fd243916dad5a5fbc65b238/Objects/bytesobject.c#L3823-L3829
gh-139877https://github.com/python/cpython/issues/139877
PR #139976https://github.com/python/cpython/pull/139976
gh-155460https://github.com/python/cpython/issues/155460
PR #155464https://github.com/python/cpython/pull/155464
gh-161478: Collect large compression outputs in blocks again, except with glibc #161482https://github.com/python/cpython/pull/161482
maikschulzehttps://github.com/maikschulze
type-bugAn unexpected behavior, bug, or errorhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22type-bug%22
on Nov 6, 2026https://github.com/python/cpython/issues/161478#event-35341632371
maikschulzehttps://github.com/maikschulze
on Nov 6, 2026https://github.com/python/cpython/issues/161478#event-35341830608
pythongh-161478: Collect large compression outputs in blocks again, e…https://github.com/maikschulze/cpython/commit/cceb7826fc31848a4675533db497583d0ca4a825
pythongh-161478https://github.com/python/cpython/issues/161478
cceb782https://github.com/maikschulze/cpython/commit/cceb7826fc31848a4675533db497583d0ca4a825
NekoAsakurahttps://github.com/NekoAsakura
performancePerformance or resource usagehttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22performance%22
extension-modulesC modules in the Modules dirhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22extension-modules%22
on Nov 6, 2026https://github.com/python/cpython/issues/161478#event-35347499701
https://github.com/rhpvorderman
rhpvordermanhttps://github.com/rhpvorderman
on Nov 9, 2026https://github.com/python/cpython/issues/161478#issuecomment-6735532889
maikschulzehttps://github.com/maikschulze
on Nov 9, 2026https://github.com/python/cpython/issues/161478#event-35576260116
https://github.com/maikschulze
maikschulzehttps://github.com/maikschulze
on Nov 9, 2026https://github.com/python/cpython/issues/161478#issuecomment-6742604328
@rhpvordermanhttps://github.com/rhpvorderman
#161482https://github.com/python/cpython/pull/161482
https://github.com/vstinner
vstinnerhttps://github.com/vstinner
on Nov 9, 2026https://github.com/python/cpython/issues/161478#issuecomment-6745393903
https://github.com/maikschulze
maikschulzehttps://github.com/maikschulze
on Nov 10, 2026https://github.com/python/cpython/issues/161478#issuecomment-6769244313
@vstinnerhttps://github.com/vstinner
Sign up for freehttps://github.com/signup?return_to=https://github.com/python/cpython/issues/161478
Sign in to commenthttps://github.com/login?return_to=https://github.com/python/cpython/issues/161478
extension-modulesC modules in the Modules dirhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22extension-modules%22
performancePerformance or resource usagehttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22performance%22
type-bugAn unexpected behavior, bug, or errorhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22type-bug%22
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.