Title: zlib, gzip, lzma and zstd in 3.15: decompressing large outputs is slower on macOS and Windows, and the process footprint grows on macOS (bisected to f262297) · Issue #161478 · python/cpython · GitHub
Open Graph Title: zlib, gzip, lzma and zstd in 3.15: decompressing large outputs is slower on macOS and Windows, and the process footprint grows on macOS (bisected to f262297) · Issue #161478 · python/cpython
X Title: zlib, gzip, lzma and zstd in 3.15: decompressing large outputs is slower on macOS and Windows, and the process footprint grows on macOS (bisected to f262297) · Issue #161478 · python/cpython
Description: Bug report Since 3.15, decompressing a few MiB or more is markedly slower on macOS and Windows than in 3.14: 1.5 to 3 times for zstd, up to 1.5 times for zlib, gzip and lzma, and not measurably for bz2. Linux is unaffected. On macOS, the...
Open Graph Description: Bug report Since 3.15, decompressing a few MiB or more is markedly slower on macOS and Windows than in 3.14: 1.5 to 3 times for zstd, up to 1.5 times for zlib, gzip and lzma, and not measurably for...
X Description: Bug report Since 3.15, decompressing a few MiB or more is markedly slower on macOS and Windows than in 3.14: 1.5 to 3 times for zstd, up to 1.5 times for zlib, gzip and lzma, and not measurably for...
Opengraph URL: https://github.com/python/cpython/issues/161478
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"zlib, gzip, lzma and zstd in 3.15: decompressing large outputs is slower on macOS and Windows, and the process footprint grows on macOS (bisected to f262297)","articleBody":"### Bug report\n\nSince 3.15, decompressing a few MiB or more is markedly slower on macOS and Windows than in 3.14: 1.5 to 3 times for zstd, up to 1.5 times for zlib, gzip and lzma, and not measurably for bz2. Linux is unaffected. On macOS, the process also accumulates memory call after call: after a loop of 64 MiB decompressions, its footprint has grown by 3.4 GiB, against 134 MiB in 3.14. Both effects bisect to [f262297](https://github.com/python/cpython/commit/f262297d525e87906c5e4ab28e80284189641c9e) ([gh-139877](https://github.com/python/cpython/issues/139877)), which moved the output buffer that zlib, bz2, lzma and zstd share to `PyBytesWriter`. The tables below have the numbers.\n\nWe found it while moving a package manager from Python 3.9 with [python-zstandard](https://github.com/indygreg/python-zstandard) to the built-in [`compression.zstd`](https://docs.python.org/3.14/library/compression.zstd.html).\n\n**Reproducer.** The script decompresses 1, 4, 16 and 64 MiB with `zstd.decompress()` and `zlib.decompress()` and prints the best time per call. The data is 1/4 random bytes and 3/4 zeros, so it compresses about 4:1. Only the standard library is needed (3.14 or later):\n\n```python\nimport random\nimport sys\nimport timeit\nimport zlib\nfrom compression import zstd\n\nMiB = 1 \u003c\u003c 20\nchunk = random.Random(0).randbytes(MiB) + bytes(3 * MiB) # compresses about 4:1\n\nfor name, module in ((\"zstd\", zstd), (\"zlib\", zlib)):\n for size in (1, 4, 16, 64):\n blob = module.compress((chunk * 16)[:size * MiB])\n best = min(timeit.repeat(lambda: module.decompress(blob), number=1, repeat=max(5, 512 // size)))\n print(f\"{sys.version.split()[0]} {name}: {size:2} MiB in {best * 1e3:7.3f} ms, {size * MiB / best / 1e9:5.1f} GB/s\")\n```\n\n**Setup.** 3.14.10 and 3.15.2 as distributed, the bisect below from source:\n- **Windows 11 x64:** Ryzen 9 7950X, 2 × 32 GB DDR5-5600; python-build-standalone builds (via uv).\n- **macOS 26.7 arm64:** Apple M4 Pro, 48 GB; python-build-standalone builds (via uv); both link zlib 1.2.12.\n- **Linux arm64:** the official `python:3.14-slim-trixie` and `python:3.15-slim-trixie` images, in Docker Desktop's VM on the same Mac.\n\nlibzstd is 1.5.7 everywhere.\n\n**Results.** The tables come from `bench_compression.py` (in the collapsed block at the end of this report), which runs the reproducer's measurement for every module, one-shot and with the decompressor objects, and also counts page faults and peak memory. Each host ran it three times per interpreter, alternating, best of three. Best time per call, 3.14.10 → 3.15.2:\n\n| `zstd.decompress()` | Windows x64 | macOS arm64 | Linux arm64 |\n|---|---:|---:|---:|\n| 1 MiB | 0.393 → 0.290 ms | 0.036 → 0.030 ms | 0.050 → 0.043 ms |\n| 4 MiB | 1.25 → 1.11 ms | 0.125 → **0.288 ms** | 0.156 → 0.122 ms |\n| 16 MiB | 4.34 → **6.51 ms** | 0.618 → **1.78 ms** | 1.82 → 0.867 ms |\n| 64 MiB | 16.7 → **27.2 ms** | 2.80 → **8.69 ms** | 4.95 → 3.65 ms |\n\n| `zlib.decompress()` | Windows x64 | macOS arm64 | Linux arm64 |\n|---|---:|---:|---:|\n| 1 MiB | 0.283 → 0.165 ms | 0.069 → 0.059 ms | 0.271 → 0.264 ms |\n| 4 MiB | 1.02 → 0.840 ms | 0.666 → **0.891 ms** | 1.54 → 1.58 ms |\n| 16 MiB | 4.25 → **6.24 ms** | 3.00 → **3.58 ms** | 10.8 → 9.50 ms |\n| 64 MiB | 17.6 → **26.4 ms** | 12.4 → **14.3 ms** | 40.6 → 42.5 ms |\n\n| time in 3.15.2 relative to 3.14.10 (+ is slower; bold: 15% or more) | 1 MiB | 4 MiB | 16 MiB | 64 MiB |\n|---|---:|---:|---:|---:|\n| Windows x64, `zstd` | -26% | -11% | **+50%** | **+62%** |\n| Windows x64, `gzip` | -24% | -18% | **+40%** | **+48%** |\n| Windows x64, `zlib` | -42% | -17% | **+47%** | **+50%** |\n| Windows x64, `lzma` | -20% | +1% | **+18%** | **+39%** |\n| Windows x64, `bz2` | +2% | +1% | +3% | +4% |\n| macOS arm64, `zstd` | -16% | **+130%** | **+188%** | **+210%** |\n| macOS arm64, `gzip` | -18% | **+26%** | **+42%** | **+44%** |\n| macOS arm64, `zlib` | -14% | **+34%** | **+19%** | **+16%** |\n| macOS arm64, `lzma` | -1% | +1% | -1% | +7% |\n| macOS arm64, `bz2` | 0% | 0% | +1% | +1% |\n| Linux arm64, `zstd` | -14% | -22% | -52% | -26% |\n| Linux arm64, `gzip` | -7% | -4% | -11% | +1% |\n| Linux arm64, `zlib` | -2% | +3% | -12% | +5% |\n| Linux arm64, `lzma` | -2% | 0% | -10% | -1% |\n| Linux arm64, `bz2` | +3% | +3% | +5% | +4% |\n\n- **The pattern is the same on macOS and Windows:** the slowdown starts between 1 and 4 MiB of output on macOS, between 4 and 16 MiB on Windows, and grows with the size. It is largest for zstd, whose decoder is the fastest, so the buffer is the largest share of its time; bz2's decoder is slow enough to hide it. The decompressor objects (`zlib.decompressobj()`, `ZstdDecompressor`, …) show the same as the one-shot functions.\n- **On macOS, how much slower 3.15 is depends on what the process did before.** The same 64 MiB zstd call takes 5.4 ms in a fresh process (2.0× 3.14's), 6.9 ms after the reproducer's smaller cases (2.5×, what the reproducer above prints), and 8.7 ms after the script's zlib and gzip cases (3.1×), with one page fault per page in all three; 3.14.10 takes 2.7–2.8 ms in all three.\n- **At 1 MiB,** 3.15 is faster than 3.14 for zstd, gzip and zlib on all three hosts, and for lzma on Windows; bz2 is unchanged.\n- **Compression** with zlib, gzip, lzma and bz2 stays within 5% on every host. `zstd.compress()` is 1.2–1.4× faster on Windows, and 1.3× slower on macOS for incompressible data.\n- **Other zstd paths** go through the same buffer and regress the same way: `ZstdFile.read()` is 2.0× slower at 64 MiB on macOS.\n- **Still present on `main`:** a PGO build of [7054b71](https://github.com/python/cpython/commit/7054b71e1547fa575fd243916dad5a5fbc65b238) (2026-11-05) behaves like 3.15.2 in the same run of the script: zstd 64 MiB on macOS in 8.17 against 8.69 ms, zlib 14.5 against 14.3 ms.\n\n**Memory.** One fresh process per case calls `decompress()` repeatedly on the reproducer's data and discards the result (`footprint.py`, in the second collapsed block at the end). The tables show how much the process's memory has grown since before the first call: on macOS the physical footprint (`task_info(TASK_VM_INFO).phys_footprint`, what Activity Monitor shows), on Linux `RssAnon` from `/proc/self/status`, on Windows the private commit (`PrivateUsage` from `GetProcessMemoryInfo`, measured with a port of the same script; that run used 40 calls where the others use 50). 3.14.10's growth doesn't change with the number of calls, so it has one column; where 3.15.2's exceeds five outputs it's bold.\n\nmacOS arm64 (`phys_footprint`):\n\n| | 3.14.10, any number of calls | 3.15.2, 10 calls | 50 calls | 100 calls | 200 calls | 400 calls |\n|---|---:|---:|---:|---:|---:|---:|\n| `zstd.decompress()`, 1 MiB | 3.0 MiB | 2.1 MiB | 2.1 MiB | 4.2 MiB | 4.2 MiB | 4.2 MiB |\n| `zstd.decompress()`, 4 MiB | 12 MiB | **45 MiB** | **208 MiB** | **260 MiB** | **260 MiB** | **260 MiB** |\n| `zstd.decompress()`, 16 MiB | 50 MiB | **165 MiB** | **518 MiB** | **518 MiB** | **504 MiB** | **507 MiB** |\n| `zstd.decompress()`, 64 MiB | 134 MiB | **646 MiB** | **1.2 GiB** | **2.2 GiB** | **2.4 GiB** | **3.4 GiB** |\n| `zlib.decompress()`, 1 MiB | 2.0 MiB | 1.5 MiB | 1.5 MiB | 1.5 MiB | 1.5 MiB | 2.6 MiB |\n| `zlib.decompress()`, 4 MiB | 11 MiB | **42 MiB** | **206 MiB** | **262 MiB** | **261 MiB** | **261 MiB** |\n| `zlib.decompress()`, 16 MiB | 46 MiB | **163 MiB** | **645 MiB** | **979 MiB** | **965 MiB** | **920 MiB** |\n| `zlib.decompress()`, 64 MiB | 128 MiB | **640 MiB** | **800 MiB** | **1.2 GiB** | **1.2 GiB** | **1.2 GiB** |\n\nWindows x64 (private commit retained):\n\n| | 3.14.10, any number of calls | 3.15.2, 10 calls | 40 calls | 100 calls | 200 calls | 400 calls |\n|---|---:|---:|---:|---:|---:|---:|\n| `zstd.decompress()`, 1 MiB | 0.3 MiB | 0.7 MiB | 0.7 MiB | 0.7 MiB | 0.7 MiB | 0.7 MiB |\n| `zstd.decompress()`, 4 MiB | 0.3 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB |\n| `zstd.decompress()`, 16 MiB | 0.5 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB |\n| `zstd.decompress()`, 64 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB |\n| `zlib.decompress()`, 1 MiB | 0.0 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB |\n| `zlib.decompress()`, 4 MiB | 0.2 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB |\n| `zlib.decompress()`, 16 MiB | 0.2 MiB | 0.5 MiB | 0.5 MiB | 0.5 MiB | 0.5 MiB | 0.5 MiB |\n| `zlib.decompress()`, 64 MiB | 0.0 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB | 0.6 MiB |\n\nLinux arm64 (`RssAnon`):\n\n| | 3.14.10, any number of calls | 3.15.2, 10 calls | 50 calls | 100 calls | 200 calls | 400 calls |\n|---|---:|---:|---:|---:|---:|---:|\n| `zstd.decompress()`, 1 MiB | 0.0 MiB | 1.0 MiB | 1.0 MiB | 1.0 MiB | 1.0 MiB | 1.0 MiB |\n| `zstd.decompress()`, 4 MiB | 0.0 MiB | 3.6 MiB | 3.6 MiB | 3.6 MiB | 3.6 MiB | 3.6 MiB |\n| `zstd.decompress()`, 16 MiB | 0.0 MiB | 11 MiB | 11 MiB | 11 MiB | 11 MiB | 11 MiB |\n| `zstd.decompress()`, 64 MiB | 13 MiB | 6.4 MiB | 6.4 MiB | 6.4 MiB | 6.4 MiB | 6.4 MiB |\n| `zlib.decompress()`, 1 MiB | 0.0 MiB | 0.0 MiB | 0.0 MiB | 0.0 MiB | 0.0 MiB | 0.0 MiB |\n| `zlib.decompress()`, 4 MiB | 0.0 MiB | 1.2 MiB | 1.2 MiB | 1.2 MiB | 1.2 MiB | 1.2 MiB |\n| `zlib.decompress()`, 16 MiB | 0.0 MiB | 6.4 MiB | 6.4 MiB | 6.4 MiB | 6.4 MiB | 6.4 MiB |\n| `zlib.decompress()`, 64 MiB | 13 MiB | 0.0 MiB | 0.0 MiB | 0.0 MiB | 0.0 MiB | 0.0 MiB |\n\n- **On macOS, 3.15 grows with every call,** by about one output per call at first, until the allocator trims, at a bound that depends on the size: about 260 MiB for 4 MiB outputs, 0.5–1 GiB for 16 MiB, 1.2–3.4 GiB for 64 MiB. 3.14 stays at 2–3× the output. On Linux and Windows, neither version grows: glibc and the UCRT give large blocks back.\n- **Python doesn't leak:** tracemalloc's current traced memory is unchanged after the calls. The allocator keeps the freed memory, and macOS counts it against the process: it isn't marked reusable, and it isn't compressed. No other process was short of memory (65% of 48 GB free).\n- **Windows:** nothing accumulates, but the peak commit during a call is higher in 3.15 from 4 MiB (2.6–4.2× the output against 2.2–3.0× in 3.14), because `realloc` holds the old and the new block at once, plus the over-allocation; tracemalloc doesn't see that.\n\n### Bisected to f262297\n\n[f262297](https://github.com/python/cpython/commit/f262297d525e87906c5e4ab28e80284189641c9e) ([gh-139877](https://github.com/python/cpython/issues/139877), [PR #139976](https://github.com/python/cpython/pull/139976), \"Use PyBytesWriter in pycore_blocks_output_buffer.h\") against its parent [4044255](https://github.com/python/cpython/commit/404425575c68bef9d2f042710fc713134d04c23f), both built from source with PGO (and LTO on macOS), run alternately three times, best of three. \"Page faults per output page\" is the process's page faults during one call, divided by the output size in pages (16 KiB on macOS, 4 KiB elsewhere): 1.00 means the whole output was written into memory the process had never touched.\n\n| macOS arm64 | 4044255 | f262297 | page faults per output page |\n|---|---:|---:|---:|\n| `zstd.decompress()`, 1 MiB | 0.036 ms | 0.030 ms | 0.00 → 0.00 |\n| `zstd.decompress()`, 4 MiB | 0.127 ms | **0.279 ms** | 0.00 → 1.01 |\n| `zstd.decompress()`, 16 MiB | 0.624 ms | **1.81 ms** | 0.00 → 1.00 |\n| `zstd.decompress()`, 64 MiB | 2.72 ms | **8.93 ms** | 0.00 → 1.00 |\n| `zlib.decompress()`, 1 MiB | 0.069 ms | 0.064 ms | 0.00 → 0.00 |\n| `zlib.decompress()`, 4 MiB | 0.745 ms | **0.874 ms** | 0.00 → 1.01 |\n| `zlib.decompress()`, 16 MiB | 2.99 ms | **3.66 ms** | 0.00 → 1.00 |\n| `zlib.decompress()`, 64 MiB | 12.3 ms | **14.5 ms** | 0.00 → 1.00 |\n\nlzma gets 4–13% slower and bz2 about 1%, with the same rise in page faults. Across all modules, one-shot functions and decompressor objects alike, f262297 against its parent reproduces the 3.14.10 → 3.15.2 ratios on macOS within about 10% (zstd 64 MiB: 3.3× against 3.1×, zlib 1.18× against 1.15×).\n\n| Windows x64 | 4044255 | f262297 | page faults per output page |\n|---|---:|---:|---:|\n| `zstd.decompress()`, 1 MiB | 0.380 ms | 0.278 ms | 2.71 → 2.02 |\n| `zstd.decompress()`, 4 MiB | 1.27 ms | 1.10 ms | 2.50 → 2.08 |\n| `zstd.decompress()`, 16 MiB | 4.38 ms | **6.56 ms** | 2.13 → 3.28 |\n| `zstd.decompress()`, 64 MiB | 16.8 ms | **27.1 ms** | 2.03 → 3.33 |\n| `zlib.decompress()`, 1 MiB | 0.265 ms | 0.150 ms | 1.71 → 1.01 |\n| `zlib.decompress()`, 4 MiB | 1.01 ms | 0.832 ms | 1.94 → 1.51 |\n| `zlib.decompress()`, 16 MiB | 4.05 ms | **6.11 ms** | 1.99 → 3.13 |\n| `zlib.decompress()`, 64 MiB | 17.2 ms | **25.9 ms** | 2.00 → 3.29 |\n\nOn Windows too, the pair reproduces the 3.14.10 → 3.15.2 ratios for zlib, gzip and zstd: faster up to 4 MiB, 1.4–1.6× slower from 16 MiB. lzma gets up to 1.17× slower and bz2 stays within 2%, with the same rise in page faults.\n\n### Cause\n\nf262297 replaced the list of fixed-size blocks, joined at the end, with one `bytes` object grown with `PyBytesWriter_Grow()`. That saves the final copy, and its own benchmarks found decompression 10–30% faster. But each growth step is a `realloc`, and what that costs depends on the allocator:\n\n- **Linux (glibc):** `realloc` grows a large block in place (`mremap`). Page faults stay at 0.0–0.5 per output page on both versions (the VM uses transparent huge pages); zstd is faster in 3.15 and the other modules are within about 12%.\n- **macOS:** 3.14's blocks have fixed sizes, and `malloc` hands the previous call's blocks back, so the output is written into memory that's already mapped: 0.00 page faults per output page. In 3.15, `realloc` grows the buffer mostly in place, but into address space the process has never touched, so the decoder faults in every output page on every call: 1.00 per page from 4 MiB, in every module. (Measured by interposing `realloc` for a 64 MiB `zstd.decompress()`: seven large `realloc` calls per call, six in place, one move that copies 1.7 MiB.) The next call starts small and grows into new space again, so the freed block isn't reused, and the allocator keeps it: that's the footprint growth above. 3.15 copies less than 3.14, which joins its blocks with one full copy, and is still slower: faulting in fresh pages costs more than copying into mapped memory.\n- **Windows:** the process heap behind the UCRT's `malloc` returns blocks larger than about 512 KiB to the OS when they're freed, so 3.14 faults in fresh memory too, 2.0 per output page. 3.15 adds the `realloc` copies and [over-allocates by 50%](https://github.com/python/cpython/blob/7054b71e1547fa575fd243916dad5a5fbc65b238/Objects/bytesobject.c#L3823-L3829): an instrumented build showed every `realloc` from 1.34 MiB on moving the block and copying its whole capacity, including the tail that was never written (2, 8 and 20 MiB as the output passes 1.34, 5.34 and 13.34 MiB), for 3.3 faults per output page.\n\n### Fix\n\nCollect large outputs in blocks again and join them once, as 3.14 did, everywhere but on Linux, where `realloc` grows a block in place. I'll open a PR for this. With it, decompression is back at 3.14's speed within about 10% from 4 MiB on macOS and from 16 MiB on Windows, macOS sees no page faults, and its footprint over repeated calls stays at 2–3× the output. Two costs: 3.14's peak within one call, 2.2–2.9× the output instead of 1.5–2.3×, because the blocks and the joined result exist together at the end; and on Windows at 4 MiB, 3.14's speed too, where 3.15 had been about 20% faster.\n\n### Related\n\n- [gh-139877](https://github.com/python/cpython/issues/139877) and [PR #139976](https://github.com/python/cpython/pull/139976): the move to `PyBytesWriter`.\n- [gh-155460](https://github.com/python/cpython/issues/155460) and [PR #155464](https://github.com/python/cpython/pull/155464): allocating the zstd output from the recorded size. My earlier comment there has timings for multi-threaded zstd on Windows.\n- pyperformance seems to have no benchmark that exercises the compression modules; I'd be happy to try to contribute one, as a separate issue there.\n\n\u003cdetails>\n\u003csummary>bench_compression.py\u003c/summary>\n\n```python\n\"\"\"Output-buffer cost across the compression modules: zlib, gzip, bz2, lzma and zstd, one-shot\nfunctions and (de)compressor objects, decompression and compression, 1 to 64 MiB.\n\nFor each case: the best time per call, page faults per output page (Windows:\nGetProcessMemoryInfo(); elsewhere: getrusage()), and tracemalloc's peak during one call divided by\nthe output size. Standard library only (3.14 or later).\n\n python bench_compression.py [filter] # filter: substring of the case name\n\"\"\"\nimport bz2, gzip, lzma, mmap, platform, random, sys, time, tracemalloc, zlib\nfrom compression import zstd\n\nif sys.platform == \"win32\": # the process's page fault counter, from GetProcessMemoryInfo()\n import ctypes\n from ctypes import wintypes\n\n class Counters(ctypes.Structure):\n _fields_ = [(\"cb\", wintypes.DWORD), (\"PageFaultCount\", wintypes.DWORD),\n (\"rest\", ctypes.c_size_t * 8)]\n\n _k32 = ctypes.WinDLL(\"kernel32\")\n _k32.GetCurrentProcess.restype = wintypes.HANDLE\n _k32.K32GetProcessMemoryInfo.argtypes = [wintypes.HANDLE, ctypes.POINTER(Counters),\n wintypes.DWORD]\n\n def faults():\n c = Counters(cb=ctypes.sizeof(Counters))\n _k32.K32GetProcessMemoryInfo(_k32.GetCurrentProcess(), ctypes.byref(c), c.cb)\n return c.PageFaultCount\nelse: # minor and major page faults, from getrusage()\n import resource\n\n def faults():\n r = resource.getrusage(resource.RUSAGE_SELF)\n return r.ru_minflt + r.ru_majflt\n\nMiB = 1 \u003c\u003c 20\nRNG = random.Random(0)\nCHUNK = RNG.randbytes(MiB) + bytes(3 * MiB) # \"mixed\": about 4:1, as in bench_presize.py\nRANDOM = RNG.randbytes(64 * MiB) # \"random\": incompressible, output ≈ input\n\ndef mixed(size):\n return (CHUNK * (size // len(CHUNK) + 1))[:size]\n\n# (module, one-shot compress, one-shot decompress, decompressor object factory, slow codec)\nMODULES = [\n (\"zlib\", zlib.compress, zlib.decompress, zlib.decompressobj, False),\n (\"gzip\", gzip.compress, gzip.decompress, None, False),\n (\"zstd\", zstd.compress, zstd.decompress, zstd.ZstdDecompressor, False),\n (\"lzma\", lzma.compress, lzma.decompress, lzma.LZMADecompressor, True),\n (\"bz2\", bz2.compress, bz2.decompress, bz2.BZ2Decompressor, True),\n]\n# Fast settings for compressing, so the codec takes less of the time than the output buffer.\nFAST = {\"zlib\": lambda d: zlib.compress(d, 1), \"gzip\": lambda d: gzip.compress(d, 1),\n \"zstd\": lambda d: zstd.compress(d, 1), \"lzma\": lambda d: lzma.compress(d, preset=0),\n \"bz2\": lambda d: bz2.compress(d, 1)}\n\ndef cases():\n for name, compress, decompress, factory, slow in MODULES:\n for m in (1, 4, 16, 64):\n blob = compress(mixed(m * MiB))\n yield f\"{name}.decompress(), mixed\", m, decompress, blob, slow\n if factory is not None:\n yield (f\"{name} decompressor object, mixed\", m,\n lambda b, f=factory: f().decompress(b), blob, slow)\n for m in ((4, 16) if slow else (4, 16, 64)):\n yield f\"{name}.compress(), random\", m, FAST[name], RANDOM[:m * MiB], True\n if not slow:\n yield f\"{name}.compress(), mixed\", m, FAST[name], mixed(m * MiB), True\n\ndef main():\n flt = sys.argv[1] if len(sys.argv) > 1 else \"\"\n print(sys.version.split()[0], \"|\", platform.platform(), \"| page\", mmap.PAGESIZE,\n \"| zlib\", zlib.ZLIB_RUNTIME_VERSION, \"| libzstd\", zstd.zstd_version)\n print(\"| case | MiB in | best ms | output MiB | page faults per output page | peak / output |\")\n print(\"|---|---|---|---|---|---|\")\n for label, m, func, arg, slow in cases():\n if flt not in label:\n continue\n out = len(func(arg))\n repeats = max(3, (64 if slow else 512) // m)\n best = (float(\"inf\"), 0)\n for _ in range(repeats):\n f = faults()\n t = time.perf_counter()\n func(arg)\n t = time.perf_counter() - t\n best = min(best, (t, faults() - f))\n tracemalloc.start()\n func(arg)\n peak = tracemalloc.get_traced_memory()[1]\n tracemalloc.stop()\n t, f = best\n print(f\"| {label} | {m} | {t * 1e3:.4f} | {out / MiB:.2f} | {f * mmap.PAGESIZE / out:.2f} \"\n f\"| {peak / out:.2f} |\", flush=True)\n\nmain()\n```\n\n\u003c/details>\n\n\u003cdetails>\n\u003csummary>footprint.py\u003c/summary>\n\n```python\n\"\"\"Process footprint over repeated decompression calls, macOS and Linux. One process per case.\n\n python footprint.py make \u003cdir> # write the compressed inputs once\n python footprint.py run \u003cdir> \u003cmodule> \u003cMiB> # one case, prints one line\n\nSame data as bench_compression.py (\"mixed\"), compressed beforehand, so the measuring process\nholds only the compressed blob. After 10/50/100/200/400 calls, prints the growth since before\nthe first call, in multiples of the output size, of:\n- macOS: phys_footprint (task_info TASK_VM_INFO, what Activity Monitor shows as memory);\n- Linux: RssAnon (/proc/self/status, the anonymous memory the process holds);\nand tracemalloc's current traced memory (0.00 = no Python-level leak).\nCases: zstd and zlib at 1/4/16/64 MiB, as every other table. Windows: a port of this script\nreading PrivateUsage from GetProcessMemoryInfo().\n\"\"\"\nimport os, random, sys, tracemalloc, zlib\nfrom compression import zstd\n\nMiB = 1 \u003c\u003c 20\nMODS = {\"zlib\": zlib, \"zstd\": zstd}\nCASES = tuple((m, s) for m in (\"zstd\", \"zlib\") for s in (1, 4, 16, 64))\nCALLS = (10, 50, 100, 200, 400)\n\nif sys.platform == \"darwin\":\n import ctypes\n _libc = ctypes.CDLL(None)\n _libc.mach_task_self.restype = ctypes.c_uint\n METRIC = \"phys_footprint\"\n\n def mem():\n buf = (ctypes.c_uint8 * 1024)()\n cnt = ctypes.c_uint(256)\n if _libc.task_info(_libc.mach_task_self(), 22, buf, ctypes.byref(cnt)) != 0:\n raise OSError(\"task_info failed\")\n return int.from_bytes(bytes(buf[144:152]), \"little\") # phys_footprint\nelse:\n METRIC = \"RssAnon\"\n\n def mem():\n with open(\"/proc/self/status\") as f:\n for line in f:\n if line.startswith(\"RssAnon:\"):\n return int(line.split()[1]) * 1024\n raise OSError(\"no RssAnon in /proc/self/status\")\n\nif sys.argv[1] == \"make\":\n d = sys.argv[2]\n os.makedirs(d, exist_ok=True)\n chunk = random.Random(0).randbytes(MiB) + bytes(3 * MiB)\n for name, m in CASES:\n data = (chunk * (m // 4 + 1))[:m * MiB]\n with open(os.path.join(d, f\"{name}-{m}.bin\"), \"wb\") as f:\n f.write(MODS[name].compress(data))\nelse:\n d, name, m = sys.argv[2], sys.argv[3], int(sys.argv[4])\n with open(os.path.join(d, f\"{name}-{m}.bin\"), \"rb\") as f:\n blob = f.read()\n dec = MODS[name].decompress\n n = m * MiB\n tracemalloc.start()\n t0 = tracemalloc.get_traced_memory()[0]\n base = mem()\n done, row = 0, []\n for c in CALLS:\n while done \u003c c:\n out = dec(blob)\n assert len(out) == n\n del out\n done += 1\n traced = tracemalloc.get_traced_memory()[0] - t0\n row.append(f\"{c}: {(mem() - base) / n:.1f} (traced {traced / n:.2f})\")\n print(f\"| {sys.version.split()[0]} | {name} {m} MiB | {METRIC} | \" + \" | \".join(row) + \" |\",\n flush=True)\n```\n\n\u003c/details>\n\n\n### CPython versions tested on:\n\n3.14, 3.15, CPython main branch\n\n### Operating systems tested on:\n\nWindows, macOS, Linux\n\n\u003c!-- gh-linked-prs -->\n### Linked PRs\n* gh-161482\n\u003c!-- /gh-linked-prs -->\n","author":{"url":"https://github.com/maikschulze","@type":"Person","name":"maikschulze"},"datePublished":"2026-11-06T11:17:19Z","url":"https://github.com/python/cpython/issues/161478"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:0f50a5a8-7a3f-b2be-10fa-97b16bec7bb4 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | DBF0:158DF1:13643B3:199FDF3:6B01EE12 |
| html-safe-nonce | 7722cd7aea6576fcd789bb36b1bb3d63756763395a383a48efc84e4a3e3259fe |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJEQkYwOjE1OERGMToxMzY0M0IzOjE5OUZERjM6NkIwMUVFMTIiLCJ2aXNpdG9yX2lkIjoiNjk1NTEyNDM3MjgyNjg3NzQ1OCIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 25d405429d3a918e366eeb8cd1793166bd91cbcfdc8f6c86c159f30ccd50ee1b |
| hovercard-subject-tag | issue:6241686425 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/161478/issue_layout |
| twitter:image | https://opengraph.githubassets.com/a3d93605dddf54339a2c23805198c038551f1b3daa0dc4692fb2e516d108b18a/python/cpython/issues/161478 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/a3d93605dddf54339a2c23805198c038551f1b3daa0dc4692fb2e516d108b18a/python/cpython/issues/161478 |
| og:image:alt | Bug report Since 3.15, decompressing a few MiB or more is markedly slower on macOS and Windows than in 3.14: 1.5 to 3 times for zstd, up to 1.5 times for zlib, gzip and lzma, and not measurably for... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | maikschulze |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width