René's URL Explorer Experiment


Title: Pre-allocate the output buffer in compression.zstd decompression from the frame's recorded content size · Issue #155460 · python/cpython · GitHub

Open Graph Title: Pre-allocate the output buffer in compression.zstd decompression from the frame's recorded content size · Issue #155460 · python/cpython

X Title: Pre-allocate the output buffer in compression.zstd decompression from the frame's recorded content size · Issue #155460 · python/cpython

Description: Feature or enhancement (performance) Proposal compression.zstd decompression always builds its output through the growing output buffer (initial 32 KiB, then progressively larger growth steps, finished with a resize to the actual size), ...

Open Graph Description: Feature or enhancement (performance) Proposal compression.zstd decompression always builds its output through the growing output buffer (initial 32 KiB, then progressively larger growth steps, fini...

X Description: Feature or enhancement (performance) Proposal compression.zstd decompression always builds its output through the growing output buffer (initial 32 KiB, then progressively larger growth steps, fini...

Opengraph URL: https://github.com/python/cpython/issues/155460

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Pre-allocate the output buffer in compression.zstd decompression from the frame's recorded content size","articleBody":"# Feature or enhancement (performance)\n\n## Proposal\n\n`compression.zstd` decompression always builds its output through the growing\noutput buffer (initial 32 KiB, then progressively larger growth steps, finished\nwith a resize to the actual size), even though most zstd frames record the\nexact decompressed size in the frame header: the one-shot compression APIs\n(`zstd.compress()`, `ZSTD_compress2()`) write it by default.\n\n### Dealing with the growing buffer causes unnecessary work\n\nFor frames that record the size, `ZstdDecompressor` (and thereby\n`zstd.decompress()`) can read it via `ZSTD_getFrameContentSize()` and\n**allocate the output buffer exactly once**.\n\nThe exactly-filled buffer is then returned **without any resize or copy**.\n\n### The patched code is safe\n\nThis is only a sizing hint: decompression does not rely on it, so a\nhand-crafted header recording a wrong size gives exactly the same\nresults/exceptions as today (the buffer grows further if needed, or is\nshrunk to the actual size on finish).\n\nTwo guards keep hand-crafted headers from requesting large spurious\nallocations:\n\n- the recorded size is only trusted up to a fixed limit (1 GiB) — larger\n  outputs just keep using the growing buffer;\n- a zstd block cannot expand to more than 128 KiB from less than 4 bytes of\n  input, so recorded sizes claiming more than a 32768x expansion of the\n  available input are ignored as implausible.\n\n### The patched code is usually faster\n\n\n#### Benchmark setup\n\nMeasured with [pyperf](https://pyperf.readthedocs.io/) (mean +- standard\ndeviation over many worker processes). The corpus is deterministic and cached\non disk, so every worker process decompresses byte-identical input: a 16 MiB\nblob of random 16-byte words drawn from a 200-word dictionary, sliced or\nrepeated to the requested size. The full script is at the bottom of this post.\n\n| | Linux/x86-64 | macOS/arm64 |\n|---|---|---|\n| CPU | AMD Ryzen 5 8500GE | Apple M3 Pro |\n| OS / libc | Debian 13, glibc 2.41 | macOS 15.7.9 |\n| compiler | gcc 14.2 | Apple clang 17.0 |\n| libzstd | 1.5.7 | 1.5.7 |\n| conditions | idle, performance governor, pinned to one core | cannot be quiesced or pinned |\n| pyperf worker processes | 20 (default) | 10 |\n\nBoth builds are plain `./configure -q` (no PGO, no LTO, `OPT = -DNDEBUG -g -O3\n-Wall`), differing only in the patch. Note that essentially all of the time is\nspent inside libzstd, a shared library that CPython's own build options do not\naffect.\n\n`compression.zstd`'s default compression level is 3, so the tables below use\nlevel 3; levels 1 and 10 were measured too and change little.\n\n#### Constant decompressed size benchmarks\n\nLinux/x86-64, level 3:\n\n| decompressed size | main | patched | speedup |\n|---|---|---|---|\n| 64 KiB | 34.1 µs +- 576 ns | 32.2 µs +- 683 ns | +5.6% |\n| 256 KiB | 252 µs +- 4.05 µs | 123 µs +- 3.71 µs | +51.2% |\n| 1 MiB | 1.01 ms +- 7.22 µs | 516 µs +- 5.27 µs | +49.2% |\n| 2 MiB | 3.56 ms +- 35.2 µs | 1.04 ms +- 9.28 µs | +70.7% |\n| 3 MiB | 2.97 ms +- 38.4 µs | 1.58 ms +- 27.1 µs | +47.0% |\n| 4 MiB | 3.99 ms +- 67.8 µs | 2.11 ms +- 39.9 µs | +47.2% |\n| 5 MiB | 5.03 ms +- 85.7 µs | 2.64 ms +- 35.3 µs | +47.6% |\n| 6 MiB | 6.10 ms +- 73.3 µs | 3.17 ms +- 66.9 µs | +48.1% |\n| 8 MiB | 8.54 ms +- 147 µs | 4.22 ms +- 76.6 µs | +50.6% |\n| 12 MiB | 12.9 ms +- 201 µs | 6.33 ms +- 48.5 µs | +50.8% |\n| 16 MiB | 17.2 ms +- 320 µs | 8.51 ms +- 178 µs | +50.6% |\n| 32 MiB | 33.9 ms +- 627 µs | 33.7 ms +- 654 µs | +0.6% (*) |\n| 64 MiB | 68.2 ms +- 576 µs | 68.2 ms +- 997 µs | −0.1% (*) |\n| 128 MiB | 138 ms +- 1.86 ms | 138 ms +- 1.59 ms | +0.1% (*) |\n| 256 MiB | 278 ms +- 4.56 ms | 277 ms +- 2.61 ms | +0.5% (*) |\n\nmacOS/arm64, level 3:\n\n| decompressed size | main | patched | speedup |\n|---|---|---|---|\n| 64 KiB | 33.3 µs +- 1.61 µs | 32.8 µs +- 142 ns | +1.7% (*) |\n| 256 KiB | 132 µs +- 550 ns | 121 µs +- 9.32 µs | +8.5% |\n| 1 MiB | 563 µs +- 29.3 µs | 519 µs +- 27.4 µs | +7.7% (*) |\n| 2 MiB | 1.16 ms +- 83.9 µs | 1.09 ms +- 73.8 µs | +6.7% (*) |\n| 4 MiB | 2.33 ms +- 176 µs | 2.22 ms +- 100 µs | +4.7% (*) |\n| 8 MiB | 5.22 ms +- 227 µs | 4.20 ms +- 230 µs | +19.5% |\n| 16 MiB | 9.69 ms +- 622 µs | 8.55 ms +- 26.2 µs | +11.8% |\n| 32 MiB | 19.7 ms +- 1.32 ms | 18.3 ms +- 145 µs | +7.0% (*) |\n| 128 MiB | 79.3 ms +- 3.58 ms | 76.8 ms +- 4.23 ms | +3.1% (*) |\n\n(*) marks differences smaller than the combined standard deviations.\n\nSpeedup by compression level, Linux/x86-64 — the level makes little\ndifference:\n\n| decompressed size | level 1 | level 3 | level 10 |\n|---|---|---|---|\n| 64 KiB | +5.6% | +5.6% | +6.5% |\n| 256 KiB | +51.2% | +51.2% | +54.7% |\n| 1 MiB | +48.9% | +49.2% | +49.4% |\n| 2 MiB | +50.6% | +70.7% | +72.2% |\n| 3 MiB | +48.2% | +47.0% | +48.8% |\n| 4 MiB | +48.1% | +47.2% | +49.5% |\n| 5 MiB | +48.6% | +47.6% | +49.0% |\n| 6 MiB | +48.6% | +48.1% | +50.2% |\n| 8 MiB | +51.1% | +50.6% | +52.5% |\n| 12 MiB | +51.1% | +50.8% | +52.3% |\n| 16 MiB | +51.4% | +50.6% | +52.1% |\n| 32–256 MiB | +0.8…+1.6% (*) | −0.1…+0.6% (*) | +0.4…+1.2% (*) |\n\n#### Varying decompressed size benchmarks\n\nA workload closer to real chunk-oriented use: 1000 frames whose decompressed\nsizes are drawn from a normal distribution in log2(size), centred on 2 MiB and\nspanning 512 KiB to 8 MiB at +-3 sigma (2269 MiB of plaintext in total). One\niteration decompresses all 1000 frames.\n\n| level | | main | patched | speedup |\n|---|---|---|---|---|\n| 1 | Linux | 1.29 s +- 20.6 ms | 1.14 s +- 5.78 ms | +12.2% |\n| 3 | Linux | 1.44 s +- 9.42 ms | 1.28 s +- 9.55 ms | +10.9% |\n| 10 | Linux | 1.27 s +- 9.07 ms | 1.11 s +- 23.7 ms | +12.7% |\n| 1 | macOS | 1.21 s +- 81.0 ms | 1.23 s +- 68.7 ms | −1.5% (*) |\n| 3 | macOS | 1.32 s +- 83.4 ms | 1.27 s +- 81.1 ms | +3.5% (*) |\n| 10 | macOS | 1.21 s +- 128 ms | 1.04 s +- 81.6 ms | +14.2% (*) |\n\nAll three macOS rows are within one standard deviation, so that machine cannot\nresolve this workload — its standard deviations are 5–10% of the mean, against\n0.5–2% on the pinned, idle Linux machine. The Linux rows are consistent at\n+11–13%.\n\nBecause the sizes vary from call to call, this workload gives the allocator\nmuch less opportunity to reuse a buffer of exactly the right size, so it is the\nmore conservative measurement of the two.\n\n#### What the numbers mean\n\nAvoiding the growth-path reallocations and the final shrink is only part of it.\nA full-size output buffer also lets libzstd decompress directly into it:\n`ZSTD_decompressStream` can skip its internal window buffer, and the copies\nthat go with it, when the remaining output buffer is large enough for the whole\nframe. The growing buffer never permits that.\n\nTwo things stand out in the tables above.\n\n**The gains are much larger on glibc than on macOS.** The patched build performs\nalmost identically on both machines (8 MiB: 4.22 ms on Linux, 4.20 ms on macOS)\n— it is the unpatched build that is far slower on Linux (8.54 ms vs 5.22 ms).\nThe reason is glibc's *dynamic* mmap threshold: the growing buffer's\nreallocations produce a succession of differently sized large blocks, and once\nthose exceed the threshold each one is mmap'd and munmap'd again per call, so\nevery call re-faults and re-zeroes the whole output. `perf` attributes 43–46% of\nthe unpatched build's samples to the kernel fault paths (`do_anonymous_page`,\n`kernel_init_pages`) from 6 MiB upwards, against 0.2–0.3% for the patched build;\nmeasured with `getrusage()`, the unpatched build takes 1712/4099/8195 minor\nfaults per call at 6/16/32 MiB where the patched build takes none. The patched\nbuild asks for the same exact size on every call, so its allocation is simply\nrecycled.\n\nA corollary worth stating explicitly: **the unpatched path's cost depends on the\nprocess's malloc history**, because anything else that frees a large block\nraises that threshold. A process that compresses before it decompresses (as a\nself-contained benchmark naturally does) is measurably faster in the unpatched\nbuild than a process that only ever decompresses — which is the real workload\nhere, since data usually arrives already compressed. At 1 MiB on Linux, level 1,\nthe unpatched build takes 952 µs when the frame is simply read from a file, but\n532 µs in a process that compressed it first, and 513 µs if\n`MALLOC_MMAP_THRESHOLD_` is raised by hand; the patched build takes 485–496 µs\nin all three cases. Earlier revisions of this post quoted the primed figures,\nwhich understated the difference. The benchmark script below decompresses frames\nread from disk.\n\n**The unpatched path is not monotonic in output size.** On Linux at level 3 it\nneeds 3.56 ms for 2 MiB but only 2.97 ms for 3 MiB, reproducibly, because 2 MiB\nlands just past a capacity step and pays a growth reallocation plus a large\nfinal shrink. The patched build is linear across the same range (1.04, 1.58,\n2.11 ms for 2, 3, 4 MiB).\n\nAt 32 MiB and above both builds exceed glibc's mmap threshold ceiling\n(`DEFAULT_MMAP_THRESHOLD_MAX`), so both fault in the whole output on every call;\nthat cost dominates and the remaining difference falls within the noise.\nRaising the thresholds so that neither build returns memory to the kernel\n(`MALLOC_MMAP_THRESHOLD_`/`MALLOC_TRIM_THRESHOLD_` = 512 MiB) removes that\neffect and leaves the allocator-independent part of the win: at 32 MiB,\n19.46 ms vs 15.52 ms (+20%).\n\n\u003cdetails>\n\u003csummary>Benchmark script\u003c/summary>\n\n```python\n#!/usr/bin/env python3\n\"\"\"Benchmark compression.zstd decompression (CPython gh-155460).\n\nMeasures ``zstd.decompress()`` with pyperf, on two workloads:\n\n* one frame of a fixed decompressed size, for a range of sizes;\n* a chunk-oriented workload: 1000 frames whose decompressed sizes are drawn\n  from a normal distribution in log2(size), centred on 2 MiB and spanning\n  512 KiB to 8 MiB at +-3 sigma (the endpoints are geometrically symmetric\n  around the target).  One iteration decompresses all 1000 frames.\n\nThe corpus is deterministic (fixed seeds) and cached on disk next to this\nscript, so every pyperf worker process decompresses byte-identical input:\na 16 MiB blob of random 16-byte words drawn from a 200-word dictionary,\nwhich compresses about 9:1 at level 1.\n\nRun it once per interpreter, then compare:\n\n    ./python bench_zstd_decompress.py --level 3 -o main.json\n    ../other/python bench_zstd_decompress.py --level 3 -o patched.json\n    ./python bench_zstd_decompress.py --compare main.json patched.json\n\nThe last command prints a table of mean +- standard deviation for both\nruns.  ``python -m pyperf compare_to main.json patched.json --table`` works\ntoo and adds pyperf's significance test.\n\nRequires pyperf (``pip install pyperf``) for both interpreters.\n\"\"\"\n\nimport math\nimport os\nimport random\nimport struct\nimport sys\n\nfrom compression import zstd\n\nCACHE_DIR = os.path.dirname(os.path.abspath(__file__))\nBLOB_SEED = 42\nCHUNK_SEED = 1234\nNCHUNKS = 1000\nMIN_CHUNK, TARGET_CHUNK, MAX_CHUNK = 512 \u003c\u003c 10, 2 \u003c\u003c 20, 8 \u003c\u003c 20\n\nSIZES = {\n    \"64K\": 1 \u003c\u003c 16, \"256K\": 1 \u003c\u003c 18, \"1M\": 1 \u003c\u003c 20, \"2M\": 2 \u003c\u003c 20,\n    \"3M\": 3 \u003c\u003c 20, \"4M\": 4 \u003c\u003c 20, \"5M\": 5 \u003c\u003c 20, \"6M\": 6 \u003c\u003c 20,\n    \"7M\": 7 \u003c\u003c 20, \"8M\": 8 \u003c\u003c 20, \"9M\": 9 \u003c\u003c 20, \"10M\": 10 \u003c\u003c 20,\n    \"11M\": 11 \u003c\u003c 20, \"12M\": 12 \u003c\u003c 20, \"13M\": 13 \u003c\u003c 20, \"14M\": 14 \u003c\u003c 20,\n    \"15M\": 15 \u003c\u003c 20, \"16M\": 16 \u003c\u003c 20, \"32M\": 32 \u003c\u003c 20, \"64M\": 64 \u003c\u003c 20,\n    \"128M\": 128 \u003c\u003c 20, \"256M\": 256 \u003c\u003c 20,\n}\nDEFAULT_SIZES = \"64K,256K,1M,2M,3M,4M,5M,6M,8M,12M,16M,32M,64M,128M,256M\"\n\n\ndef get_blob():\n    \"\"\"16 MiB of moderately compressible data (cached on disk).\"\"\"\n    path = os.path.join(CACHE_DIR, \"zstd_bench_blob.bin\")\n    if not os.path.exists(path):\n        rng = random.Random(BLOB_SEED)\n        words = [bytes(rng.randrange(256) for _ in range(16))\n                 for _ in range(200)]\n        blob = b\"\".join(rng.choice(words) for _ in range(1 \u003c\u003c 20))\n        _write_atomic(path, blob)\n    with open(path, \"rb\") as fp:\n        return fp.read()\n\n\ndef _write_atomic(path, data):\n    with open(path + \".tmp\", \"wb\") as fp:\n        fp.write(data)\n    os.replace(path + \".tmp\", path)\n\n\ndef get_frame(label, level):\n    \"\"\"One frame of the given decompressed size (cached on disk).\n\n    Cached so that a high compression level is not paid once per pyperf\n    worker process, and so that all workers use byte-identical input.\n    \"\"\"\n    path = os.path.join(CACHE_DIR, f\"zstd_bench_frame_{label}_l{level}.bin\")\n    if not os.path.exists(path):\n        size = SIZES[label]\n        blob = get_blob()\n        data = (blob * (size // len(blob) + 1))[:size]\n        frame = zstd.compress(data, level)\n        assert zstd.decompress(frame) == data, \"round-trip failed\"\n        del data, blob\n        _write_atomic(path, frame)\n    with open(path, \"rb\") as fp:\n        return fp.read()\n\n\ndef chunk_sizes():\n    \"\"\"1000 chunk sizes: normal in log2(size), 512 KiB..8 MiB, mode 2 MiB.\"\"\"\n    lo, mu, hi = (math.log2(MIN_CHUNK), math.log2(TARGET_CHUNK),\n                  math.log2(MAX_CHUNK))\n    sigma = (hi - mu) / 3.0\n    rng = random.Random(CHUNK_SEED)\n    sizes, offsets = [], []\n    while len(sizes) \u003c NCHUNKS:\n        x = rng.gauss(mu, sigma)\n        if lo \u003c= x \u003c= hi:\n            sizes.append(int(2 ** x))\n            offsets.append(rng.randrange(1 \u003c\u003c 24))\n    return sizes, offsets\n\n\ndef get_chunk_frames(level):\n    \"\"\"The 1000 compressed chunks (cached on disk, one file per level).\"\"\"\n    path = os.path.join(CACHE_DIR, f\"zstd_bench_chunks_l{level}.bin\")\n    if not os.path.exists(path):\n        pool = get_blob() * 2\n        sizes, offsets = chunk_sizes()\n        buf = []\n        for size, off in zip(sizes, offsets):\n            frame = zstd.compress(pool[off:off + size], level)\n            buf.append(struct.pack(\"\u003cI\", len(frame)))\n            buf.append(frame)\n        _write_atomic(path, b\"\".join(buf))\n    frames = []\n    with open(path, \"rb\") as fp:\n        while header := fp.read(4):\n            frames.append(fp.read(struct.unpack(\"\u003cI\", header)[0]))\n    return frames\n\n\ndef decompress_all(frames):\n    for frame in frames:\n        out = zstd.decompress(frame)\n    del out\n\n\ndef human_time(seconds):\n    for unit, scale in ((\"s\", 1), (\"ms\", 1e-3), (\"us\", 1e-6), (\"ns\", 1e-9)):\n        if seconds >= scale or unit == \"ns\":\n            return f\"{seconds / scale:.3g} {unit}\"\n\n\ndef compare(path_a, path_b):\n    \"\"\"Print a human-readable mean +- stdev comparison of two pyperf runs.\"\"\"\n    import pyperf\n\n    def load(path):\n        suite = pyperf.BenchmarkSuite.load(path)\n        return {bench.get_name(): bench for bench in suite}\n\n    a, b = load(path_a), load(path_b)\n    names = [n for n in a if n in b]\n    if not names:\n        sys.exit(f\"no benchmark names in common between {path_a} and {path_b}\")\n\n    label_a, label_b = os.path.basename(path_a), os.path.basename(path_b)\n    width = max(len(n) for n in names)\n    col = max(len(label_a), len(label_b), 22)\n    print(f\"{'benchmark':\u003c{width}}  {label_a:>{col}}  {label_b:>{col}}  \"\n          f\"{'speedup':>9}\")\n    print(\"-\" * (width + 2 * col + 17))\n    for name in names:\n        mean_a, sd_a = a[name].mean(), a[name].stdev()\n        mean_b, sd_b = b[name].mean(), b[name].stdev()\n        cell_a = f\"{human_time(mean_a)} +- {human_time(sd_a)}\"\n        cell_b = f\"{human_time(mean_b)} +- {human_time(sd_b)}\"\n        speedup = (mean_a - mean_b) / mean_a * 100.0\n        # flag differences smaller than the combined spread as inconclusive\n        flag = \" ~\" if abs(mean_a - mean_b) \u003c (sd_a + sd_b) else \"\"\n        print(f\"{name:\u003c{width}}  {cell_a:>{col}}  {cell_b:>{col}}  \"\n              f\"{speedup:+8.1f}%{flag}\")\n    print(\"\\n'~' marks differences smaller than the combined standard \"\n          \"deviations.\\nFor pyperf's significance test, use: \"\n          f\"python -m pyperf compare_to {path_a} {path_b} --table\")\n\n\ndef add_cmdline_args(cmd, args):\n    # pyperf spawns worker processes; custom arguments must be forwarded\n    # explicitly, otherwise the workers silently use the defaults.\n    cmd.extend((\"--level\", str(args.level)))\n    cmd.extend((\"--sizes\", args.sizes))\n    if args.skip_chunks:\n        cmd.append(\"--skip-chunks\")\n\n\ndef main():\n    if \"--compare\" in sys.argv:\n        pos = sys.argv.index(\"--compare\")\n        try:\n            return compare(sys.argv[pos + 1], sys.argv[pos + 2])\n        except IndexError:\n            sys.exit(\"--compare needs two pyperf JSON files\")\n\n    import pyperf\n\n    runner = pyperf.Runner(add_cmdline_args=add_cmdline_args)\n    runner.argparser.add_argument(\n        \"--level\", type=int, default=zstd.COMPRESSION_LEVEL_DEFAULT,\n        help=\"zstd compression level used to build the corpus \"\n             f\"(default: {zstd.COMPRESSION_LEVEL_DEFAULT}, the \"\n             \"compression.zstd default)\")\n    runner.argparser.add_argument(\n        \"--sizes\", default=DEFAULT_SIZES,\n        help=f\"comma-separated decompressed sizes (default: {DEFAULT_SIZES})\")\n    runner.argparser.add_argument(\n        \"--skip-chunks\", action=\"store_true\",\n        help=\"skip the 1000-chunk mixed-size workload\")\n    runner.argparser.add_argument(\n        \"--compare\", nargs=2, metavar=(\"A.json\", \"B.json\"),\n        help=\"print a mean +- stdev comparison of two result files and exit\")\n    args = runner.parse_args()\n\n    runner.metadata[\"zstd_version\"] = zstd.zstd_version\n    runner.metadata[\"zstd_level\"] = str(args.level)\n\n    for label in args.sizes.split(\",\"):\n        label = label.strip()\n        if label not in SIZES:\n            sys.exit(f\"unknown size {label!r}, pick from {', '.join(SIZES)}\")\n        frame = get_frame(label, args.level)\n        runner.bench_func(f\"decompress {label} (level {args.level})\",\n                          zstd.decompress, frame)\n\n    if not args.skip_chunks:\n        frames = get_chunk_frames(args.level)\n        runner.bench_func(\n            f\"decompress {len(frames)} chunks (level {args.level})\",\n            decompress_all, frames)\n\n\nif __name__ == \"__main__\":\n    main()\n```\n\nRun it once per interpreter, then compare:\n\n```console\n$ ./python bench_zstd_decompress.py --level 3 -o main.json\n$ ../patched/python bench_zstd_decompress.py --level 3 -o patched.json\n$ ./python bench_zstd_decompress.py --compare main.json patched.json\n```\n\n`python -m pyperf compare_to main.json patched.json --table` works too.\n\n\u003c/details>\n\n## Prior art\n\n`python-zstandard`'s `decompress()` allocates the output buffer from\nthe frame's recorded content size, and pyzstd (from which this module\ndescends) did the same before the stdlib port.\n\n## Real-world motivation\n\nChunk-oriented workloads such as borgbackup decompress millions of\n~64 KiB–8 MiB frames on restore; the output-buffer overhead is ~6%\nof total extraction CPU there.\n\n\n\n\u003c!-- gh-linked-prs -->\n### Linked PRs\n* gh-155464\n\u003c!-- /gh-linked-prs -->\n\n","author":{"url":"https://github.com/ThomasWaldmann","@type":"Person","name":"ThomasWaldmann"},"datePublished":"2026-08-10T06:47:24Z","url":"https://github.com/python/cpython/issues/155460"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:dc791e67-2ed4-b497-d13a-a8fa8816f253
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-id9C48:161320:159D890:1CA7402:6B01FE74
html-safe-nonce9ef46198ffd47e6708215d08c9101b919d567da95bdf3d9c8f01ac1f6c30bbee
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5QzQ4OjE2MTMyMDoxNTlEODkwOjFDQTc0MDI6NkIwMUZFNzQiLCJ2aXNpdG9yX2lkIjoiMTkyMjA0NzUwNzk4NzI5OTk1NiIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmacac238c4db101d50de7cb1b3c8ef68ef6d456741879e5ab07b487542c150af64c
hovercard-subject-tagissue:5107427928
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/155460/issue_layout
twitter:imagehttps://opengraph.githubassets.com/2fa7c8f096f058e75ec275f053706d703523915b211d94aac24640cfd79218b0/python/cpython/issues/155460
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/2fa7c8f096f058e75ec275f053706d703523915b211d94aac24640cfd79218b0/python/cpython/issues/155460
og:image:altFeature or enhancement (performance) Proposal compression.zstd decompression always builds its output through the growing output buffer (initial 32 KiB, then progressively larger growth steps, fini...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernameThomasWaldmann
hostnamegithub.com
expected-hostnamegithub.com
None87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456
turbo-cache-controlno-preview
go-importgithub.com/python/cpython git https://github.com/python/cpython.git
octolytics-dimension-user_id1525981
octolytics-dimension-user_loginpython
octolytics-dimension-repository_id81598961
octolytics-dimension-repository_nwopython/cpython
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id81598961
octolytics-dimension-repository_network_root_nwopython/cpython
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release423dbb54c5e7bfecc838eb387c1435eb3d0e760d
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/python/cpython/issues/155460#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fissues%2F155460
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fissues%2F155460
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=python%2Fcpython
Reloadhttps://github.com/python/cpython/issues/155460
Reloadhttps://github.com/python/cpython/issues/155460
Reloadhttps://github.com/python/cpython/issues/155460
pythonhttps://github.com/python
cpythonhttps://github.com/python/cpython
Codehttps://github.com/python/cpython
Issues5k+ (5k+)https://github.com/python/cpython/issues
Pull requests2.4k (2.4k)https://github.com/python/cpython/pulls
Actionshttps://github.com/python/cpython/actions
Projectshttps://github.com/python/cpython/projects
Security and qualityhttps://github.com/python/cpython/security
Insightshttps://github.com/python/cpython/insights
https://github.com/python/cpython
New issuehttps://github.com/login?return_to=https://github.com/python/cpython/issues/155460
New issuehttps://github.com/login?return_to=https://github.com/python/cpython/issues/155460
Pre-allocate the output buffer in compression.zstd decompression from the frame's recorded content sizehttps://github.com/python/cpython/issues/155460#top
extension-modulesC modules in the Modules dirhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22extension-modules%22
performancePerformance or resource usagehttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22performance%22
type-featureA feature request or enhancementhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22type-feature%22
https://github.com/ThomasWaldmann
ThomasWaldmannhttps://github.com/ThomasWaldmann
on Aug 10, 2026https://github.com/python/cpython/issues/155460#issue-5107427928
pyperfhttps://pyperf.readthedocs.io/
gh-155460: Pre-allocate the output buffer in compression.zstd decompression #155464https://github.com/python/cpython/pull/155464
https://github.com/ThomasWaldmann
ThomasWaldmannhttps://github.com/ThomasWaldmann
on Aug 10, 2026https://github.com/python/cpython/issues/155460#issuecomment-5236956899
@emmatypinghttps://github.com/emmatyping
@Rogdhamhttps://github.com/Rogdham
@AA-Turnerhttps://github.com/AA-Turner
picnixzhttps://github.com/picnixz
type-featureA feature request or enhancementhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22type-feature%22
stdlibStandard Library Python modules in the Lib/ directoryhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22stdlib%22
on Aug 10, 2026https://github.com/python/cpython/issues/155460#event-29207678997
picnixzhttps://github.com/picnixz
Compression issues 🗜https://github.com/orgs/python/projects/20
on Aug 10, 2026https://github.com/python/cpython/issues/155460#event-29207693761
https://github.com/picnixz
picnixzhttps://github.com/picnixz
on Aug 10, 2026https://github.com/python/cpython/issues/155460#issuecomment-5237094172
picnixzhttps://github.com/picnixz
performancePerformance or resource usagehttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22performance%22
extension-modulesC modules in the Modules dirhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22extension-modules%22
pendingThe issue will be closed if no feedback is providedhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22pending%22
stdlibStandard Library Python modules in the Lib/ directoryhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22stdlib%22
on Aug 10, 2026https://github.com/python/cpython/issues/155460#event-29207832563
https://github.com/ThomasWaldmann
ThomasWaldmannhttps://github.com/ThomasWaldmann
on Aug 10, 2026https://github.com/python/cpython/issues/155460#issuecomment-5237241149
@picnixzhttps://github.com/picnixz
https://github.com/picnixz
picnixzhttps://github.com/picnixz
on Aug 10, 2026https://github.com/python/cpython/issues/155460#issuecomment-5237272077
https://github.com/ThomasWaldmann
ThomasWaldmannhttps://github.com/ThomasWaldmann
on Aug 10, 2026https://github.com/python/cpython/issues/155460#issuecomment-5237302476
https://github.com/Rogdham
Rogdhamhttps://github.com/Rogdham
on Aug 10, 2026https://github.com/python/cpython/issues/155460#issuecomment-5237558455
@ThomasWaldmannhttps://github.com/ThomasWaldmann
https://github.com/Rogdham/zstd-benchmarkhttps://github.com/Rogdham/zstd-benchmark
https://github.com/picnixz
picnixzhttps://github.com/picnixz
on Aug 10, 2026https://github.com/python/cpython/issues/155460#issuecomment-5238040218
@ThomasWaldmannhttps://github.com/ThomasWaldmann
https://github.com/ThomasWaldmann
ThomasWaldmannhttps://github.com/ThomasWaldmann
on Aug 10, 2026https://github.com/python/cpython/issues/155460#issuecomment-5238501035
picnixzhttps://github.com/picnixz
pendingThe issue will be closed if no feedback is providedhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22pending%22
on Aug 10, 2026https://github.com/python/cpython/issues/155460#event-29215327214
https://github.com/picnixz
picnixzhttps://github.com/picnixz
on Aug 10, 2026https://github.com/python/cpython/issues/155460#issuecomment-5238768252
https://github.com/ThomasWaldmann
ThomasWaldmannhttps://github.com/ThomasWaldmann
on Aug 10, 2026https://github.com/python/cpython/issues/155460#issuecomment-5238968581
@picnixzhttps://github.com/picnixz
https://github.com/ThomasWaldmann
ThomasWaldmannhttps://github.com/ThomasWaldmann
on Aug 10, 2026https://github.com/python/cpython/issues/155460#issuecomment-5239004688
@picnixzhttps://github.com/picnixz
https://github.com/picnixz
picnixzhttps://github.com/picnixz
on Aug 10, 2026https://github.com/python/cpython/issues/155460#issuecomment-5239434508
https://github.com/maikschulze
maikschulzehttps://github.com/maikschulze
on Nov 4, 2026https://github.com/python/cpython/issues/155460#issuecomment-6632392557
f88565bhttps://github.com/python/cpython/commit/f88565bcd00ac258379c4b0d381d5580a40ac8a7
maikschulzehttps://github.com/maikschulze
on Nov 6, 2026https://github.com/python/cpython/issues/155460#event-35317944952
pythongh-155460: Only pre-size zstd1 frames; match libzstd's default …https://github.com/maikschulze/cpython/commit/fe3ccb4387adeaae24d645032ab5eab951235c31
pythongh-155460https://github.com/python/cpython/issues/155460
fe3ccb4https://github.com/maikschulze/cpython/commit/fe3ccb4387adeaae24d645032ab5eab951235c31
https://github.com/maikschulze
maikschulzehttps://github.com/maikschulze
on Nov 6, 2026https://github.com/python/cpython/issues/155460#issuecomment-6686082950
#161478https://github.com/python/cpython/issues/161478
#161482https://github.com/python/cpython/pull/161482
@picnixzhttps://github.com/picnixz
RFC 8878, 3.1.1.2https://datatracker.ietf.org/doc/html/rfc8878#section-3.1.1.2
#155464https://github.com/python/cpython/pull/155464
maikschulze@2fa76bdhttps://github.com/maikschulze/cpython/commit/2fa76bd9b48a537cc347c038bb73deeb2f1d235f
#161482https://github.com/python/cpython/pull/161482
#155464https://github.com/python/cpython/pull/155464
#155464https://github.com/python/cpython/pull/155464
#161482https://github.com/python/cpython/pull/161482
#155464https://github.com/python/cpython/pull/155464
@ThomasWaldmannhttps://github.com/ThomasWaldmann
@emmatypinghttps://github.com/emmatyping
@Rogdhamhttps://github.com/Rogdham
https://github.com/ThomasWaldmann
ThomasWaldmannhttps://github.com/ThomasWaldmann
on Nov 6, 2026https://github.com/python/cpython/issues/155460#issuecomment-6698492758
2781ffehttps://github.com/python/cpython/commit/2781ffe1f29b0ecf64b011c927ad71df2321b6af
https://github.com/ThomasWaldmann
ThomasWaldmannhttps://github.com/ThomasWaldmann
on Nov 6, 2026https://github.com/python/cpython/issues/155460#issuecomment-6698502600
@maikschulzehttps://github.com/maikschulze
@picnixzhttps://github.com/picnixz
ThomasWaldmannhttps://github.com/ThomasWaldmann
on Nov 6, 2026https://github.com/python/cpython/issues/155460#event-35399852531
pythongh-155460: Pre-allocate the output buffer in compression.zstd d…https://github.com/ThomasWaldmann/cpython/commit/0814951d81cdf6eae01ada14ecf366b5d0109604
pythongh-155460https://github.com/python/cpython/issues/155460
0814951https://github.com/ThomasWaldmann/cpython/commit/0814951d81cdf6eae01ada14ecf366b5d0109604
pythongh-155460: Pre-size each whole zstd frame, respect window_log_maxhttps://github.com/ThomasWaldmann/cpython/commit/431f0f1255f6afb352e047f1da344d5eee7753b7
pythongh-155460https://github.com/python/cpython/issues/155460
431f0f1https://github.com/ThomasWaldmann/cpython/commit/431f0f1255f6afb352e047f1da344d5eee7753b7
pythongh-155460: Make the multi-frame zstd test depend on whole-frame…https://github.com/ThomasWaldmann/cpython/commit/d81eed6fe3efa8fdf6973895de9631c892a59422
pythongh-155460https://github.com/python/cpython/issues/155460
d81eed6https://github.com/ThomasWaldmann/cpython/commit/d81eed6fe3efa8fdf6973895de9631c892a59422
Sign up for freehttps://github.com/signup?return_to=https://github.com/python/cpython/issues/155460
Sign in to commenthttps://github.com/login?return_to=https://github.com/python/cpython/issues/155460
extension-modulesC modules in the Modules dirhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22extension-modules%22
performancePerformance or resource usagehttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22performance%22
type-featureA feature request or enhancementhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22type-feature%22
Compression issues 🗜https://github.com/orgs/python/projects/20
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.