René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"zlib: createZstdCompress appends an empty frame when end() is called with writes still queued","articleBody":"### Version\n\nv26.9.0 (also v24.15.0)\n\n### Platform\n\n```text\nDarwin 25.6.0 arm64 (also seen on Linux x64)\n```\n\n### Subsystem\n\nzlib\n\n### What steps will reproduce the bug?\n\n```js\n'use strict';\nconst zlib = require('node:zlib');\n\nfunction compress(create, writes) {\n  return new Promise((resolve, reject) => {\n    const stream = create();\n    const chunks = [];\n    stream.on('data', (chunk) => chunks.push(chunk));\n    stream.on('end', () => resolve(Buffer.concat(chunks)));\n    stream.on('error', reject);\n    for (const w of writes) stream.write(w);\n    stream.end();\n  });\n}\n\n(async () => {\n  const oneWrite = await compress(zlib.createZstdCompress, ['hello world']);\n  const twoWrites = await compress(zlib.createZstdCompress, ['hello ', 'world']);\n\n  console.log('one write: ', oneWrite.toString('hex'));\n  console.log('two writes:', twoWrites.toString('hex'));\n  console.log('extra bytes:', twoWrites.subarray(oneWrite.length).toString('hex'));\n\n  // Same input, queued writes: gzip and brotli are unaffected.\n  for (const [name, create, decompress] of [\n    ['gzip', zlib.createGzip, zlib.gunzipSync],\n    ['brotli', zlib.createBrotliCompress, zlib.brotliDecompressSync],\n  ]) {\n    const a = await compress(create, ['hello world']);\n    const b = await compress(create, ['hello ', 'world']);\n    console.log(name, 'lengths', a.length, b.length, decompress(b).toString());\n  }\n})();\n```\n\n### How often does it reproduce? Is there a required condition?\n\nEvery time `end()` is called while at least one write is still queued in the compressor. In practice that is most pipelines whose source ends quickly, e.g. `pipeline(tar.create(...), zlib.createZstdCompress(), fs.createWriteStream(...))`: every archive we packed that way had the extra frame.\n\n### What is the expected behavior? Why is that the expected behavior?\n\nOne zstd frame, the same bytes whether the input arrived in one write or several:\n\n```\none write:  28b52ffd005859000068656c6c6f20776f726c64\ntwo writes: 28b52ffd005859000068656c6c6f20776f726c64\n```\n\nThat's what gzip and brotli do. How the input was split into writes shouldn't change the compressed output.\n\n### What do you see instead?\n\n```\none write:  28b52ffd005859000068656c6c6f20776f726c64\ntwo writes: 28b52ffd005859000068656c6c6f20776f726c6428b52ffd2000010000\nextra bytes: 28b52ffd2000010000\ngzip lengths 31 31 hello world\nbrotli lengths 15 15 hello world\n```\n\nA second, empty zstd frame (`28b52ffd 20 00 01 00 00`: magic, single-segment descriptor with content size 0, one empty raw last block) is appended.\n\n### Additional information\n\nI think the cause is in `lib/zlib.js`:\n\n- In `ZlibBase#_transform`, when `this.writableEnded && this.writableLength === chunk.byteLength`, the last queued chunk is processed with `_finishFlushFlag`, which ends the frame.\n- `ZlibBase#_flush` then calls `_transform` again with an empty buffer. `writableEnded` is still true and `writableLength` is 0, so that call gets the finish flag too.\n\nFor deflate and brotli a second finish on a finished stream emits nothing. For zstd, `ZSTD_compressStream2(..., ZSTD_e_end)` on a context whose frame has just completed starts and completes a new frame. `ZstdCompressContext::DoThreadPoolWork` in `src/node_zlib.cc` doesn't guard against that. When `end()` is called with nothing queued, the frame is only finished once, so the output is a single frame.\n\nThe extra frame is valid zstd, but it has real consequences:\n\n1. The output depends on stream timing, not content. We hash the archive for deduplication, so identical inputs can hash differently.\n2. In released versions (including v26.9.0), `createZstdDecompress` throws `Unknown frame descriptor` when a read chunk boundary falls inside those 9 bytes. With `fs.createReadStream`'s default 64 KiB chunks, that's about 1 archive in 8,200, and it fails the same way every time. We hit this in production on an archive that `zstd -t` and `zstdDecompressSync` both accept. I believe #65865 fixes the decoding side on `main`, but the compressor still writes the extra frame.\n\nOur workaround is to strip a trailing `28b52ffd2000010000` after compressing.","author":{"url":"https://github.com/trotzig","@type":"Person","name":"trotzig"},"datePublished":"2026-09-17T06:22:45Z","url":"https://github.com/nodejs/node/issues/66078"}