René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"UTF-8 character corruption in fast-utf8-stream.js via releaseWritingBuf() leads to silent data loss on partial writes","articleBody":"### Version\n\n_No response_\n\n### Platform\n\n```text\n**Summary:** releaseWritingBuf() in lib/internal/streams/fast-utf8-stream.js incorrectly calculates string slice positions when fs.write returns a byte count that splits a multi-byte UTF-8 character, causing silent data corruption (lost characters, lone surrogates in output).\n\n**Description:** \n\nThe releaseWritingBuf function (line 896) converts bytes-written to character count using:\n\n\nn = Buffer.from(writingBuf).subarray(0, n).toString().length;\n\n\nWhen n bytes cuts through a multi-byte character, the incomplete UTF-8 sequence becomes U+FFFD (replacement character) via .toString(). This replacement character has a different .length than the original character in JS UTF-16, causing .slice(n) to cut at the wrong position:\n\n- 3-byte characters (CJK, most non-Latin): character silently dropped from output\n- 4-byte characters (emoji, supplementary CJK): lone low surrogate left in remaining buffer, producing invalid UTF-8 on next write\n\nThe file was recently added (January 2026), ported from SonicBoom. It is used as the fast path for streaming UTF-8 output.\n\n## Steps To Reproduce:\n\n1. Save this as poc.js and run with node poc.js:\n\n\n// Reproduces the releaseWritingBuf logic from lib/internal/streams/fast-utf8-stream.js lines 896-906\nfunction releaseWritingBuf(writingBuf, len, n) {\n  if (typeof writingBuf === 'string' \u0026\u0026 Buffer.byteLength(writingBuf) !== n) {\n    n = Buffer.from(writingBuf).subarray(0, n).toString().length;\n  }\n  len = Math.max(len - n, 0);\n  writingBuf = writingBuf.slice(n);\n  return { writingBuf, len };\n}\n\n// Case 1: 4-byte emoji split at byte 7 — lone surrogate\nconst r1 = releaseWritingBuf(\"hello🌍world\", 14, 7);\nconsole.log(\"Case 1 - Emoji split:\");\nconsole.log(\"  Result:\", JSON.stringify(r1.writingBuf));\nconsole.log(\"  Expected:\", JSON.stringify(\"🌍world\"));\nconsole.log(\"  First char code: 0x\" + r1.writingBuf.charCodeAt(0).toString(16));\nconsole.log(\"  Is lone surrogate:\", r1.writingBuf.charCodeAt(0) \u003e= 0xDC00 \u0026\u0026\nr1.writingBuf.charCodeAt(0) \u003c= 0xDFFF);\n\n// Case 2: 3-byte CJK char split at byte 4 — character lost\nconst r2 = releaseWritingBuf(\"abc中def\", 9, 4);\nconsole.log(\"\\nCase 2 - CJK split:\");\nconsole.log(\"  Result:\", JSON.stringify(r2.writingBuf));\nconsole.log(\"  Expected:\", JSON.stringify(\"中def\"));\nconsole.log(\"  Character 中 lost:\", !r2.writingBuf.includes(\"中\"));\n\n\n2. Output shows:\n\nCase 1 - Emoji split:\n  Result: \"\\udf0dworld\"        ← CORRUPTED (lone surrogate)\n  Expected: \"🌍world\"\n  First char code: 0xdf0d\n  Is lone surrogate: true\n\nCase 2 - CJK split:\n  Result: \"def\"                ← CHARACTER LOST\n  Expected: \"中def\"\n  Character 中 lost: true\n\n\n3. The vulnerable code is at:\nhttps://github.com/nodejs/node/blob/main/lib/internal/streams/fast-utf8-stream.js#L896-L906\n\nPartial fs.write returns are possible when writing to pipes near capacity, under disk I/O pressure, or to Docker log pipes (the exact use case mentioned in the file's comments on line 69-70).\n\nAdditional finding: Line 240 has a typo from the SonicBoom port — this._asyncDrainScheduled should be this.#asyncDrainScheduled. All other 5 references use the private field correctly. The newListener handler is effectively dead code.\n\n## Impact:\n\nSilent data corruption in output files. Applications using Utf8Stream for logging with international characters (CJK, emoji, Cyrillic) can produce corrupted output when partial writes occur. 3-byte characters are silently lost (no error emitted). 4-byte characters produce invalid UTF-8 (lone surrogates). This is especially relevant for the Docker container logging use case the file was designed for.\n\n## Supporting Material/References:\n\n- Vulnerable function: releaseWritingBuf() at https://github.com/nodejs/node/blob/main/lib/internal/streams/fast-utf8-stream.js#L896-L906\n- Typo (secondary): line 240, _asyncDrainScheduled vs #asyncDrainScheduled\n- File derived from SonicBoom (https://github.com/pinojs/sonic-boom) — the original has a similar issue but uses _ prefix consistently\n- The PoC script above is standalone and runs on any Node.js version\n```\n\n### Subsystem\n\n_No response_\n\n### What steps will reproduce the bug?\n\n1. Save the following script as `poc.js` and run it with `node poc.js`.\n\n2. The script reproduces the exact logic from\n   `lib/internal/streams/fast-utf8-stream.js` (lines 896–906),\n   specifically the `releaseWritingBuf()` function.\n\n3. The script simulates partial `fs.write()` behavior where the number\n   of bytes written splits a multi-byte UTF-8 character.\n\n4. Observe the output:\n   - When a 4-byte UTF-8 character (emoji) is split, a lone surrogate\n     remains in the output.\n   - When a 3-byte UTF-8 character (CJK) is split, the character is\n     silently dropped.\n\n5. This demonstrates incorrect string slicing caused by converting\n   byte counts to character counts via `.toString().length`.\n\n### How often does it reproduce? Is there a required condition?\n\nIt reproduces deterministically whenever `fs.write()` (or an equivalent\ninternal write) returns a byte count that splits a multi-byte UTF-8\ncharacter.\n\nThe issue is not timing-dependent or race-based. The required condition\nis a partial write that ends in the middle of a UTF-8 sequence.\n\nThis can occur when writing to pipes, sockets, or log streams under\nbackpressure (e.g. near-capacity pipes, Docker container logs, or heavy\nI/O), which is a documented and expected behavior of `fs.write()`.\n\n### What is the expected behavior? Why is that the expected behavior?\n\nThe output must always preserve valid UTF-8 and must not silently\ncorrupt data.\n\nWhen a partial write ends in the middle of a multi-byte UTF-8 character,\nthe remaining bytes for that character should be preserved and written\nin a subsequent write, rather than being dropped or converted into\nreplacement characters.\n\nThis is the expected behavior because:\n- `fs.write()` is documented to return partial byte counts.\n- UTF-8 stream handling must be byte-safe across writes.\n- Producing lone surrogates or dropping characters violates UTF-8\n  correctness and results in silent data corruption.\n\nThe current behavior breaks UTF-8 invariants and can corrupt log output\nin real-world streaming scenarios, such as container logging and pipe-\nbased streams, which this module explicitly targets.\n\n### What do you see instead?\n\nInstead of preserving valid UTF-8 output, the stream produces corrupted\nresults when a partial write splits a multi-byte character.\n\nSpecifically:\n- For 3-byte UTF-8 characters (e.g. CJK), the character is silently\n  dropped from the output with no error.\n- For 4-byte UTF-8 characters (e.g. emoji), the remaining buffer starts\n  with a lone UTF-16 surrogate, producing invalid UTF-8 on subsequent\n  writes.\n\nNo error or warning is emitted, resulting in silent data corruption in\nthe output stream.\n\n### Additional information\n\n_No response_","author":{"url":"https://github.com/pitiflautico","@type":"Person","name":"pitiflautico"},"datePublished":"2026-02-08T18:50:26.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/61744/node/issues/61744"}