René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"TextDecoder is wrong and very slow","articleBody":"## Correctness\n\nEncodings that return invalid results:\n * Single-byte:\n    * `ibm866` (fails at even ascii input)\n    * `koi8-u`\n    * `windows-874`\n    * `windows-1252`\n    * `windows-1253`\n    * `windows-1255`\n * Multi-byte (all except `gb18030`):\n   * `gbk` (should be identical to `gb18030` but it is instead broken)\n   * `big5`\n   * `euc-jp`\n   * `iso-2022-jp`\n   * `shift_jis` (fails at even ascii input)\n   * `euc-kr`\n\nUnimplemented encodings that throw:\n   * `iso-8859-16`\n   * `x-user-defined`\n\n\nIf built without `icu`, `utf-16le` encoding also returns invalid results:\n```js\n\u003e new TextDecoder('utf-16le').decode(Uint16Array.of(0xd800))\n'�' // correct\n'\\ud800' // no ICU\n```\n\n## Performance\n\n* `utf-8` (aka default) `TextDecoder` is much slower on ascii input than it can and should be\n   `1.3x` on 4096 bytes, `~3x` on 1 MiB input\n*  The above applies to `buffer.toString()` too\n   It's much slower on ASCII input than a checked js impl (same `1.3x`-`3x`)\n*  `windows-1252` aka `new TextDecoder('ascii')` aka `new TextDecoder('latin1')`\n    is ~`2x-4x` slower than an optimized impl on ascii input\n* `windows-1252` aka `new TextDecoder('latin1')`\n   is ~`6x-12x` slower than an optimized impl on latin1 input\n* `windows-1252` is ~`7x-12x` slower than an optimized js impl\n* Other single-byte encodings that are significantly slower than js impl even on non-ascii input:\n   `iso-8859-3`, `iso-8859-6`, `iso-8859-7`, `iso-8859-8`, `iso-8859-8-i`, `windows-1253`, `windows-1255`, `windows-1257`\n* None of the single-byte encodings are _faster_ than the js impl even on non-ascii input\n* All of the single-byte encodings except `windows-1252` are `\u003e=10x` slower than the js impl on ascii input\n  (`windows-1252` is _only_ ~`2-4x` slower)\n\n## References\n\nNothing of the above requires any changes on the native side, I compared to a somewhat optimized JS implementation\n\nSee https://docs.google.com/spreadsheets/d/1pdEefRG6r9fZy61WHGz0TKSt8cO4ISWqlpBN5KntIvQ/edit\n\nSee tests in https://github.com/ExodusOSS/bytes/blob/master/tests/encoding/mistakes.test.js (comment out the import and it can be run on Node.js without deps with only that file)\n\n## Suggestions\n\n1. Add a proper ASCII fast path to `buffer.toString()`\n   https://github.com/nodejs/node/pull/61119\n2. Add a proper ASCII fast path to `new TextDecoder().decode(arg)`\n   https://github.com/nodejs/node/pull/61119\n3. Perhaps replace single-byte decoders with a js impl, remove native paths and lib usage.   They all are just mappers, the implementation for all of them is identical\n   Or at least replace the slow, unsupported, or invalid ones.\n   #61093 \n4. Remove `gbk` decoder path and make it do the same as `gb18030` as the spec says\n    https://github.com/nodejs/node/pull/61099\n5. For utf16 decode optimistically using existing fast apis, then check the string for validity\n    https://github.com/nodejs/node/pull/61559\n6. Fix bugs in the non-ICU codepath\n    https://github.com/nodejs/node/pull/61409\n    https://github.com/nodejs/node/pull/61549\n    https://github.com/nodejs/node/pull/61559\n7. Fix or replace implementations for `big5`, `euc-jp`, `iso-2022-jp`, `shift_jis`, `euc-kr`\n    To fix legacy multi-byte decoders, attempt to re-use what Chromium has or import js code from `@exodus/bytes`\n","author":{"url":"https://github.com/ChALkeR","@type":"Person","name":"ChALkeR"},"datePublished":"2025-12-13T05:32:02.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":12},"url":"https://github.com/61041/node/issues/61041"}