René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Not a Node bug: \"bad record mac\" at end of stream — proxy stack injects an extra TLS record","articleBody":"### Version\n\nv24.21.0; Also reproduces on `v20.20.2` - see the runtime matrix under Additional information.\n\n### Platform\n\n```text\nWindows: Microsoft Windows NT 10.0.26200.0, x64\nLinux:   Ubuntu-24.04 WSL2 guest on the same host, x64\n---\nWindows: Microsoft Windows NT 10.0.19045.0, x64\n```\n\n### Subsystem\n\n`tls` - the failure is in the TLS record read path. `zlib` is the trigger (the backpressure shape) and `http` is the layer that masks the error, but neither is where the fault is.\n\n### What steps will reproduce the bug?\n\nAn HTTPS response is piped through a `node:zlib` stream that applies backpressure to the\nresponse, with the connection going through an HTTP CONNECT proxy:\n\n```js\nconst res = await get(url);                  // HTTPS through a CONNECT tunnel\nawait pipeline(\n  res,\n  countBytes,                                // plain Transform\n  zlib.createZstdDecompress(),               // or createGzip() - both reproduce\n  countBytes2,\n  slowSink,                                  // anything that lags the consumer\n  fileOrNullSink\n);\n```\n\nRequirements, each with a control that turns it off:\n\n- The body must be large enough to reach steady state (~80 MB observed; smaller was not\n  characterised).\n- The zlib stream must be **directly downstream of `res`**. Moving a lagging plain `Transform`\n  in front of the zlib stream does not reproduce.\n- The consumer downstream of zlib must lag enough to backpressure the response. Packet capture\n  shows repeated `TCP Zero Window` segments on the client's receive side. Without the artificial\n  lag the failure still occurs, but intermittently; the lag only raises the trigger rate to about\n  100%.\n- The relay in front of the client must be an HTTP CONNECT proxy; see \"How often does it\n  reproduce\".\n\nSelf-contained script (Node standard library only, no third-party dependencies). It needs Node\n>= 22.15 for `zlib.createZstdDecompress`; substitute `createGzip()` on older versions:\n\n```js\n// repro.mjs\n//\n//   node repro.mjs --url \u003clarge HTTPS artifact, ~80 MB or more> --proxy \u003chttp://host:port>\n//   node repro.mjs --url \u003csame> --proxy \u003csame> --decode none      # control: no decompressor\n//   node repro.mjs --url \u003csame> --proxy \u003csame> --drain-first      # control: no backpressure\nimport http from 'node:http';\nimport https from 'node:https';\nimport tls from 'node:tls';\nimport zlib from 'node:zlib';\nimport { Readable, Transform, Writable } from 'node:stream';\nimport { pipeline } from 'node:stream/promises';\nimport { parseArgs } from 'node:util';\n\nconst { values } = parseArgs({ options: {\n  url: { type: 'string' },\n  proxy: { type: 'string' },\n  decode: { type: 'string', default: 'zstd' },\n  lag: { type: 'string', default: '100' },\n  'drain-first': { type: 'boolean', default: false }\n} });\n\nconst target = new URL(values.url);\nconst proxy = new URL(values.proxy);\n\nasync function connect() {\n  const req = http.request({\n    host: proxy.hostname,\n    port: Number(proxy.port || 80),\n    method: 'CONNECT',\n    path: `${target.hostname}:${target.port || 443}`\n  });\n  req.end();\n  return new Promise((resolve, reject) => {\n    req.once('connect', (r, socket) =>\n      r.statusCode === 200 ? resolve(socket) : reject(new Error(`CONNECT ${r.statusCode}`)));\n    req.once('error', reject);\n  });\n}\n\nconst tunnel = await connect();\nconst res = await new Promise((resolve, reject) => {\n  const req = https.request({\n    host: target.hostname,\n    port: Number(target.port || 443),\n    path: target.pathname + target.search,\n    createConnection: () => tls.connect({ socket: tunnel, servername: target.hostname })\n  }, resolve);\n  req.once('error', reject);\n  req.end();\n});\n\nlet wire = 0;\nconst count = new Transform({ transform(c, _e, cb) { wire += c.length; cb(null, c); } });\n\n// Consumer lag: hold the callback for a timer tick every 1 MB. This is what makes\n// the response backpressure.\nconst lagMs = Number(values.lag);\nlet acc = 0;\nconst lag = new Transform({\n  highWaterMark: 4194304,\n  transform(c, _e, cb) {\n    acc += c.length;\n    if (acc \u003c 1048576) return cb(null, c);\n    acc = 0;\n    setTimeout(() => cb(null, c), lagMs);\n  }\n});\n\nconst sink = new Writable({ highWaterMark: 4194304, write(_c, _e, cb) { cb(); } });\n\ntry {\n  if (values['drain-first']) {\n    // Control: consume the response with no backpressure, decompress afterwards.\n    const chunks = [];\n    await new Promise((resolve, reject) => {\n      res.on('data', (c) => { wire += c.length; chunks.push(c); });\n      res.once('end', resolve);\n      res.once('error', reject);\n    });\n    await pipeline(Readable.from(chunks), zlib.createZstdDecompress(), lag, sink);\n  } else {\n    const stages = values.decode === 'zstd' ? [zlib.createZstdDecompress()] : [];\n    await pipeline(res, count, ...stages, lag, sink);\n  }\n  console.log(`OK wire=${wire}`);\n} catch (err) {\n  console.log(`FAIL code=${err.code} msg=${JSON.stringify(err.message)} wire=${wire}`);\n  console.log(`res.complete=${res.complete} res.aborted=${res.aborted}` +\n    ` res.readableEnded=${res.readableEnded} res.destroyed=${res.destroyed}`);\n}\n```\n\nRun each configuration several times, and run a passing configuration in the same minute: the\nfailure is intermittent, so a single run proves nothing in either direction.\n\n\n### How often does it reproduce? Is there a required condition?\n\nWith the consumer lag in place, essentially every run fails - the transfer stops at\n99.76% - 99.96% of the declared body. Without the lag it fails intermittently, which is why\nevery measurement in this report was paired with a same-minute positive control.\n\nThree conditions must hold at once:\n\n1. A `node:zlib` stream directly downstream of the response.\n2. Backpressure from the consumer reaching the response through that zlib stream.\n3. A relay that re-chunks the stream with a particular **write-size distribution** - see\n   Additional information.\n\nCondition 3 is the one that is **not** reproduced without a real relay. The reproduction below\nstill needs one, and this issue does not claim any specific proxy is required - only that a\nthird-party relay is currently the only thing that triggers it dependably on this path. A fully\nlocal rig (self-signed TLS origin on loopback) never reproduces.\n\n### What is the expected behavior? Why is that the expected behavior?\n\nA complete, correctly transmitted TLS stream must decrypt regardless of how the application\napplies backpressure. Backpressure is an application-level property and cannot change the bytes\non the wire - and in this case the bytes were proven intact offline (see Additional\ninformation). Failing to decrypt them is a defect in the reading side.\n\nSecondarily, the HTTP client should not translate a TLS failure into `aborted` /\n`ECONNRESET`: that rewrites the root cause into a network error and is why applications\ndownstream blame the proxy or the network.\n\n### What do you see instead?\n\n```\nERR_SSL_DECRYPTION_FAILED_OR_BAD_RECORD_MAC\nerror:0A000119:SSL routines:tls_get_more_records:decryption failed or bad record mac\n```\n\nfollowed by a fatal `bad_record_mac` alert being sent to the peer.\n\nEvent order, from listeners attached to every object in the chain:\n\n```\n21.153 tlsSocket.error  ERR_SSL_DECRYPTION_FAILED_OR_BAD_RECORD_MAC\n21.153 rawSocket.close\n21.153 tlsSocket.close\n21.154 res.aborted\n21.154 res.error        ECONNRESET: aborted\n21.155 (pipeline tears down every downstream stage)\n```\n\nThe TLS error is first; everything after it is a consequence.\n\nWhat the application sees, because Node's HTTP client observes the socket closing while\n`res.complete === false` and replaces the error with `connResetException('aborted')`:\n\n```\ncode=ECONNRESET  msg=\"aborted\"\nres.complete=false  res.aborted=true  res.readableEnded=false  res.destroyed=true\n```\n\n### Additional information\n\n### The ciphertext on the wire is intact\n\nThe inner TLS stream (the CONNECT tunnel on loopback) was captured in full, and the session keys\nwere exported from the failing run with `--tls-keylog`. Decrypting the capture offline with\nthose keys: **8,491 application-data records, 80,603,843 bytes, zero decryption failures**. The\nsame bytes, processed in-process by the running Node client at the same moment, produced a MAC\nfailure.\n\nCommands used:\n\n```\n# capture the inner TLS stream, full packets\ntshark -i \u003cloopback interface> -f \"tcp port \u003cproxy port>\" -w \u003ccapture> -a duration:70\n\n# export session keys from the failing run\nnode --tls-keylog=\u003ckeylog> repro.mjs ...\n\n# any record that failed to decrypt would appear in expert info\ntshark -r \u003ccapture> -o tls.keylog_file:\u003ckeylog> -d tcp.port==\u003cproxy port>,http \\\n       -Y \"tcp.stream==\u003cn> && _ws.expert\" -T fields -e frame.number -e _ws.expert.message\n\n# and the decrypted application-data volume must match expectations\ntshark -r \u003ccapture> -o tls.keylog_file:\u003ckeylog> -d tcp.port==\u003cproxy port>,http \\\n       -Y \"tcp.stream==\u003cn> && tls.record.content_type==23\" -T fields -e tls.record.length\n```\n\n`--export-objects http,\u003cdir>` is not usable here: it silently exports nothing when the response\nbody is large.\n\n### Platform / TLS-library / runtime matrix\n\nSame proxy, same origin, same artifact, comparable consumer lag. Every row was run with a\nsame-minute positive control, because the failure is intermittent. The percentage is where the\ntransfer stopped, not a failure rate.\n\n| Runtime | TLS library | Event library | Platform | Result (stop point, % of declared body) |\n|---|---|---|---|---|\n| Node 24.21.0 | OpenSSL 3.5.8 (bundled) | libuv 1.52.1 | Windows 11 x64 | **FAIL** 99.963% |\n| Node 24.21.0 | OpenSSL 3.5.8 (bundled) | libuv 1.52.1 | Linux (WSL2) | **FAIL** 99.760% |\n| Node 20.20.2 | OpenSSL 3.0.19 (bundled) | libuv 1.46.0 | Linux (WSL2) | **FAIL** 99.963% |\n| Python 3.12.3 | OpenSSL 3.0.13 (system) | - | Linux (WSL2) | **OK**, correct hash |\n\nThe two Node failures start with the same error text but go through different OpenSSL internals:\n\n```\nNode 24 / 3.5.8    tls_get_more_records   tls_common.c\nNode 20 / 3.0.19   ssl3_get_record        ssl3_record.c\n```\n\nSo: not platform-specific (the Windows overlapped-I/O read path is out), and not an OpenSSL\nversion regression (3.0.19 and 3.5.8 both fail, via different code). The difference that matters\nis the runtime - same OpenSSL major, Python completes and Node does not.\n\n### Mechanism hypothesis (narrowed, not proven)\n\nPython lets OpenSSL read the socket itself; Node pushes TCP chunks into a memory BIO and lets\nOpenSSL read from there. If Node reuses, truncates or misaligns the buffer it feeds to that BIO\nacross a backpressure pause/resume, the result is exactly `bad record mac`. This fits every known\ncondition: needs backpressure (for the pause/resume), needs a relay (so records straddle read\nboundaries), platform-independent, OpenSSL-version-independent.\n\nCaveat: Python's read shape (blocking `read` plus sleeps) is not identical to Node's stream\nbackpressure, so its pass shows the failure is not universal to TLS clients under backpressure -\nit does not prove that no other read shape can drive OpenSSL into this.\n\n### The relay's write-size sequence\n\nFailing and passing paths were compared by recording the size of every downstream write the relay\nmade:\n\n| | failing relay | passing relay (`ssh -L`) |\n|---|---|---|\n| writes | 1461 | 1295 |\n| smallest write | 616 | 2608 |\n| writes `\u003c= 16 KB` | **14.0%** | 2.4% |\n| writes `\u003c= 4 KB` | **6.6%** | 0.4% |\n| common small sizes | 2824 / 5648 / 8472 (multiples of 1412) | rare |\n| inter-arrival p50 / p95 | 0.152 / 16.95 ms | 0.175 / 12.56 ms |\n\nInter-arrival timing is nearly identical; the measured difference is concentrated in write\nsizes. The failing relay is a local HTTP CONNECT proxy (mihomo) forwarding to a remote egress;\nthe passing one is an `ssh -L` tunnel to the same egress host.\n\nReplaying the recorded sequence through a purpose-written CONNECT proxy on the passing path does\nreproduce the failure - but only sometimes, so write sizes are **a** relevant characteristic, not\na sufficient one:\n\n| Synthesised | Result |\n|---|---|\n| Recorded write-size sequence | 2 failures out of 7 attempts |\n| Write sizes **plus** recorded inter-arrival gaps | 0 out of 4 |\n| Either of the above **plus** unbounded read-ahead | 0 out of 2 |\n| Local TLS origin forced to emit small (2824-byte) records | 0 out of 1 |\n| Uniform chunking at 16384 / 4096 / 1412 bytes, with `TCP_NODELAY` | 0 out of 3 |\n\nThe recorded median inter-arrival gap is **0.152 ms**. JavaScript timers cannot reproduce\nsub-millisecond arrival structure, so anything below 1 ms degrades to `setImmediate`. That is a\ntooling ceiling here, not a matter of trying more patterns; going further would need\npacket-level shaping or pcap replay.\n\nWhat these attempts do establish is that no data tampering is involved - a relay that merely\nre-chunks a correct byte stream can trigger it at least sometimes.\n\n### The stop point\n\nThe failure always lands very close to the end, and repeatedly on exactly the same offset -\nincluding across two **different** upstream egress servers in different regions running\ndifferent proxy-server implementations:\n\n```\nbytes delivered into the pipeline: 80,428,448\nbytes emitted by the decompressor: 222,453,760\n```\n\nAcross all runs the stop point takes a small number of discrete values in the range\n99.76% - 99.96% of the body rather than a continuum. That suggests fixed client-side buffer\nboundaries rather than anything network-timing-dependent, but the value is not constant, so it\nshould not be read as a single magic offset.\n\n### Ruled out\n\nEach item has a controlled experiment behind it.\n\n| Hypothesis | Verdict | Control |\n|---|---|---|\n| The proxy corrupts bytes | **No** | Offline decryption of the captured ciphertext with `--tls-keylog` keys: all records decrypt, no failures. Same trigger conditions over **plain HTTP** at comparable throughput: 104,857,600 bytes, SHA-256 identical to baseline, while a same-minute HTTPS positive control failed. Inserting a purpose-written proxy in front of it does not neutralise the failure. |\n| That specific proxy is strictly required | **Unresolved** | Replaying its recorded write-size sequence through a purpose-written proxy on a path it is not part of reproduced the failure 2/7. Enough to show no tampering is needed; not enough to call it unnecessary. |\n| Windows-specific I/O read path | **No** | Same build reproduces on Linux. |\n| OpenSSL version regression | **No** | OpenSSL 3.0.19 and 3.5.8 both reproduce, through different internal code paths. |\n| TLS records being small | **No** | A local origin forced to emit 2824-byte records does not reproduce. |\n| Data expansion by decompression | **No** | `createGzip()` on already-compressed input (in ~= out) reproduces. |\n| Slow consumer alone | **No** | A plain `Transform` with the same total lag and the same backpressure completes. |\n| Per-chunk async deferral | **No** | A `Transform` deferring every chunk via `setImmediate`, same total lag, completes. |\n| libuv threadpool contention | **No** | `UV_THREADPOOL_SIZE` of 1 and of 32 both reproduce. |\n| Backpressure on `res` | **Required** | Draining the response fully into memory first, then decompressing with the same lag downstream, completes with a correct hash. |\n| A `zlib` stream in the chain | **Required** | Same lag and same backpressure; the only difference is whether that stage is `zlib`. |\n| Ordering | **Matters** | Putting the lagging plain `Transform` *before* the zlib stream does not reproduce. Backpressure must reach `res` through the zlib stream. |\n| Going through the proxy | **Required** | The identical configuration over an `ssh -L` tunnel to the same egress host completes with a correct hash. |\n\n### Workaround\n\nDo not place a streaming decompressor directly downstream of the HTTP response. Drain the\nresponse without backpressure, then decompress. Verified on the same path with the same\ndecompressor and the same total lag: correct output hash.\n\n### Not yet tested\n\n- macOS\n- Bare-metal Linux (the Linux result above is from WSL2; a positive there is still decisive, but\n  the kernel is a vendor fork behind a virtual NIC)\n- A non-OpenSSL runtime such as Go, which would tell whether any TLS implementation can be driven\n  into this by the same relay\n- HTTP/2 (the failing path is HTTP/1.1 inside the tunnel)\n- Which relay characteristic is actually required (see the synthetic-relay table above)\n\n---\n\nAI assistance: the investigation, experiment design and write-up of this report were done\nwith Claude (Opus 5). All runs were executed on the reporting host against the real path;\nevery number in this report is measured on that host, not estimated.\n","author":{"url":"https://github.com/yx64","@type":"Person","name":"yx64"},"datePublished":"2026-09-12T15:43:40Z","url":"https://github.com/nodejs/node/issues/66001"}