René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Assertion failure crash in TLSWrap::DoWrite with zombie HTTP/2 session (close event not propagated from OS-level CLOSED socket)","articleBody":"# Version\n\nv22.13.1\n\n# Platform\n\nDarwin 24.3.0 (macOS, arm64)\n\n# Subsystem\n\nhttp2, tls\n\n# What steps will reproduce the bug?\n\n## Reproducible test case (requires sudo for firewall manipulation)\n\nWe created a test that simulates a network \"black hole\" using macOS pf firewall:\n\n```javascript\n// test-zombie-blackhole.js\n// Run with: sudo node test-zombie-blackhole.js\n\nconst http2 = require('http2')\nconst { spawn, execSync } = require('child_process')\n\nconst PORT = 8444\n\nasync function main() {\n    if (process.getuid() !== 0) {\n        console.log('Run with: sudo node test-zombie-blackhole.js')\n        process.exit(1)\n    }\n\n    // Start server as child process\n    const server = spawn('node', ['-e', `\n        const http2 = require('http2')\n        const fs = require('fs')\n        const server = http2.createSecureServer({\n            key: fs.readFileSync('./key.pem'),\n            cert: fs.readFileSync('./cert.pem'),\n        })\n        server.on('stream', (s, h) =\u003e {\n            s.respond({ ':status': 200 })\n            s.end('ok')\n        })\n        server.listen(${PORT}, () =\u003e console.log('Server ready'))\n        setInterval(() =\u003e {}, 10000)\n    `], { stdio: 'inherit' })\n\n    await new Promise(r =\u003e setTimeout(r, 1000))\n\n    // Connect client\n    const session = http2.connect(\\`https://localhost:\\${PORT}\\`, { rejectUnauthorized: false })\n\n    session.on('error', e =\u003e console.log('error event:', e.message))\n    session.on('close', () =\u003e console.log('close event'))\n\n    await new Promise(r =\u003e session.on('connect', r))\n    console.log('Connected')\n\n    // Make initial request to establish session\n    await new Promise((resolve, reject) =\u003e {\n        const s = session.request({ ':path': '/init' })\n        s.on('end', resolve)\n        s.on('error', reject)\n        s.end()\n    })\n\n    // Block traffic with firewall (black hole - no RST/FIN, packets just disappear)\n    execSync(\\`echo \"block drop quick proto tcp from any to 127.0.0.1 port \\${PORT}\" | pfctl -a zombie_test -f -\\`)\n    execSync('pfctl -e 2\u003e/dev/null || true')\n    console.log('Firewall blocking traffic')\n\n    // Wait - session should remain \"healthy\" looking\n    await new Promise(r =\u003e setTimeout(r, 5000))\n\n    console.log('Session state:', {\n        closed: session.closed,\n        destroyed: session.destroyed,\n        queueSize: session.state?.outboundQueueSize\n    })\n\n    // Attempt write to trigger crash\n    const syms = Object.getOwnPropertySymbols(session)\n    const sockSym = syms.find(s =\u003e s.toString().includes('socket'))\n    const tls = sockSym ? session[sockSym] : session.socket\n\n    console.log('Writing to socket...')\n    tls.write('test')  // CRASHES HERE\n\n    // Cleanup (won't reach here)\n    execSync('pfctl -a zombie_test -F all')\n    server.kill()\n}\n\nmain()\n```\n\n## What happens\n\n1. Server and client establish HTTP/2 connection over TLS\n2. Firewall rule creates a \"black hole\" (packets dropped, no RST/FIN sent)\n3. Session continues to report `closed: false`, `destroyed: false`\n4. No error or close events fire\n5. Writing to the TLS socket triggers an assertion failure crash\n\n## Original discovery\n\nThis was originally discovered in a long-running production process (~2 days) where the TCP socket entered CLOSED state at the OS level (visible via `lsof`) but Node.js never received the close event.\n\n# How often does it reproduce? Is there a required condition?\n\n**100% reproducible** with the firewall-based test above.\n\nIn production, it's intermittent and requires:\n- Long-running HTTP/2 session\n- Network event that causes packet loss without proper TCP RST/FIN (NAT timeout, network partition, etc.)\n\n# What is the expected behavior? Why is that the expected behavior?\n\n1. **Close events should propagate** - When the OS-level TCP socket enters CLOSED state, this should propagate up through TLS and HTTP/2 layers, setting `session.closed = true` and emitting appropriate events\n\n2. **Write should fail gracefully** - Even if the zombie state occurs, calling `.write()` should return an error via callback, not crash with an assertion failure\n\n# What do you see instead?\n\n1. **Close event is lost** - All layers above TCP continue to report healthy status\n2. **Writes queue up** - `session.state.outboundQueueSize` grows indefinitely (we observed 2815 queued frames)\n3. **Crash on write** - Calling `.write()` crashes the process with assertion failure\n\n# Detailed debugging evidence\n\nWe attached Chrome DevTools inspector to the running process and gathered the following:\n\n**OS level** (via `lsof -p \u003cpid\u003e`):\n```\nnode \u003cpid\u003e 2128u IPv4 ... TCP ...:62689-\u003e...:https (CLOSED)\n```\n\nThe file descriptor 2128 is in **CLOSED** state at the OS level.\n\n**Node level** (via inspector):\n```javascript\nsessionInfo.session.closed      // false - thinks it's open!\nsessionInfo.session.destroyed   // false\nsessionInfo.session.connecting  // false\n\n// TLS socket also reports healthy:\nsocket.destroyed   // false\nsocket.readable    // true\nsocket.writable    // true\nsocket.readyState  // 'open'\n\n// But the outbound queue is stuck:\nsession.state.outboundQueueSize // 2815 frames queued!\nsessionInfo.pendingRejects.size // 3 requests waiting\n\n// The TLS socket's underlying TCP handle IS the CLOSED fd:\nsocket._handle._parent.fd       // 2128 (the CLOSED socket!)\n```\n\n**Ping test** (callback never fires):\n```javascript\nsessionInfo.session.ping((err, duration) =\u003e console.log(err, duration))\n// Returns true (ping \"sent\") but callback NEVER executes\n```\n\n**Fresh connection to same host works fine:**\n```javascript\nrequire('http2').connect('https://same-host.com').ping((e,d) =\u003e console.log(e,d))\n// -\u003e connected!\n// -\u003e ping: null 19.748583 (success, 20ms latency)\n```\n\nThis proves the server is reachable; only the cached zombie session is broken.\n\n# Crash output\n\nWe've observed two different assertion failures depending on the scenario:\n\n## Crash 1: From reproducible test (firewall black hole)\n\n```\n#  node[7869]: virtual void node::http2::Http2Session::OnStreamAfterWrite(node::WriteWrap *, int) at ../src/node_http2.cc:1741\n#  Assertion failed: is_write_in_progress()\n\n----- Native stack trace -----\n\n 1: 0x1043e8d1c node::Assert(node::AssertionInfo const\u0026)\n 2: 0x10601d89c node::http2::Http2Session::OnStreamAfterWrite(node::WriteWrap*, int) (.cold.1)\n 3: 0x10441a9a4 node::http2::Http2Session::ClearOutgoing(int)\n 4: 0x1044f8520 node::WriteWrap::OnDone(int)\n 5: 0x1044f8848 node::StreamReq::Done(int, char const*)\n 6: 0x104576f2c node::crypto::TLSWrap::InvokeQueued(int, char const*)\n 7: 0x104578c88 node::crypto::TLSWrap::OnStreamAfterWrite(node::WriteWrap*, int)\n...\n```\n\n## Crash 2: From production debugging (long-running zombie session)\n\n```\n#  node[71801]: virtual int node::crypto::TLSWrap::DoWrite(node::WriteWrap *, uv_buf_t *, size_t, uv_stream_t *) at ../src/crypto/crypto_tls.cc:1033\n#  Assertion failed: !current_write_\n\n----- Native stack trace -----\n\n 1: 0x102978d1c node::Assert(node::AssertionInfo const\u0026)\n 2: 0x1045de9bc node::crypto::TLSWrap::DoWrite(node::WriteWrap*, uv_buf_t*, unsigned long, uv_stream_s*) (.cold.8)\n 3: 0x102b09e24 node::crypto::TLSWrap::DoWrite(node::WriteWrap*, uv_buf_t*, unsigned long, uv_stream_s*)\n 4: 0x102a85198 node::StreamBase::Write(uv_buf_t*, unsigned long, uv_stream_s*, v8::Local\u003cv8::Object\u003e, bool)\n 5: 0x102a89288 int node::StreamBase::WriteString\u003c(node::encoding)1\u003e(v8::FunctionCallbackInfo\u003cv8::Value\u003e const\u0026)\n...\n\n----- JavaScript stack trace -----\n\n1: handleWriteReq (node:internal/stream_base_commons:62:21)\n2: writeGeneric (node:internal/stream_base_commons:148:15)\n3: Socket._writeGeneric (node:net:971:11)\n4: Socket._write (node:net:983:8)\n5: writeOrBuffer (node:internal/streams/writable:572:12)\n6: _write (node:internal/streams/writable:501:10)\n7: Writable.write (node:internal/streams/writable:510:10)\n```\n\nBoth crashes indicate internal state corruption in the TLS/HTTP2 layers when the underlying connection is silently broken.\n\n# Relationship to existing issues\n\nThis appears related to but distinct from previously fixed issues:\n\n- **#49307** (HTTP/2 segfault if underlying socket is unexpectedly closed) - Fixed via PR #49327. That fix handles the case where the socket is *destroyed*, but our scenario involves a socket in CLOSED state at the OS level where **no close/error event ever propagated to Node.js**.\n\n- **#30896** (TLS assertion error in DoWrite) - Same assertion failure `!current_write_`, attributed to memory exhaustion. Our case is not memory-related; it's a zombie session with a silently-closed TCP connection.\n\n- **[PR #18987](https://github.com/nodejs/node/pull/18987)** (Handle writes after SSL destroy more gracefully) - This fix handles writes after SSL is destroyed, but in our case the SSL layer doesn't know it should be destroyed.\n\nThe key difference in our scenario: **the OS socket is CLOSED but Node.js never received the close event**, leaving the TLS and HTTP/2 layers in an inconsistent state where they believe the connection is healthy.\n\n# Additional information\n\nThe zombie session persisted for an extended period (potentially hours) before we discovered it via debugging. All requests to the affected origin silently failed (queued but never sent), while requests to other origins continued to work normally.\n\nThe `session.state.outboundQueueSize` growing while `bytesWritten` remains static is a clear indicator of this zombie state, but there's no documented way to detect this condition - all public APIs (`session.closed`, `socket.writable`, etc.) report the connection as healthy.\n","author":{"url":"https://github.com/dglittle","@type":"Person","name":"dglittle"},"datePublished":"2026-01-07T07:48:30.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/61304/node/issues/61304"}