René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Intermittent fatal CHECK in compiler/heap-refs on v24.x — request to backport V8 14.6's JSHeapBroker thread-safety fixes","articleBody":"### Version\n\nv24.11.1 (V8 `13.6.233.10-node.28`)\n\n### Platform\n\n```\nLinux x86_64, kernel 6.1.134-152.225.amzn2023.x86_64\nlibc: musl (Alpine-based container image)\ncgroup v2, CPU quota = 12 cores, memory limit 24576 MiB, swap off\n```\n\n### Subsystem\n\ndeps/v8 (compiler)\n\n### What steps will reproduce the bug?\n\nI don't have a minimal reproduction, and I want to be upfront about that.\n\nThe workload is a Vitest 5 run in `forks` pool with `isolate: true`: one short-lived `node` child process per test file, 1579 children per run, 32 concurrent against a 12-core quota, ~740 s wall clock. Each child spends most of its life loading and evaluating modules (transform 61%, import 27%, tests 5%), so it is optimizing-compiler-heavy startup, repeated 1579 times. The children are plain JS workloads — component/unit tests on jsdom. No `node:http2`, no network server, no HTTP/2 client anywhere in them.\n\nOccasionally one child dies on a signal. The rest of the run is green.\n\n### How often does it reproduce? Is there a required condition?\n\n**7 runs with at least one crash out of 87 runs (~8%)**, i.e. roughly 1 in 20000 child processes. One run produced three separate events.\n\nCPU oversubscription appears to be required: I have never reproduced it on an idle machine, and a fork-storm stress test (tens of thousands of bare `node -e ''` spawns) never hit it — the child has to actually be compiling JavaScript. Anyone trying to reproduce should think in terms of *total* contention (host baseline + concurrent children ÷ available cores); the `loadavg` in my logs is sampled at run start and excludes the load the run itself creates.\n\n### What is the expected behavior?\n\nThe child process should not die.\n\n### What do you see instead?\n\nThree signatures so far, all in the same subsystem:\n\n- **A.** `Check failed: (data_) != nullptr.` with SIGTRAP\n- **B.** `Check failed: IsJSFunction().` with SIGTRAP\n- **C.** plain SIGSEGV with no V8 output at all\n\nSignature A verbatim; B is identical in shape:\n\n```\n#\n# Fatal error in , line 0\n# Check failed: (data_) != nullptr.\n#\n#\n#\n#FailureMessage Object: 0x7fa7fa0b27e0\n----- Native stack trace -----\n\n```\n\nThe parent reports `Worker exited unexpectedly with signal \u003cSIG>`. SIGTRAP rather than SIGABRT is consistent with `--hard-abort` (default true).\n\nAt `13.6.233.10` both CHECKs sit in `src/compiler/heap-refs.h`: A is `CHECK_NOT_NULL(data_)`, which appears at [360](https://chromium.googlesource.com/v8/v8/+/refs/tags/13.6.233.10/src/compiler/heap-refs.h#360) and [386](https://chromium.googlesource.com/v8/v8/+/refs/tags/13.6.233.10/src/compiler/heap-refs.h#386) and is reached through the latter in a release build, and B is the `CHECK(Is##Name())` in [`DEFINE_REF_CONSTRUCTOR`](https://chromium.googlesource.com/v8/v8/+/refs/tags/13.6.233.10/src/compiler/heap-refs.h#514). The null check runs first, so **B means a non-null `ObjectData` of the wrong type**.\n\n### Additional information\n\n**What I'm asking for.** Please consider backporting these to `v24.x`:\n\n| CL | title | bug |\n|---|---|---|\n| [`c3553509f`](https://chromium-review.googlesource.com/c/v8/v8/+/7460949) | `[compiler] Thread-safe IsArrayOrObjectPrototype` | [467311868](https://issues.chromium.org/issues/467311868) |\n| [`b1c2cda9b`](https://chromium-review.googlesource.com/c/v8/v8/+/7483030) | `[compiler] Thread-safe MapRef::GetConstructor,GetBackpointer` | [467311868](https://issues.chromium.org/issues/467311868) |\n| [`0bb186fd9`](https://chromium-review.googlesource.com/c/v8/v8/+/8334889) | `[objects][compiler] Release-store the initial map into the prototype tuple` | [542923494](https://issues.chromium.org/issues/542923494) |\n\nAll three are already-accepted upstream fixes for unsynchronized reads in `JSHeapBroker`, which is exactly where we crash; the CLs carry the mechanism. Signature B — non-null, wrong type — matches both shapes they fix: a slot holding either a `JSFunction` or a `Map` read mid-flip (`b1c2cda9b`), and a map observed before `Factory::InitializeMap`'s stores are visible (`0bb186fd9`).\n\nVersion containment, checked with `gh api repos/v8/v8/compare/\u003ctag>...\u003csha>`:\n\n| Node branch | V8 | has `c3553509f` / `b1c2cda9b` | has `0bb186fd9` |\n|---|---|---|---|\n| v24.x | 13.6.233.17 | no | no |\n| v25.x | 14.1.146.11 | no | no |\n| v26.x / main | 14.6.202.34 | **yes** | no |\n\nStaying on 24.x doesn't help: `13.6.233.10 → 13.6.233.17` adds 5 substantive cherry-picks, none touching `heap-refs`, the broker, or handles.\n\nI'm not asking anyone to debug my workload — the report is that a supported LTS line ships a V8 predating this cluster of fixes. Cost disclosed up front: `b1c2cda9b` widens a public return type to `OptionalObjectRef` and updates four caller files that have moved a lot since 13.6, so it isn't a mechanical cherry-pick. \"Too invasive for an LTS V8\" is a legitimate answer; I'd then suggest the class is still worth recording against 24.x, since the failure mode is a hard process death with no diagnosable output.\n\n**What distinguishes this from #64841.** Same shape — intermittent Linux crash, `JSHeapBroker` involved, contention required, `--no-maglev` making it go away — but that one resolved to invalid nghttp2 accesses and is now scoped to HTTP/2, which this workload never touches, so I'm deliberately not commenting there.\n\nThe distinguishing evidence is that for some of our crashes there was no third-party native code in the process at all. I hooked `process.dlopen` from a `--require` shim inherited by the forked children and enumerated a full run: the children load at most `bufferutil` (884 of 1579) and `bigint-buffer` (18), while the bundler's native binding stays in the parent because children receive transformed modules over IPC. Of the six distinct test files that have crashed, **two ran in processes with no addon at all**, and the most recent crash came from a run where `bufferutil` and `utf-8-validate` were blocked at `require()` time. Node's own built-in native code remains in scope — I can't turn that off to test it.\n\n\u003cdetails>\n\u003csummary>Ruled out, two observability notes, and the mitigation in flight\u003c/summary>\n\n**Ruled out.** Container OOM: cgroup `memory.peak` 10844/24576 MiB, `memory.events` `oom_kill=0`, swap off, `pids` 385 of 37827 — and it would be SIGKILL, with a V8 heap OOM being SIGABRT. musl's 128 KiB default thread stack: concurrent compilation runs on Node's platform workers, which `node_platform.cc` creates via `uv_thread_create` → libuv's `uv__thread_stack_size()` (`RLIMIT_STACK`, else 2 MB), so the musl default never applies; verified in `node:24.11.1-alpine`, where V8's own `regress-crbug-388320179.js` passes a 2000-link prototype chain that a 128 KiB stack would have killed at ~340. Stale V8 code cache: the runner deletes `NODE_COMPILE_CACHE` for the children and nothing sets it.\n\n**Observability, independent of the root cause.** The release `V8_Fatal` path passes an empty file and line 0, so the assertion text is the only usable signal — unique for both of my signatures, but it would not be for a commoner `CHECK`. And `----- Native stack trace -----` prints its header and then nothing on musl, because Node's backtrace depends on glibc's `execinfo.h`; a reader cannot distinguish \"no frames available on this libc\" from \"the process died before it could walk the stack\". A one-line explanation instead of an empty section would save people real time, which also bears on the Alpine tier-2 discussion in #62764.\n\n**Mitigation in flight.** `--no-concurrent-recompilation` for the children, both as mitigation and as the only discriminating experiment I have left: a recurrence under that flag would falsify the attribution above in a single run. I'll report back either way, though a clean result is much weaker evidence — at an 8% per-run base rate it takes ~28 consecutive clean runs to reach p≈0.1.\n\n\u003c/details>\n\n\u003csub>Body condensed at maintainer request; the original long-form version remains in this issue's edit history.\u003c/sub>\n","author":{"url":"https://github.com/Feahter","@type":"Person","name":"Feahter"},"datePublished":"2026-09-19T06:02:16Z","url":"https://github.com/nodejs/node/issues/66126"}