René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"SIGSEGV in `MarkingWorklists::Local::~Local()` and `CHECK(val->IsString() == true)` in `spawnSync`, on worker threads (v24.18.1, v24.19.0, darwin-arm64)","articleBody":"### Version\n\nv24.18.1 and v24.19.0: the official darwin-arm64 builds, signed by the Node.js Foundation (V8 13.6.233.17-node.50 and 13.6.233.17-node.51)\n\n### Platform\n\n```text\nDarwin 25.6.0 Darwin Kernel Version 25.6.0: Fri Jul 31 19:18:48 PDT 2026; root:xnu-12377.161.14~5/RELEASE_ARM64_T6020 arm64\n```\n\nmacOS 26.6.2 (25G83), Apple silicon (Mac14,6). The host name is left out of the `uname -a` line.\n\n### Subsystem\n\nworker_threads; V8 (garbage collection, WebAssembly code GC); child_process (`spawnSync` on a worker thread)\n\n### What steps will reproduce the bug?\n\nWe have no minimal reproduction. Every crash below came from Vitest 4.1.11 running one of two test suites in its `threads` pool (`worker_threads`), with Vite 8.2.0 and rolldown 1.2.1, and with `maxWorkers` 5 except where noted:\n\n- a suite of TypeScript tests for AWS CDK constructs (aws-cdk-lib 2.265.0), many of which build CDK apps inside the worker. Its size varied from 342 to 697 test files across the commits involved; the table under \"How often\" gives it at each crash;\n- a CLI's test suite (824 and 854 test files at its two crashes), some of which render Ink 7.1.1 components. Ink calls terminal-size 4.0.1, which runs `execFileSync('tput', …)` on the worker thread.\n\nBoth suites leave `isolate` at its default, so Vitest 4.1.11 runs each test file in a new worker and starts terminating the previous one without waiting for it: worker threads are created and torn down throughout a run.\n\nWe have not run these suites with `--no-sparkplug` (see #62393 under Additional information) or with `--no-wasm-code-gc` (see D).\n\n### How often does it reproduce? Is there a required condition?\n\nRarely: 12 crashes in the 883 runs that started at least one whole suite on one machine, about 1 in 74.\n\nOur test wrapper logs every run it starts: the command, the Node version, how the run ended, and the machine's average idle CPU over the run. From 2026-09-25 12:10 UTC, when that log begins, to the end of 2026-10-12 UTC, it records 5,545 runs of these two suites' tests. 14 of them ended in a native crash: 13 with SIGSEGV and 1 with the abort in C.\n\n- 12 came from the 883 runs that started a whole suite: 210 runs of every workspace's tests (`npm run test:all`, which includes both suites), 348 runs of the CDK suite alone and 306 of the CLI suite alone (`npm test`), 11 bare `vitest run`, and 8 runs of a suite started from a wrapper script or a multi-workspace command. 11 of the 883 stopped at a build-freshness check before any test ran, and 29 were stopped before they finished; on the runs that ran any test, the rate is 12 in 864. Ten of the 12 were in the CDK suite and two in the CLI suite.\n- 2 came from the 32 runs of the CDK suite's `deploy-tests` directory alone (222 test files at both crashes, which were consecutive runs of the same code).\n- None came from the other 4,587 runs, which ran part of a suite or used another Vitest config (206 did): 3,664 of them ran a single test file, and 4,375 ran ten or fewer. 43 more runs were cancelled before they started. Another 3,058 runs of 20 test files or fewer ran outside the wrapper, which logs them without their outcome, so they are not counted.\n- By version: 4 of the 87 whole-suite runs on v24.18.1 crashed, and 8 of the 796 on v24.19.0. Both `deploy-tests` crashes were on v24.19.0.\n- 10 of the 13 crashed runs with a CPU reading averaged under 5% idle over the run. So did 685 of the 870 whole-suite runs with a reading (79%), so load does not set the crashed runs apart.\n\nWhich signature each crash was:\n\n- The ten crashes up to 2026-10-05 left no crash report that we still have: the oldest report on this machine is from 2026-10-07 UTC. So their signatures are unknown, except C's, whose assertion and stack are in its run's output.\n- The four crashes on 2026-10-11 and 2026-10-12 all left a trace in macOS's crash reports: three are A and one is the stack of #62393. The reports for two of the A crashes and for the #62393 one are in https://gist.github.com/peoram/cd10c877f515c51f2476ce6015363ef7. The third A crash is described under A below.\n\nTwo more crashes are not in these counts:\n\n- D, below, happened on 2026-09-24 UTC, before the log began.\n- One SIGSEGV killed the CDK suite (697 test files) 18 seconds into a run on a second machine (Mac mini, Apple M2, macOS 26, v24.18.1, the same `threads` pool with 6 Vitest workers) on 2026-10-12 UTC. We have not retrieved its crash report, and the run's output ends without a stack.\n\n\u003cdetails>\n\u003csummary>Every crash, with the suite's size at the time\u003c/summary>\n\nTimes are UTC. The crash reports' file names are in local time (UTC+11:00).\n\n| Crash | When | Node | Suite (test files) | Run | Ended | Signature |\n|---|---|---|---|---|---|---|\n| – | 09-24, about 17:09 | v24.18.1 | CDK (342) | `test:all` | V8 fatal, exit 133 | D (run output) |\n| 1 | 09-26 07:41 | v24.18.1 | CLI (824) | suite alone | SIGSEGV | unknown |\n| 2 | 09-28 02:41 | v24.19.0 | CDK (468) | `test:all` | SIGSEGV | unknown |\n| 3 | 09-28 05:51 | v24.19.0 | CDK (476) | `test:all` | SIGSEGV | unknown |\n| 4 | 09-29 01:07 | v24.18.1 | CDK (479) | suite alone | SIGSEGV | unknown |\n| 5 | 09-29 06:50 | v24.19.0 | CLI (854) | `test:all` | abort, exit 134 | C (run output) |\n| 6 | 09-30 09:11 | v24.19.0 | CDK (628) | `test:all` | SIGSEGV | unknown |\n| 7 | 09-30 17:45 | v24.19.0 | CDK (496) | suite alone | SIGSEGV | unknown |\n| 8 | 10-05 02:33 | v24.18.1 | CDK (651) | `test:all` | SIGSEGV | unknown |\n| 9 | 10-05 17:44 | v24.19.0 | CDK (222 of 682) | `deploy-tests` only | SIGSEGV | unknown |\n| 10 | 10-05 18:08 | v24.19.0 | CDK (222 of 682) | `deploy-tests` only | SIGSEGV | unknown |\n| 11 | 10-11 06:11 | v24.18.1 | CDK (689) | suite alone | SIGSEGV | A (`node-2026-10-11-171152.000.ips.txt`) |\n| 12 | 10-11 14:08 | v24.19.0 | CDK (692) | suite alone | SIGSEGV | A (`node-2026-10-12-010821.ips.txt`) |\n| 13 | 10-11 18:10 | v24.19.0 | CDK (697) | suite alone | SIGSEGV | #62393 (`node-2026-10-12-051041.ips.txt`) |\n| 14 | 10-12 11:08 | v24.19.0 | CDK (694) | suite alone | SIGSEGV | A (macOS repeat count on crash 12's report) |\n| – | 10-12 23:42 | v24.18.1 | CDK (697) | suite alone, second machine | SIGSEGV | unknown (crash report not retrieved) |\n\n\u003c/details>\n\n### What is the expected behavior? Why is that the expected behavior?\n\nThe process does not crash.\n\n### What do you see instead?\n\nEach signature below is on a `worker_threads` thread: each stack runs from `node::worker::Worker::Run()`, and in the crash reports the faulting thread is named `WorkerThread`.\n\n**A. SIGSEGV in `MarkingWorklists::Local::~Local()` while starting incremental marking: three times, all in the CDK suite (crashes 11, 12 and 14 in the table)**\n\nCrash reports for the first two are in https://gist.github.com/peoram/cd10c877f515c51f2476ce6015363ef7: `node-2026-10-11-171152.000.ips.txt` (v24.18.1) and `node-2026-10-12-010821.ips.txt` (v24.19.0). Both show `EXC_BAD_ACCESS (SIGSEGV)`, `KERN_INVALID_ADDRESS at 0x0000000000000089`, with the same frames and offsets (argument lists shortened to `(...)`):\n\n```\n0  v8::internal::MarkingWorklists::Local::~Local() + 16\n1  v8::internal::MarkCompactCollector::StartMarking(std::__1::shared_ptr\u003cheap::base::IncrementalMarkingSchedule>) + 1308\n2  v8::internal::IncrementalMarking::StartMarkingMajor() + 412\n3  v8::internal::IncrementalMarking::Start(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason) + 1912\n4  v8::internal::Heap::StartIncrementalMarking(...) + 244\n5  v8::internal::IncrementalMarkingJob::Task::RunInternal() + 256\n6  node::PerIsolatePlatformData::RunForegroundTask(...) + 224\n7  node::PerIsolatePlatformData::FlushForegroundTasksInternal() + 960\n8  uv__async_io + 304\n9  uv__io_poll + 1432\n10 uv_run + 568\n11 node::SpinEventLoopInternal(node::Environment*) + 248\n12 node::worker::Worker::Run() + 2304\n13 node::worker::Worker::StartThread(...)::$_0::__invoke(void*) + 80\n14 _pthread_start + 136\n15 thread_start + 8\n```\n\nThe two processes had 49 and 48 threads: 6 and 5 of them named `WorkerThread`, and 18 named `rolldown-worker` in each.\n\nThe third (crash 14, v24.19.0) has no report of its own. About 4 seconds after that run was killed by SIGSEGV, the modification time of `node-2026-10-12-010821.ips` (crash 12's report) moved to that moment, and that report now carries a `Multiple` extended attribute set to 1, which the copy in the gist does not carry. We read that as macOS counting a repeat of crash 12 instead of writing a new report. No other run in our log crashed within 12 hours of crash 14. We count crash 14 as A on that evidence, but we have no stack for it.\n\n**C. `CHECK(val->IsString() == true)` in `StringBytes::Write`, from `spawnSync` on a worker thread: once, v24.19.0, 2026-09-29, in the CLI suite (854 test files), from the run's output (the run exited 134)**\n\nPaths are shortened to the package, and the native stack is abridged.\n\n```\n  #  node[51777]: static size_t node::StringBytes::Write(Isolate *, char *, size_t, Local\u003cValue>, enum encoding) at ../src/string_bytes.cc:283\n  #  Assertion failed: val->IsString() == true\n\n----- Native stack trace -----\n\n 1: node::Assert(node::AssertionInfo const&)\n 2: node::StringBytes::Write(v8::Isolate*, char*, unsigned long, v8::Local\u003cv8::Value>, node::encoding) (.cold.2)\n 3: node::StringBytes::Write(v8::Isolate*, char*, unsigned long, v8::Local\u003cv8::Value>, node::encoding)\n 4: node::SyncProcessRunner::CopyJsStringArray(v8::Local\u003cv8::Value>, char**)\n 5: node::SyncProcessRunner::ParseOptions(v8::Local\u003cv8::Value>)\n 6: node::SyncProcessRunner::TryInitializeAndRunLoop(v8::Local\u003cv8::Value>)\n 7: node::SyncProcessRunner::Run(v8::Local\u003cv8::Value>)\n 8: node::SyncProcessRunner::Spawn(v8::FunctionCallbackInfo\u003cv8::Value> const&)\n    …\n60: node::worker::MessagePort::OnMessage(node::worker::MessagePort::MessageProcessingMode)\n    …\n65: node::worker::Worker::Run()\n\n----- JavaScript stack trace -----\n\n1: spawnSync (node:internal/child_process:1146:29)\n2: spawnSync (node:child_process:928:24)\n3: execFileSync (node:child_process:971:15)\n4: exec (terminal-size/index.js:10:2)\n5: tput (terminal-size/index.js:115:19)\n6: terminalSize (terminal-size/index.js:95:22)\n7: getWindowSize (ink/build/utils.js:13:26)\n8: Ink (ink/build/ink.js:246:34)\n9: ink/build/render.js:22:59\n10: getInstance (ink/build/render.js:50:29)\n```\n\nThe caller is terminal-size 4.0.1: `execFileSync('tput', ['cols'], {encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'], timeout: 500, env: {TERM: 'dumb', ...process.env}})`. `normalizeSpawnArguments` hands the binding `args` as `['tput', 'cols']` and builds each `envPairs` entry with a template literal, so both arrays hold only strings. In v24.19.0, `SyncProcessRunner::CopyJsStringArray` stores every element in `values` as a `String` in its first loop (the element itself once `IsString()` holds, or else its `ToString()`), and its second loop passes `values[i]` to `StringBytes::Write`, where the CHECK failed. So the object behind one of those handles was a string in the first loop and not one in the second. From a single occurrence we cannot tell what changed it.\n\n**D. V8 fatal `Check failed: end > addr.` in `ThreadIsolation::LookupJitPage`, while freeing dead WebAssembly code on a worker thread: once, v24.18.1, 2026-09-24, from the run's output (the run exited 133)**\n\nThe CDK suite (342 test files at that commit, with Vitest's default worker count) stopped before it reported any test file. A re-run of the suite alone passed. Addresses and the binary's path are removed from each frame, two long argument lists are shortened to `…`, and the stack is abridged: frames 14 to 18 are bare addresses (the WebAssembly code between the two wrapper frames), frames 21 to 69 are JavaScript, builtin, microtask and callback frames, and frames 39 and 40 were not captured.\n\n```\n#\n# Fatal error in , line 0\n# Check failed: end > addr.\n#\n#\n#\n#FailureMessage Object: 0x176953188\n----- Native stack trace -----\n\n 1: node::NodePlatform::GetStackTracePrinter()::$_0::__invoke()\n 2: V8_Fatal(char const*, ...)\n 3: v8::internal::ThreadIsolation::LookupJitPage(unsigned long, unsigned long)\n 4: v8::internal::ThreadIsolation::UnregisterWasmAllocation(unsigned long, unsigned long)\n 5: v8::internal::wasm::WasmCodeAllocator::FreeCode(v8::base::Vector\u003cv8::internal::wasm::WasmCode* const>)\n 6: v8::internal::wasm::WasmImportWrapperCache::Free(std::__1::vector\u003cv8::internal::wasm::WasmCode*, std::__1::allocator\u003cv8::internal::wasm::WasmCode*>>&)\n 7: v8::internal::wasm::WasmEngine::FreeDeadCodeLocked(…)\n 8: v8::internal::wasm::WasmEngine::FreeDeadCode(…)\n 9: v8::internal::wasm::WasmCode::DecrementRefCount(v8::base::Vector\u003cv8::internal::wasm::WasmCode* const>)\n10: v8::internal::wasm::WasmCodeRefScope::~WasmCodeRefScope()\n11: v8::internal::Runtime_TierUpWasmToJSWrapper(int, unsigned long*, v8::internal::Isolate*)\n12: Builtins_WasmCEntry\n13: Builtins_WasmToJsWrapperCSA\n    …\n19: Builtins_JSToWasmWrapperAsm\n20: Builtins_JSToWasmWrapper\n    …\n70: node::worker::MessagePort::OnMessage(node::worker::MessagePort::MessageProcessingMode)\n71: uv__async_io\n72: uv__io_poll\n73: uv_run\n74: node::SpinEventLoopInternal(node::Environment*)\n75: node::worker::Worker::Run()\n76: node::worker::Worker::StartThread(v8::FunctionCallbackInfo\u003cv8::Value> const&)::$_0::__invoke(void*)\n77: _pthread_start\n78: thread_start\n```\n\nFrames 4 to 11, from `UnregisterWasmAllocation` down to `Runtime_TierUpWasmToJSWrapper`, are the frames of the worker-thread stack in #66366, but a different CHECK fails: `end > addr` in `LookupJitPage` here, and `jit_page_->allocations_.erase(addr) == 1` in `JitPageReference::UnregisterAllocation` there. #66366 traces its CHECK to an import wrapper that is freed twice, fixed in V8 by 9b8ca54d5a. Its backport, #66376, is open, and the 24.22.0 release proposal does not include it. We do not know whether D is the same bug, which WebAssembly module was running, or whether macOS wrote a crash report for it (none from that date is left). We have seen D only once.\n\n### Additional information\n\n- **#62393.** The third crash report, `node-2026-10-12-051041.ips.txt` (crash 13 in the table: v24.19.0, the CDK suite at 697 test files), shows the stack reported in #62393: the same frames from `ClearStaleLeftTrimmedPointerVisitor::VisitRootPointers` down to `Builtins_BaselineOutOfLinePrologue + 116`, at `0xe`, on a `WorkerThread`. It is in the gist only for comparison with A. We do not know whether A or C shares its cause. We will add here whether either recurs on v24.22.0, whose release proposal (#66667) includes #65753, the backport of the V8 fix for #62393.\n- **vitest-dev/vitest#11291** (Vitest 4.1.10, Vite 8.1.4, `pool: 'threads'`, macOS 15 arm64) lists two signatures from \"~15 crash reports on this machine alone\", each with `WorkerThread` as the faulting thread. One is A: `IncrementalMarkingJob::Task::RunInternal` → `MarkCompactCollector::StartMarking` → `MarkingWorklists::Local::~Local`, a near-null dereference at `0x89`. The other is at worker teardown: `Isolate::Deinit` → `Heap::StartTearDown` → `CppHeap::StartDetachingIsolate` → `Heap::CollectGarbage` → `GlobalHandles::InvokeFirstPassWeakCallbacks`. It names the same two `.node` addons and 18 `rolldown-worker` threads. It says that Node 24.19.0 and 26.8.1 both reproduce, and that the crash was also seen on GitHub Actions' arm64 macOS runners, without saying which signature each version or machine gave. Its author later wrote that their machine was \"under really high load\" when they hit it, and that they could not get their reproduction to work. It was closed on 2026-09-24 without one.\n- **Native modules.** In each of the three crash reports, the only `.node` images on any thread's stack are rolldown 1.2.1's binding (whose own pool had 18 to 21 threads named `rolldown-worker`) and fsevents. A report's `usedImages` lists only the images that some thread's frames reference (none of the three lists libc++ or libsystem_c, both of which the node binary loads), so these processes may also have loaded other native modules.\n- **Crash reports** (https://gist.github.com/peoram/cd10c877f515c51f2476ce6015363ef7). The three crash reports as macOS wrote them, except that every file path and the identifiers of the machine and of the crash (the incident id, the crash-reporter key, and the boot and sleep/wake session ids) read `\u003cremoved>`. The binary images' UUIDs are kept: `55FD9EB9-D7D1-3FB3-829D-EE7233BE3116` is the official v24.18.1 build and `5CF2F254-9668-37EE-9BAE-7B6F0ECE16F6` the official v24.19.0 build.\n- We are moving these suites to Vitest's `forks` pool (one child process per worker), with a reporter that names each test file whose worker was lost, and the worker's pid where the worker recorded one.\n","author":{"url":"https://github.com/peoram","@type":"Person","name":"peoram"},"datePublished":"2026-10-13T22:22:58Z","url":"https://github.com/nodejs/node/issues/66714"}