René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"fix(nodejs): `--test-force-exit` silently drops unit tests (fewer tests reported, exit 0 when the rest pass)","articleBody":"## Summary\n\nEvery unit-test script in `package.json`, plus the TIA step in `quality.yml`, runs `node --test --test-force-exit`. On Node 24 (the version CI uses; upstream reports the same on 22 and 26) this drops tests from the results without any warning. There are two separate mechanisms:\n\n1. The child's report stream gets cut off (random). Each test file runs in a child process that streams its results to the runner over a stdout pipe. When the child finishes, `--test-force-exit` calls `process.exit()` while some of that pipe output is still queued. (On POSIX, pipe writes from `process.stdout` are asynchronous; on Windows they are synchronous, so this is a Linux/macOS problem.) The tail of the file's results is lost. The runner prints a smaller `ℹ tests` total and exits 0. This is a Node bug (https://github.com/nodejs/node/issues/64833, fix proposed in https://github.com/nodejs/node/pull/64875, both still open). It reproduces on Node v24.22.0 without tsx or any repo code.\n2. Tests registered after a top-level `await` never run (deterministic for the affected files). If a file registers a `test()`, then hits a top-level `await` (for example `await createChatPipelineHarness(...)`), and registers more tests after it, the first test can finish while the module is still suspended. At that point \"all known tests have finished\", so the force-exit path exits the child before the remaining tests exist. That matches the documented behaviour of `--test-force-exit`, but the result is that those tests never run and nothing reports it.\n\nIn mechanism 1, a failure in the dropped tail still makes the child exit non-zero, so the run goes red but the failure detail can be missing. In mechanism 2 the dropped tests never execute, so their failures can't turn CI red.\n\n## Environment\n\n- Base commit `5b3be70ba8` (release/v3.8.53 line), clean checkout\n- Node v24.22.0, Linux x64\n- tsx 4.23.13\n- CI: Node 24 (`.nvmrc`, `CI_NODE_VERSION`), ubuntu-latest; `engines` also allows `>=22.22.2 \u003c23`\n\n## Reproduction\n\n### A. Repo files (mechanism 1)\n\n```bash\nF=\"tests/unit/i18n-kilo-pass-keys-12561.test.ts tests/unit/i18n-missing-keys-12272.test.ts\"\nfor i in $(seq 1 40); do\n  DISABLE_SQLITE_AUTO_BACKUP=true node --max-old-space-size=8192 --import tsx/esm \\\n    --import ./open-sse/utils/setupPolyfill.ts --import ./tests/_setup/isolateDataDir.ts \\\n    --test --test-force-exit --test-concurrency=4 $F 2>&1 | grep '^ℹ tests'\ndone | sort | uniq -c\n```\n\nExpected: 137 every time (69 + 68). Observed:\n\n| Mode | Runs | `ℹ tests` | Exit code |\n|---|---|---|---|\n| `--test-force-exit` | 40 | 137 in 31 runs; 78, 101, 103, 103, 111, 114, 118, 122, 135 in the other 9 | 0 in all 40, with 0 fail and 0 cancelled |\n| without `--test-force-exit` | 40 | 137 in all 40 | 0 |\n| `--test-force-exit` + child stdout set to blocking (see below) | 40 | 137 in all 40 | 0 |\n\nIn a short run, the spec output stops partway through the `locale:` subtests, for example after `locale: mt`. The parent test shows `▶` with no `✔`, and then the summary prints. A `process.exit` tracer preloaded into the children showed that each child exits through the runner's own force-exit path (`Test.run` → `process.exit()`). It also showed whether `process.stdout.writableLength` was non-zero at that moment. In 4 of 30 traced runs a child exited with bytes still queued on stdout (472, 13906, 23159 and 60597). The three with 13 KB or more queued were the runs that came up short (128, 122 and 98 tests); the 472-byte run still reported 137. Every run with nothing queued reported 137.\n\n### B. Minimal repro, no tsx and no repo code (mechanism 1)\n\n```bash\nmkdir fe-repro && cd fe-repro\ncat > f1.test.mjs \u003c\u003c'JS'\nimport test from 'node:test';\nimport assert from 'node:assert/strict';\ntest('parent with many awaited subtests', async (t) => {\n  for (let i = 0; i \u003c 400; i++) {\n    await t.test(`subtest ${i} ${'x'.repeat(200)}`, () => assert.ok(true, 'y'.repeat(500)));\n  }\n});\nJS\nfor k in 2 3 4; do cp f1.test.mjs f$k.test.mjs; done\nfor i in $(seq 1 15); do node --test --test-force-exit --test-concurrency=4 f*.test.mjs 2>&1 | grep '^ℹ tests'; done | sort | uniq -c\n```\n\nExpected: 1604. Observed with `--test-force-exit`: 14 of 15 runs were short (1524–1593), all with exit 0. Without the flag: 1604 in 15 of 15 runs. With `--test-isolation=none` (no pipe): 1604 in 6 of 6 runs.\n\n### C. Tests after a top-level `await` (mechanism 2)\n\n```bash\nnode --import tsx/esm --import ./open-sse/utils/setupPolyfill.ts --import ./tests/_setup/isolateDataDir.ts \\\n  --test --test-force-exit tests/unit/embed-ws-host-empty-messages-5110.test.ts   # ℹ tests 1\nnode --import tsx/esm --import ./open-sse/utils/setupPolyfill.ts --import ./tests/_setup/isolateDataDir.ts \\\n  --test tests/unit/embed-ws-host-empty-messages-5110.test.ts                     # ℹ tests 3\n```\n\n`#5110-1` is registered at line 20. Line 51 is `const harness = await createChatPipelineHarness(...)`, and both `#5110-4` tests come after it. With force-exit they never run. Base has 7 affected files that lose 22 tests every time. This was measured with mechanism 1 neutralised, comparing force-exit against no force-exit:\n\n| File | Tests with force-exit | Tests without force-exit |\n|---|---|---|\n| tests/unit/cli-tools-crush.test.ts | 3 | 11 |\n| tests/unit/resolve-proxy-family.test.ts | 2 | 6 |\n| tests/unit/bai-provider.test.ts | 6 | 8 |\n| tests/unit/dgrid-provider.test.ts | 5 | 7 |\n| tests/unit/qiniu-provider.test.ts | 5 | 7 |\n| tests/unit/responses-case-insensitive-combo-guard.test.ts | 4 | 6 |\n| tests/unit/embed-ws-host-empty-messages-5110.test.ts | 1 | 3 |\n\nThe dropped tests pass when they do run. An AST scan of the 7,694 files under `tests/unit` finds 49 with the same shape: a top-level test, then a top-level `await`, then more top-level tests. Only the 7 above lost tests in our runs; the other 42 lost none, because the `await` settled before the first test finished. Whether they drop tests depends on that timing.\n\n### D. Larger sample\n\nOn the 103-file i18n/translator batch from the same base, 3 runs per mode:\n\n| Mode | `ℹ tests` | Wall time |\n|---|---|---|\n| `--test-force-exit` | 1248 / 1213 / 1262 | 62–68 s |\n| `--test-force-exit` + blocking child stdout | 1262 / 1262 / 1262 | 61–62 s |\n| without `--test-force-exit` | 1264 / 1264 / 1264 | 90–100 s |\n\nThe remaining 2 tests are the `#5110-4` pair (mechanism 2). The extra ~30 s without force-exit is explained under \"Why the flag is there\" below.\n\n## Affected configuration\n\n- `package.json:140-150` and `:313`: `test`, `test:unit` (which also chains `test:unit:serial`), `test:unit:ci`, `test:unit:ci:shard` (used by the 8-shard unit job in `ci.yml`, the 4-shard `fast-unit` job in `quality.yml`, and `nightly-release-green.yml` / `nightly-compat.yml`), `test:unit:fast`, `test:unit:shard:1/2`, `test:unit:serial`. Each script has 2–3 `node --test --test-force-exit` invocations.\n- `package.json:155, 272-281, 288, 312`: property, integration, combo-matrix, services, boundary, heap, chaos, system and coverage-runner scripts. (`test:property` does not load `tests/_setup/isolateDataDir.ts`.)\n- `.github/workflows/quality.yml:381` and `:384`: the TIA impacted-tests step (REST and DASH invocations).\n\n## Why the flag is there\n\nWithout force-exit, some test children stay alive after their last test:\n\n- The call-log artifact worker: `src/lib/usage/callLogArtifactWriter.ts` keeps its `worker_threads` Worker ref'd while idle and terminates it only after a 30 s idle timer (`IDLE_TIMEOUT_MS = 30_000`, line 11; the timer itself is unref'd at line 127, so it is the idle Worker that holds the event loop). Shortening that timer to 100 ms alone cut `embed-ws-host-empty-messages-5110` from 35.8 s to 3.8 s without force-exit. Any file that writes a call log stays alive for about 30 s after it finishes, unless it calls `closeCallLogSaves()` or `closeCallLogArtifactWriter()`. `tests/integration/_chatPipelineHarness.ts`'s `cleanup()` only awaits `waitForCallLogSaves()`. In the 103-file batch, 4 files took about 30 s longer each without force-exit, and apart from stdout/stderr, the only ref'd handle left was that worker's MessagePort. 41 unit files use this harness.\n- tsx disk cache writes: tsx's disk cache stores each transform with an un-awaited `fs.promises.writeFile` and ignores errors. If force-exit calls `process.exit()` while those writes are still in flight, they are cut off and can leave 0-byte entries in `os.tmpdir()/tsx-\u003cuid>`. A development machine had 20,830 of them among 133,963 entries; a full disk is another possible cause. Under force-exit, a test child has occasionally hung during exit (`modality-bridge-video-i18n.test.ts`). The stuck child had a defunct esbuild child and open write fds on 0-byte tsx cache files. This was seen while the disk was full; with free disk it did not reproduce (0/20 runs with force-exit, 0/20 without).\n\n## Proposed fix direction\n\n1. Short term, keep force-exit but stop losing data:\n   - In the shared test preload `tests/_setup/isolateDataDir.ts`, which every affected script already imports, make the child's stdout pipe blocking. Only do this inside test-runner children (`NODE_TEST_CONTEXT` set): `process.stdout._handle?.setBlocking?.(true)`. This is the same approach as the proposed upstream fix. Locally it removed mechanism 1 completely: 40/40 and 25/25 full counts, and no measurable change in wall time.\n   - Move the top-level `await` in the 7 files above so it comes before the first `test()`, or into a `before()` hook. Add a cheap static check so new files don't reintroduce this pattern.\n2. Follow-up, remove `--test-force-exit`. Unref the call-log worker while it is idle (ref it while a write is in flight), and have the chat pipeline harness close the writer in `cleanup()`. Then run the suite without force-exit to find any remaining handles before dropping the flag. A naive full unref breaks tests that are still waiting for a worker reply, so ref-while-active is required.\n3. Remove the stdout workaround once a Node 24.x release ships the upstream fix.\n","author":{"url":"https://github.com/HouMinXi","@type":"Person","name":"HouMinXi"},"datePublished":"2026-10-24T12:31:52Z","url":"https://github.com/cheaper-inference/OmniRoute/issues/18045"}