René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"oracle-gaps: an internal error (exit 3) in CI during the e2e stability rerun under load","articleBody":"## Summary\n\nIn CI on #48 (head 717d7cc), an `oracle-gaps` run exited 3 with an internal error. It was the stability rerun of the e2e unit `fixed seed 20261105: every outcome class`, on the edited source. Law 12 says that no legal history reaches an internal error, and a slow runner is a legal condition, so this is a defect. A rerun of the same job passed.\n\n- Run 43346685892, job 131627616481 (\"Gap report end-to-end oracle (fixed seed, 3/3)\"). The job took 14m0s, and the unit took 517 s; the same unit took 142 s in the rerun and 134 s on main.\n- The violation reported: `oracle-gaps.report violated at checkpoint \"stability\" (law: completion)`, `the CLI did not complete on the edited source (exit 3)`.\n- The CLI's stdout before the crash (as the e2e driver kept it):\n\n```text\nDifference: the CLI did not complete on the edited source (exit 3): ests/a.test.ts for 2s\nbusy: sut.ts:M170 in tests/a.test.ts for 2s\nbusy: sut.ts:M174 in tests/a.test.ts for 2s\nnot-infected L197 sut.ts:M161 ` > ` → ` >= `\nbranch-never-taken L200 sut.ts:M163 ` x > 4 ` → ` true `\ntimeout L200 sut.ts:M164 ` x > 4 ` → ` false `\nbusy: sut.ts:M174 in tests/a.test.ts for 4s\nbusy: sut.ts:M168 in tests/a.test.ts for 2s\nbusy: sut.ts:M177 in tests/a.test.ts for 2s\nbusy: sut.ts:M172 in tests/a.test.ts for 2s\nbusy: sut.ts:M175 in tests/a.test.ts for 2s\nnot-infected L200 sut.ts:M165 ` > ` → ` >= `\ntimeout L200 sut.ts:M166 ` out.push('s54'); ` → ` ; `\nbranch-never-taken L203 sut.ts:M167 ` x > 4 ` → ` true `\nbusy: sut.ts:M172 in tests/a.test.ts for 4s\nnot-infected L206 sut.ts:M173 ` > ` → ` >= `\n```\n\nThe internal error's stack is in the fixture's `out/error.log`. The e2e driver deleted that file with its scratch directory, and CI uploads no artifact, so the stack is lost.\n\n## Reproduction attempts (all exited 0)\n\nOn a kept fixture of the same constructed history, run locally under the e2e lock:\n- With tight limits, `--first-limit/--hang-limit/--mutant-limit` of 100/1000/1500 ms, 200/2000/3000 ms, and 50/600/900 ms.\n- With the e2e's own limits (1000/20000/15000 ms, `--contexts 4`), three rounds with 14 busy node loops and three rounds with 54 busy loops on an 18-core machine. These runs printed the same `timeout` lines that CI printed before the crash (M164, M166, M172, M174, M210). Those mutants are the hang witnesses, which are timeouts by design.\n\n## Hypotheses, none confirmed\n\n1. A runner path that throws an invariant error under heavy load, for example `runOwners`'s \"has no unit and no earlier kill\", or a batch whose run reports an error outside any test while copies are cut (see the e2e limits, \"a batch whose run reports an error outside any test reruns each copy alone\").\n2. #48's new wrappers in the capture (`capture-setup.mjs`): the clock and timer wrappers, the `Date` proxy, and the stack read on clock reads. They run in every copy, and a failure there under load could surface as an internal error. This is a real candidate, and #48 has not ruled it out.\n3. Something in the environment (descriptors, disk, memory) on the runner.\n\n## Diagnostic in place\n\nFrom #48 onward, when a run exits 3, the e2e driver copies the start of `error.log` into the observed stderr. The next \"did not complete (exit 3)\" difference will carry the stack.\n","author":{"url":"https://github.com/KyleAMathews","@type":"Person","name":"KyleAMathews"},"datePublished":"2026-11-13T03:19:20Z","url":"https://github.com/TanStack/oracles/issues/56"}