René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"self-oracle timeout law: 'timeout presentation was not exercised' on seed -1999165303","articleBody":"The random-seed counterexample from PR #7 CI reproduces deterministically and is not a timing problem. The self-oracle's timeout law fires because it assumes that every slow spec in a plan runs its driver. A spec whose `owns` declaration fails to load never runs the driver, so the law reports a missing timeout presentation instead of the correct registration failure. This is a self-oracle law defect, not a framework defect: the production check at `src/oracle-property.ts:828` rejects the declaration as the registration law says.\n\n## Replay\n\n```sh\nORACLE_SEED=-1999165303 ORACLE_PATH=0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0 ORACLE_PROPERTY='self-oracle.framework' ORACLE_UNIT='random seed: generated' vitest run -t '^self-oracle\\.framework(?: | > )random seed: generated$'\n```\n\nI ran it through `packages/oracles` with `--config vitest.oracle-framework.config.mjs --maxWorkers=2`, which is how the CI shard runs it.\n\nCI job: https://github.com/TanStack/oracles/actions/runs/43458451421/job/131998714996\n\nFailure: `self-oracle.framework violated at checkpoint \"child\" (law: timeout). Difference: timeout presentation was not exercised`, after 26 shrinks.\n\n## Shrunk case\n\nOne draft. The fields that matter are:\n\n- `timing: \"slow\"`, so the spec is `slow` with a 100 ms test timeout.\n- `ownsMode: \"empty\"`, so the spec declares `owns: []`.\n- `timeoutPresentation: true` on the plan.\n\nThe rest is incidental, including `source: generated, runs: 2`, `seedless: true`, `nondeterministic: true`, and `clock: true`.\n\n## Classification: deterministic, not timing\n\n- Pinned seed on the unmodified `origin/main` (affeb1a): 6 of 6 runs fail, on Vitest 4.1.12 (1 run plus 5 more) and on Vitest 5.1.0 (5 runs). Each run reports the same violation.\n- Unpinned random unit on Vitest 5.1.0: 1 failure in 10 runs, with `ORACLE_SEED=1279286490`. It shows the same law and difference. I did not separately shrink that seed.\n- The machine was under heavy load during these runs (load average about 25 to 60 on 18 cores). The failure occurred anyway, and the child run finishes in about 1 second. The 100 ms timeout is never reached.\n\n## Root cause\n\n1. The generated child spec file fails to load. Its `owns` is empty, so the child reports `owns must be a non-empty literal list of strings`, and the child runs zero tests. I confirmed this by temporarily logging the child's report in a scratch copy. The child reported `tests: []`, and no child log was written.\n2. Because the spec never runs, the driver never appends to the child log. The driver's log line and its `await` for the slow timer are at `tests/oracle-property-self-oracle-fixture.ts:837` and `:840`. So `slowCalls` is empty (`tests/oracle-property-self-oracle.test.ts:5142`), and the `timeout-presentation` observation has `exercised: false` (`:5260-5261`).\n3. The timeout check in `checkSuite` (`tests/oracle-property-self-oracle.test.ts:6356-6366`) requires an exercised timeout presentation whenever the plan has a slow spec and `timeoutPresentation` is set. It does not consult the model's `loads` expectation. It also runs before the child checks that would have matched the load error, so it reports a false violation.\n4. The model already predicts this. `predictSpec` returns `loads: false` for `ownsMode` of `empty`, `string`, or `bad entry` (`tests/oracle-property-self-oracle.test.ts:2184-2191`), and the generator produces `empty` at `:845-856`.\n\nI checked the fix direction in a scratch copy, not in the worktree. Adding `&& expected.get(spec.id)?.loads` to the timeout precondition at `tests/oracle-property-self-oracle.test.ts:6356-6357` makes the pinned replay pass 3 of 3.\n\n## Fix direction\n\nChange the timeout law's precondition from \"a slow spec exists\" to \"a slow spec the model says loads\". That obligation is grounded in the model, not in wall-clock time, so a controlled clock is not needed. Keep the registration law as the first check for such a case, so a load failure is reported as a load failure.\n\nBefore merging, run the full `vitest.oracle-framework.config.mjs` suite with the change. Other load-failing causes (`unregistered`, invalid `work.ratio`, predict mismatches, and others in the `loads: false` returns at `:2178-2312`) reach the same branch by the same mechanism. I reproduced only the `ownsMode: empty` case, so the fix should be checked against the others too.\n\n## Limits\n\n- Only one shrunk case and one unpinned seed were examined. The fix was verified only on the pinned replay.\n- The failure rate in CI is unknown beyond the 1 in 10 sample above. It is latent on `origin/main` as well, so it is not specific to PR #7.\n- No framework code was changed or pushed.\n\nThis issue was written by Isaac.\n","author":{"url":"https://github.com/KyleAMathews","@type":"Person","name":"KyleAMathews"},"datePublished":"2026-11-13T16:30:55Z","url":"https://github.com/TanStack/oracles/issues/59"}