René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"JIT: Assertion failure `Stack underflow (depth = -1)`","articleBody":"# Crash report\n\n### What happened?\n\nIt's possible to cause an assertion failure in a patched JIT build by running the short code below.\n\n\n\u003cdetails\u003e\u003csummary\u003eNecessary patch\u003c/summary\u003e\n\u003cp\u003e\n\n\n```diff\ndiff --git a/Include/internal/pycore_backoff.h b/Include/internal/pycore_backoff.h\nindex 38dd82f6fc8..e1ef449e346 100644\n--- a/Include/internal/pycore_backoff.h\n+++ b/Include/internal/pycore_backoff.h\n@@ -125,7 +125,7 @@ trigger_backoff_counter(void)\n // For example, 4095 does not work for the nqueens benchmark on pyperformance\n // as we always end up tracing the loop iteration's\n // exhaustion iteration. Which aborts our current tracer.\n-#define JUMP_BACKWARD_INITIAL_VALUE 4000\n+#define JUMP_BACKWARD_INITIAL_VALUE 63\n #define JUMP_BACKWARD_INITIAL_BACKOFF 6\n static inline _Py_BackoffCounter\n initial_jump_backoff_counter(_PyOptimizationConfig *opt_config)\n@@ -153,7 +153,7 @@ initial_resume_backoff_counter(_PyOptimizationConfig *opt_config)\n  * Must be larger than ADAPTIVE_COOLDOWN_VALUE,\n  * otherwise when a side exit warms up we may construct\n  * a new trace before the Tier 1 code has properly re-specialized. */\n-#define SIDE_EXIT_INITIAL_VALUE 4000\n+#define SIDE_EXIT_INITIAL_VALUE 63\n #define SIDE_EXIT_INITIAL_BACKOFF 6\n\n static inline _Py_BackoffCounter\ndiff --git a/Include/internal/pycore_optimizer.h b/Include/internal/pycore_optimizer.h\nindex 7c2e0e95a80..04dda1eb3a0 100644\n--- a/Include/internal/pycore_optimizer.h\n+++ b/Include/internal/pycore_optimizer.h\n@@ -304,7 +304,7 @@ PyAPI_FUNC(void) _Py_Executors_InvalidateCold(PyInterpreterState *interp);\n // Used as the threshold to trigger executor invalidation when\n // executor_creation_counter is greater than this value.\n // This value is arbitrary and was not optimized.\n-#define JIT_CLEANUP_THRESHOLD 1000\n+#define JIT_CLEANUP_THRESHOLD 10000\n\n int _Py_uop_analyze_and_optimize(\n     _PyThreadStateImpl *tstate,\ndiff --git a/Include/internal/pycore_optimizer_types.h b/Include/internal/pycore_optimizer_types.h\nindex a722652cc81..37976ba9f48 100644\n--- a/Include/internal/pycore_optimizer_types.h\n+++ b/Include/internal/pycore_optimizer_types.h\n@@ -24,7 +24,7 @@ extern \"C\" {\n // progress (and inserting a new ENTER_EXECUTOR instruction). In practice, this\n // is the \"maximum amount of polymorphism\" that an isolated trace tree can\n // handle before rejoining the rest of the program.\n-#define MAX_CHAIN_DEPTH 4\n+#define MAX_CHAIN_DEPTH 16\n\n /* Symbols */\n /* See explanation in optimizer_symbols.c */\n```\n\n\u003c/p\u003e\n\u003c/details\u003e\n\n\nMRE:\n```python\ndef f1():\n    def victim(a=0, b=float(\"nan\"), c=2):\n        return (a + b) / c\n\n    for _ in range(90):\n        res = victim()\n\nfor i in range(30):\n    f1()\n```\n\n\nBacktrace:\n```\nStack underflow (depth = -1) at Python/executor_cases.c.h:8520\n\nProgram received signal SIGABRT, Aborted.\n\n#0  __pthread_kill_implementation (threadid=\u003coptimized out\u003e, signo=6, no_tid=0) at ./nptl/pthread_kill.c:44\n#1  __pthread_kill_internal (threadid=\u003coptimized out\u003e, signo=6) at ./nptl/pthread_kill.c:89\n#2  __GI___pthread_kill (threadid=\u003coptimized out\u003e, signo=signo@entry=6) at ./nptl/pthread_kill.c:100\n#3  0x00007ffff7c45e2e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26\n#4  0x00007ffff7c28888 in __GI_abort () at ./stdlib/abort.c:77\n#5  0x0000555555e8d4d4 in _Py_assert_within_stack_bounds (frame=frame@entry=0x7e8ff6fe5318, stack_pointer=stack_pointer@entry=0x7e8ff6fe5378, filename=\u003coptimized out\u003e,\n    lineno=lineno@entry=8520) at Python/ceval.c:1005\n#6  0x0000555555f77788 in _PyTier2Interpreter (current_executor=\u003coptimized out\u003e, frame=\u003coptimized out\u003e, stack_pointer=\u003coptimized out\u003e, tstate=\u003coptimized out\u003e)\n    at Python/executor_cases.c.h:8520\n#7  0x0000555555e917bb in _PyEval_EvalFrameDefault (tstate=\u003coptimized out\u003e, frame=\u003coptimized out\u003e, throwflag=\u003coptimized out\u003e) at Python/generated_cases.c.h:5941\n#8  0x0000555555e89af8 in _PyEval_EvalFrame (tstate=0x555556a754b8 \u003c_PyRuntime+360760\u003e, frame=0x7e8ff6fe5220, throwflag=0) at ./Include/internal/pycore_ceval.h:118\n#9  _PyEval_Vector (tstate=\u003coptimized out\u003e, func=\u003coptimized out\u003e, locals=\u003coptimized out\u003e, args=\u003coptimized out\u003e, argcount=\u003coptimized out\u003e, kwnames=0x0) at Python/ceval.c:2124\n#10 0x0000555555e89515 in PyEval_EvalCode (co=\u003coptimized out\u003e, globals=\u003coptimized out\u003e, locals=0x7c7ff7089fc0) at Python/ceval.c:686\n#11 0x000055555617dad0 in run_eval_code_obj (tstate=tstate@entry=0x555556a754b8 \u003c_PyRuntime+360760\u003e, co=co@entry=0x7d1ff701ab50, globals=globals@entry=0x7c7ff7089fc0,\n    locals=locals@entry=0x7c7ff7089fc0) at Python/pythonrun.c:1369\n#12 0x000055555617cc9c in run_mod (mod=\u003coptimized out\u003e, filename=\u003coptimized out\u003e, globals=\u003coptimized out\u003e, locals=\u003coptimized out\u003e, flags=\u003coptimized out\u003e, arena=\u003coptimized out\u003e,\n    interactive_src=\u003coptimized out\u003e, generate_new_source=\u003coptimized out\u003e) at Python/pythonrun.c:1472\n```\n\nOutput from running with `PYTHON_LLTRACE=4` and `PYTHON_OPT_DEBUG=4`:\n\n\n\n\u003cdetails\u003e\u003csummary\u003eClaude's exploration of related reproducers\u003c/summary\u003e\n\u003cp\u003e\n\nHere are the additional reproducers — every one is independently confirmed to abort on unmodified `main` with `Stack underflow (depth = -1) at Python/executor_cases.c.h:8520` and to pass with the fix applied:\n\n### v2 — TOS recorded float, NOS int dividend (`int / float`)\n```python\ndef f1():\n    def victim(a=2, b=0, c=float(\"nan\")):\n        return a / (b + c)\n    for _ in range(90):\n        victim()\n\nfor i in range(30):\n    f1()\n```\nTests the `_GUARD_TOS_FLOAT` side of the bug (mirror of the original).\n\n### v3 — `%` instead of `/` (NB_REMAINDER)\n```python\ndef f1():\n    def victim(a=0, b=float(\"nan\"), c=2):\n        return (a + b) % c\n    for _ in range(90):\n        victim()\n\nfor i in range(30):\n    f1()\n```\nConfirms the bug fires on the remainder branch too — same `is_truediv || is_remainder` guard logic.\n\n### v5 — minimal: no intermediate addition\n```python\ndef f1():\n    def victim(b=float(\"nan\"), c=2):\n        return b / c\n    for _ in range(90):\n        victim()\n\nfor i in range(30):\n    f1()\n```\nDrops the `(a + b)` indirection — the bug doesn't need a chained binop, just a recorded-float operand.\n\n### v6 — opposite direction: `int / float`\n```python\ndef f1():\n    def victim(a=2, b=float(\"nan\")):\n        return a / b\n    for _ in range(90):\n        victim()\n\nfor i in range(30):\n    f1()\n```\nTOS-side guard variant of v5.\n\n### v7 — no defaults, explicit args\n```python\ndef f1():\n    def victim(b, c):\n        return b / c\n\n    nan = float(\"nan\")\n    for _ in range(90):\n        victim(nan, 2)\n\nfor i in range(30):\n    f1()\n```\nShows defaults aren't part of the trigger — any consistent (float, int) call pair into a hot truediv works.\n\n### Negative cases (don't crash on baseline) worth mentioning\n- Same shape with `float(\"inf\")` instead of `float(\"nan\")` — passes. Likely because `inf` becomes a safe const somewhere upstream and gets folded out before the buggy path.\n- Same shape with a regular float literal like `2.5` — passes. `_RECORD_TOS_TYPE` doesn't fire on operands that the optimizer already knows are constants/safe-floats, so the speculative `_GUARD_*_FLOAT` branch isn't reached.\n\nBoth negatives are useful as a clue that the bug requires the operand to be a `JIT_SYM_RECORDED_TYPE_TAG` (float) rather than a safe-const float — i.e., it has to come in via `_RECORD_*_TYPE` from a `_LOAD_FAST_BORROW`'d local.\n\n\u003c/p\u003e\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\u003csummary\u003eClaude analysis and suggested fix\u003c/summary\u003e\n\u003cp\u003e\n\n## Diagnosis \u0026 fix\n\n**Bug.** The Tier 2 abstract interpreter for `_BINARY_OP` in `Python/optimizer_bytecodes.c` (introduced by commit `95cbd4a232d`, gh-146393, GH-146397 — speculative float-divide narrowing) can drop the actual binary op from the trace.\n\nThe handler optionally emits `ADD_OP(_GUARD_TOS_FLOAT)` / `ADD_OP(_GUARD_NOS_FLOAT)` when an operand has a recorded probable type of float, then falls through to type-analysis branches. Only the `is_truediv \u0026\u0026 lhs_float \u0026\u0026 rhs_float` branch emits an `ADD_OP` for a specialized binary op. The other branches just compute a result-type symbol and return.\n\nThe `optimizer_analysis.c` driver auto-copies `*this_instr` to the output buffer only when `out_buffer.next == out_ptr` — i.e., when **no** `ADD_OP` was called that iteration. Adding only a guard therefore suppresses the auto-copy and the original `_BINARY_OP` disappears.\n\n**Concrete failing trace** (from your MRE — `(a + 0.0_nan) / 2`, hot in `victim`):\n\nThe optimizer's abstract stack still tracks `_BINARY_OP`'s `(lhs, rhs -- res, l, r)` (+1 net), but the emitted uops are only `_GUARD_NOS_FLOAT` (0) + `_POP_TOP_NOP` (-1) + `_POP_TOP_FLOAT` (-1) = -2. After the `(a+b)/c` macro the abstract stack thinks there is one return value at `[ -1 ]`; the physical stack is actually empty. `_MAKE_HEAP_SAFE` (the first half of `RETURN_VALUE`) then reads `stack_pointer[-1]` and decrements `stack_pointer` past `Stackbase`, tripping `_Py_assert_within_stack_bounds` at `Python/executor_cases.c.h:8520`.\n\nRepros only when `lhs` (or `rhs`) is a `JIT_SYM_RECORDED_TYPE_TAG` of float (`sym_has_type` false, `sym_get_probable_type` == float) and the other operand is anything `sym_matches_type` doesn't see as float — exactly what you get with `(int + float) / int` after `_RECORD_*_TYPE` runs in the second `BINARY_OP`.\n\n**Fix.** In `Python/optimizer_bytecodes.c`, restructure the `_BINARY_OP` handler so every non-specialized path also explicitly `ADD_OP(_BINARY_OP, oparg, 0)`. The only branch that omits it is the float/float truediv path that already emits its own `_BINARY_OP_TRUEDIV_FLOAT*` replacement. (Identical edit applied to the generated `Python/optimizer_cases.c.h`; `make regen-optimizer-cases` produces the same content modulo whitespace.)\n\n**Verification.**\n- `Python/optimizer_bytecodes.c:319+` (and the generated `Python/optimizer_cases.c.h` `case _BINARY_OP:`).\n- After fix, the optimizer trace at the offending site is `_GUARD_NOS_FLOAT` + `_BINARY_OP` + `_POP_TOP_NOP` + `_POP_TOP_FLOAT`, restoring the +1/-1/-1 net effect that matches the abstract stack.\n- MRE no longer aborts; four hand-built variants (truediv with NOS recorded float, truediv with TOS recorded float, remainder mirror, infinity instead of NaN) also pass on the fix and crash on baseline (truediv ones, at least).\n- `test_optimizer` and `test_capi.test_opt` show no new regressions: the only `test_capi.test_opt` failures (`test_resume`, `test_call_super`) reproduce on a clean rebuild of unmodified `main` — pre-existing, unrelated to this fix.\n\nThe minimal source diff is in `Python/optimizer_bytecodes.c` lines 319–385 (re-nested as `if {...} else { ADD_OP(_BINARY_OP, oparg, 0); ... }`); the generated `optimizer_cases.c.h` is updated to match.\n\n\u003c/p\u003e\n\u003c/details\u003e\n\n\nFound using [lafleur](https://github.com/devdanzin/lafleur/).\n\n### CPython versions tested on:\n\n3.15, CPython main branch\n\n### Operating systems tested on:\n\nLinux\n\n### Output from running 'python -VV' on the command line:\n\nPython 3.15.0a8+ (heads/main-dirty:804c213c893, Apr 27 2026, 07:24:58) [Clang 21.1.2 (2ubuntu6)]\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-149076\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/devdanzin","@type":"Person","name":"devdanzin"},"datePublished":"2026-04-27T10:36:17.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/149049/cpython/issues/149049"}