René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"CI: graph-lifetime tests flaky on free-threaded 3.14t jobs since #2371","articleBody":"## Describe the bug\n\nSince #2371 merged (2026-07-15 16:55 UTC), CI is flaky on **free-threaded Python 3.14t** test jobs: four tests in [`cuda_core/tests/graph/test_graph_definition_lifetime.py`](https://github.com/NVIDIA/cuda-python/blob/58f87330386f672cf29c716ca84ba8e832967300/cuda_core/tests/graph/test_graph_definition_lifetime.py) intermittently fail with\n\n```\nAssertionError: condition not satisfied within 5.0s\n```\n\nraised by the [`_wait_until`](https://github.com/NVIDIA/cuda-python/blob/58f87330386f672cf29c716ca84ba8e832967300/cuda_core/tests/graph/test_graph_definition_lifetime.py#L52-L75) polling helper ([`_FINALIZE_TIMEOUT = 5.0`](https://github.com/NVIDIA/cuda-python/blob/58f87330386f672cf29c716ca84ba8e832967300/cuda_core/tests/graph/test_graph_definition_lifetime.py#L29)):\n\n- `test_kernel_args_buffer_lifetime` (added in #2041)\n- `test_memcpy_buffer_allocations_released_after_graph_destroyed` (#2280)\n- `test_slot_owners_released_after_graph_destroyed` (#2280)\n- `test_pending_call_queue_saturation_preserves_cleanup` (#2371)\n\n## Impact (2026-07-15 → 2026-07-21)\n\nSurvey of all Actions runs 2026-06-15 → 2026-07-22 (246 failed/re-run runs classified by job-log signature):\n\n- **16 of 77 CI runs (~21%) hit this signature**, including 5 runs on `main` (push + scheduled). 11 were on PRs touching nothing near graph code (e.g. #2374, #2402).\n- 7 of the 16 were re-run to green — one needed [6 attempts](https://github.com/NVIDIA/cuda-python/actions/runs/29602432384) — so red-run counts understate it.\n- **Zero occurrences** in ~300 completed runs 2026-06-15 → 2026-07-14. #2280 (merged 2026-07-07) ran clean for 8 days; the first hit is 4.5 h after #2371 merged (2026-07-15 21:26 UTC, [run 29451966527](https://github.com/NVIDIA/cuda-python/actions/runs/29451966527)).\n- Only 3.14t jobs are affected — across win-64 / linux-64 / linux-aarch64, CUDA 12.9.1 / 13.0.2 / 13.3.0, local & wheels, T4 / L4 / A100. Never observed on regular-GIL jobs.\n\n## Failing jobs (all 18 instances)\n\n| Failing CI pipeline | CI log |\n|---|---|\n| Test win-64 / Python 3.14t, CUDA 13.0.2 (wheels), GPU a100 (MCDM) | [2026-07-15 · PR 2374](https://github.com/NVIDIA/cuda-python/actions/runs/29451966527/job/87480738831) |\n| Test win-64 / Python 3.14t, CUDA 13.0.2 (wheels), GPU a100 (MCDM) | [2026-07-16 · PR 2242](https://github.com/NVIDIA/cuda-python/actions/runs/29499296934/job/87629352733) |\n| Test linux-aarch64 / Python 3.14t, CUDA 13.0.2 (wheels), GPU a100 | [2026-07-17 · PR 2379](https://github.com/NVIDIA/cuda-python/actions/runs/29584165382/job/87899244042) |\n| Test win-64 / Python 3.14t, CUDA 13.0.2 (wheels), GPU a100 (MCDM) | [2026-07-17 · main](https://github.com/NVIDIA/cuda-python/actions/runs/29555550182/job/87809528510) |\n| Test win-64 / Python 3.14t, CUDA 13.0.2 (wheels), GPU a100 (MCDM) | [2026-07-17 · main](https://github.com/NVIDIA/cuda-python/actions/runs/29602432384/job/87962162694) |\n| Test win-64 / Python 3.14t, CUDA 13.0.2 (wheels), GPU a100 (MCDM) | [2026-07-19 · main](https://github.com/NVIDIA/cuda-python/actions/runs/29666641194/job/88139923507) |\n| Test linux-64 / Python 3.14t, CUDA 12.9.1 (local), GPU t4 | [2026-07-20 · PR 2390](https://github.com/NVIDIA/cuda-python/actions/runs/29724266375/job/88295548106) |\n| Test linux-64 / Python 3.14t, CUDA 13.0.2 (local), GPU l4 | [2026-07-20 · PR 2242](https://github.com/NVIDIA/cuda-python/actions/runs/29768699624/job/88444074974) |\n| Test linux-aarch64 / Python 3.14t, CUDA 13.0.2 (wheels), GPU a100 | [2026-07-20 · main](https://github.com/NVIDIA/cuda-python/actions/runs/29719293380/job/88279928623) |\n| Test win-64 / Python 3.14t, CUDA 13.0.2 (wheels), GPU a100 (MCDM) | [2026-07-20 · PR 2242](https://github.com/NVIDIA/cuda-python/actions/runs/29768699624/job/88446543396) |\n| Test win-64 / Python 3.14t, CUDA 13.0.2 (wheels), GPU a100 (MCDM) | [2026-07-20 · PR 2394](https://github.com/NVIDIA/cuda-python/actions/runs/29777373005/job/88475693911) |\n| Test linux-64 / Python 3.14t, CUDA 12.9.1 (local), GPU t4 | [2026-07-21 · PR 2398](https://github.com/NVIDIA/cuda-python/actions/runs/29849525172/job/88701835631) |\n| Test linux-64 / Python 3.14t, CUDA 12.9.1 (local), GPU t4 | [2026-07-21 · PR 2402](https://github.com/NVIDIA/cuda-python/actions/runs/29876578909/job/88790610219) |\n| Test linux-aarch64 / Python 3.14t, CUDA 13.3.0 (local), GPU l4 | [2026-07-21 · main](https://github.com/NVIDIA/cuda-python/actions/runs/29846402084/job/88690759075) |\n| Test linux-aarch64 / Python 3.14t, CUDA 13.3.0 (local), GPU l4 | [2026-07-21 · PR 2383](https://github.com/NVIDIA/cuda-python/actions/runs/29855714919/job/88722171657) |\n| Test win-64 / Python 3.14t, CUDA 12.9.1 (local), GPU l4 (TCC) | [2026-07-21 · PR 2390](https://github.com/NVIDIA/cuda-python/actions/runs/29796301934/job/88530809561) |\n| Test win-64 / Python 3.14t, CUDA 13.0.2 (wheels), GPU a100 (MCDM) | [2026-07-21 · PR 2392](https://github.com/NVIDIA/cuda-python/actions/runs/29825220814/job/88621734327) |\n| Test win-64 / Python 3.14t, CUDA 13.0.2 (wheels), GPU a100 (MCDM) | [2026-07-21 · PR 2398](https://github.com/NVIDIA/cuda-python/actions/runs/29849525172/job/88704708482) |\n\n(Excluded from the count: graph PRs failing their own branches — #2280, #2333, #2357 — and the 2026-07-17 gb300 nightly, which failed machine-wide with `ValueError: Cannot set NULL context as current` and looks unrelated.)\n\n## How to Reproduce\n\nNo deterministic local repro; it fires in roughly 1 in 5 CI runs on any 3.14t GPU test job since 2026-07-15. The four tests above are the ones to stress.\n\n## Expected behavior\n\nGraph-lifetime cleanup assertions pass reliably on free-threaded builds.\n\n## Likely mechanism\n\n#2371 defers user-object payload cleanup through a pending-call queue. On free-threaded Python (deferred reference counting), on a loaded runner, cleanup can lag past the 5 s `_FINALIZE_TIMEOUT` budget, so assertions observing release (weakref death, allocation counters, queue drain) time out. Three of the four tests predate #2371 and only began flaking once it landed.\n\nPossible directions:\n\n- Raise `_FINALIZE_TIMEOUT` on free-threaded builds (it is already 30 s under compute-sanitizer).\n- Make `_wait_until` actively drive the pending-call queue instead of only `gc.collect()` + sleep.\n- Bound cleanup latency in the implementation (e.g. drain the queue opportunistically on API calls).\n\n-- Leo's bot\n","author":{"url":"https://github.com/leofang","@type":"Person","name":"leofang"},"datePublished":"2026-07-22T01:27:34Z","url":"https://github.com/NVIDIA/cuda-python/issues/2403"}