René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Free-threaded CPython 3.14: concurrent functools.partial.__setstate__ crashes with repr(), calls, or another __setstate__","articleBody":"### Prior report\n\ngh-157099 (2026-09-07) reported the same missing synchronization in `partial_setstate`, reaching it through `partial_new` (a crash in `tuple_concat` under ASan, 3.14.7 @ 823f032). It was closed by its reporter after a request to consolidate a batch of reports (gh-157124), and the consolidated issues gh-157196 / gh-157197 / gh-157198 were then withdrawn by the reporter on 2026-09-11 to be resubmitted under the security policy. As far as I can see nothing tracks it now, and `main` @ 89c67a9 still crashes. This report was found independently (static analysis of `partial_repr` on 3.14.0rc1, then runtime confirmation) and adds a sanitizer-free 25-line reproducer covering `repr()`, calls and writer/writer, with a GIL control on the same binary. If you would rather reopen gh-157099, please close this one as a duplicate. Filed on the public tracker because triggering it requires running Python code in the process (the assessment given on gh-157196).\n\n### What happened?\n\nOn the free-threaded build, calling `partial.__setstate__` on a `functools.partial` object while another thread uses the same object (`repr(p)` or `p()`), or while a second thread also calls `__setstate__`, crashes the interpreter within seconds. The read-only control completed all five four-second runs without a crash. The same binary with `PYTHON_GIL=1` completed every run without a crash.\n\nStandalone reproducer (`repro_partial.py \u003cmode>`):\n\n```python\nimport functools, itertools, os, sys, threading\nif os.environ.get(\"ALLOW_GIL\") != \"1\":\n    assert not sys._is_gil_enabled()\nmode = sys.argv[1]\ncounter = itertools.count(10**12)          # fresh heap ints, never cached\ndef fresh(): return next(counter)\ndef f(*a, **k): return None\np = functools.partial(f, fresh(), fresh())\nstop = threading.Event()\ndef reader_repr(): repr(p)\ndef reader_call(): p()\ndef writer(): p.__setstate__((f, (fresh(), fresh()), {}, None))\nreader, nr, nw = {\"ro\": (reader_repr, 4, 0), \"rw\": (reader_repr, 4, 1),\n                  \"ww\": (None, 0, 2), \"cw\": (reader_call, 4, 1)}[mode]\ndef loop(fn):\n    while not stop.is_set():\n        fn()\nts = [threading.Thread(target=loop, args=(reader,)) for _ in range(nr)] + \\\n     [threading.Thread(target=loop, args=(writer,)) for _ in range(nw)]\nfor t in ts: t.start()\nstop.wait(4); stop.set()\nfor t in ts: t.join()\nprint(\"ok\")\n```\n\nResults, 5 runs × 4 s per mode, run with `-X faulthandler` (\"hang\" = the process had not exited 30 s after start and was killed):\n\n| mode | threads | 3.14.6t | 3.14.6t, `PYTHONMALLOC=debug` | same binary, `PYTHON_GIL=1` | `main` @ 89c67a9 (`--disable-gil`) | `main`, `PYTHONMALLOC=debug` |\n|---|---|---|---|---|---|---|\n| `ro` | 4 × `repr(p)` | 5/5 ok | 5/5 ok | — | 5/5 ok | 5/5 ok |\n| `rw` | 4 × `repr(p)` + 1 × `__setstate__` | **0/5** (SIGILL, SIGABRT, 3 hangs) | **0/5** (SIGSEGV ×5) | 3/3 ok | **0/5** (SIGABRT ×2, SIGILL, 2 hangs) | **0/5** (SIGSEGV ×5) |\n| `ww` | 2 × `__setstate__` | 4/5 (SIGSEGV ×1) | 1/5 (SIGSEGV ×4) | 3/3 ok | 2/5 (SIGSEGV ×3) | 4/5 (SIGSEGV ×1) |\n| `cw` | 4 × `p()` + 1 × `__setstate__` | **0/5** (SIGILL ×2, SIGTRAP ×2, SIGSEGV) | **0/5** (SIGSEGV ×5) | 3/3 ok | **0/5** (SIGSEGV ×3, SIGILL, 1 hang) | **0/5** (SIGSEGV ×5) |\n\n`ww` is intermittent: over several runs of this script today it crashed in 1 to 4 of 5 runs per configuration. `rw` and `cw` failed in every run of every free-threaded configuration. The SIGABRT cases are\n\n```\nFatal Python error: PyMutex_Unlock: unlocking mutex that is not locked\n```\n\nraised from the `repr` thread (faulthandler shows `reader_repr`). The `repr`/`__setstate__` shape also crashes on 3.14.0rc1 free-threaded (5/5, `PYTHONMALLOC=debug`). Note that `main` already carries the `partial_vectorcall` snapshot from gh-154189, and `cw` still crashes there.\n\n### Analysis\n\n`partial_setstate` replaces `pto->fn`, `pto->args` and `pto->kw` with `Py_SETREF` and holds no critical section on the partial object (3.13 branch `Modules/_functoolsmodule.c:514`; 3.14 branch `:762`; `main` `:857`). The readers take strong references from the fields — `partial_repr` since gh-144475 (3.14 branch `:630`), `partial_vectorcall` since gh-154189 (`main` only; not in 3.14). The source suggests an unsynchronized reference-acquisition gap: `Py_NewRef(pto->args)` does not protect the interval between loading the field and acquiring a strong reference while another thread replaces it. The observed crashes have not yet been attributed to a particular native access. Two concurrent `__setstate__` writers also crash without a reader thread; the failing access in that case has not yet been identified. gh-144475 and gh-154189 addressed the re-entrant, same-thread shape of this; gh-157099 and this report are the cross-thread shape.\n\n### Expected behaviour\n\nI expect concurrent use to avoid interpreter crashes; no particular ordering or consistent result is required by this reproducer. The free-threading HOWTO says the build \"aims to provide similar thread-safety behavior at the Python level to the default GIL-enabled build\" while noting that this \"should be treated as a description of the current implementation, not a guarantee\"; the same binary with the GIL enabled runs the reproducer without a crash.\n\n### Possible approach\n\nA possible approach is to synchronize state publication and acquisition of coherent reader snapshots on the partial object, including associated placeholder and vectorcall state; the complete access and reentrancy audit remains to be done. Happy to test a patch.\n\n### CPython versions tested on\n\n3.14.6 (free-threaded, python-build-standalone via uv), 3.14.0rc1 (free-threaded), CPython main branch @ 89c67a9 (2026-09-20, local `--disable-gil` build; `Modules/_functoolsmodule.c` identical to upstream at that commit)\n\n### Operating systems tested on\n\nmacOS (arm64, Darwin 25.6)\n\n### Output from running 'python -VV' on the command line\n\n```\nPython 3.14.6 free-threading build (main, Jul 18 2026, 17:07:52) [Clang 22.1.3 ]\nPython 3.16.0a0 free-threading build (remotes/origin/HEAD-dirty:89c67a9, Sep 20 2026, 14:12:46) [Clang 21.0.0 (clang-2100.3.34.2)]\n```\n\n### Related\n\ngh-157099 (same root cause, closed not_planned), gh-144475, gh-154189, gh-140590\n\n\n\u003c!-- gh-linked-prs -->\n### Linked PRs\n* gh-157979\n* gh-160480\n\u003c!-- /gh-linked-prs -->\n","author":{"url":"https://github.com/espressolee","@type":"Person","name":"espressolee"},"datePublished":"2026-09-20T07:06:45Z","url":"https://github.com/python/cpython/issues/157841"}