Title: Free-threaded build: crash at exit when a starting thread binds a thread state that finalization has cleared · Issue #162109 · python/cpython · GitHub
Open Graph Title: Free-threaded build: crash at exit when a starting thread binds a thread state that finalization has cleared · Issue #162109 · python/cpython
X Title: Free-threaded build: crash at exit when a starting thread binds a thread state that finalization has cleared · Issue #162109 · python/cpython
Description: Crash report I was exploring peak RSS gaps of the free-threaded build with mimalloc v3 (#159718) with overnight stress tests. With daemon threads, the process sometimes crashed at exit. The cause looks like a gap left after the fix for #...
Open Graph Description: Crash report I was exploring peak RSS gaps of the free-threaded build with mimalloc v3 (#159718) with overnight stress tests. With daemon threads, the process sometimes crashed at exit. The cause l...
X Description: Crash report I was exploring peak RSS gaps of the free-threaded build with mimalloc v3 (#159718) with overnight stress tests. With daemon threads, the process sometimes crashed at exit. The cause l...
Opengraph URL: https://github.com/python/cpython/issues/162109
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Free-threaded build: crash at exit when a starting thread binds a thread state that finalization has cleared","articleBody":"### Crash report\n\nI was exploring peak RSS gaps of the free-threaded build with mimalloc v3 (https://github.com/python/cpython/pull/159718) with overnight stress tests. With daemon threads, the process sometimes crashed at exit. The cause looks like a gap left after the fix for https://github.com/python/cpython/issues/124878 (https://github.com/python/cpython/pull/130649): a thread that is just starting can bind its thread state after finalization has already cleared it. An AI assistant helped with the analysis.\n\n### What happens\n\nThis script starts daemon threads that keep starting more daemon threads, and exits after 20 ms:\n\n```python\nimport threading, time\ndef work():\n while True:\n x = [bytes(100) for _ in range(300)]\n t = threading.Thread(target=lambda: [bytes(64) for _ in range(100)], daemon=True)\n t.start()\nfor _ in range(8):\n threading.Thread(target=work, daemon=True).start()\ntime.sleep(0.02)\n```\n\nI ran it in a loop of fresh processes on main at e8c0584877 (2026-11-10), release builds, Linux arm64. A probe (a 3 ms pause in `thread_run()` at every 16th thread start, in the fold) makes the crash frequent:\n\n| | exits | crashed (SIGSEGV) |\n|---|---:|---:|\n| free-threaded build | 10,752 | 1 |\n| free-threaded build with the probe | 12,845 | 198 |\n| default (GIL) build | 16,068 | 0 |\n\nOn macOS I saw no crash (0 of 13,034 exits). 3.14 and 3.15 have the same code; I tested main only.\n\nFinalization crashes at a decref of an object that belongs to another thread (backtrace trimmed):\n\n```\n#0 find_thread_state (thread_id=281472972548224) at Python/brc.c:40\n#1 _Py_brc_queue_object (ob=0x35708810160) at Python/brc.c:90\n#3 dictkeys_decref (dk=0x3570c06c000, use_qsbr=true) at Objects/dictobject.c:517\n#11 _PyGC_CollectNoFail (tstate=0xaaaae1580708 \u003c_PyRuntime+295328>) at Python/gc_free_threading.c:2634\n#13 _Py_Finalize (runtime=0xaaaae1538568 \u003c_PyRuntime>) at Python/pylifecycle.c:2501\n```\n\n`find_thread_state()` looks up the owning thread in the table of threads that biased reference counting keeps. In the core file, it reaches a thread state whose memory is all zero: freed, but still in the table.\n\n### Why I think it happens\n\nIn [`thread_run()`](https://github.com/python/cpython/blob/e8c0584877e0/Modules/_threadmodule.c#L369-L386), a new thread checks whether it must exit (`_PyThreadState_MustExit()`) and then binds its thread state (`_PyThreadState_Bind()`, which adds it to the table). If finalization starts between the two:\n\n1. Finalization marks the thread state as shutting down and clears it. The state isn't in the table yet, so [`_Py_brc_remove_thread()`](https://github.com/python/cpython/blob/e8c0584877e0/Python/brc.c#L195-L204) has nothing to remove. The interpreter drops its reference.\n2. The thread binds the cleared state, which adds it to the table. It fails to attach, hangs in `_PyThreadState_HangThread()` and drops the last reference. The state is freed but stays in the table.\n3. Finalization's next decref of an object owned by another thread searches that part of the table and crashes.\n\nOnly daemon threads get this far; finalization waits for the others first. The probe only makes the gap between the check and the bind longer.\n\n`PyThreadState_New()` has the same gap: [`_PyThreadState_NewBound()`](https://github.com/python/cpython/blob/e8c0584877e0/Python/pystate.c#L1801-L1814) creates the thread state and binds it right after. With a small C extension whose own threads keep creating and deleting thread states while the process exits (in the fold), and the same kind of pause between creating and binding, a debug build aborted on the assertion in `bind_tstate()` in 3,450 of 3,450 exits. `PyGILState_Ensure()` seems safe on main: it holds an interpreter guard from PEP 788 while it creates and binds, and the extension changed to use it didn't crash (0 of 12,868 exits, release build).\n\nAs far as I can tell, both cases come from one assumption: finalization ([`_PyThreadState_DeleteList()`](https://github.com/python/cpython/blob/e8c0584877e0/Python/pystate.c#L2183-L2202)) clears the thread states of the other threads as if a thread that isn't attached didn't use its thread state. A thread that is binding one does.\n\n### Related reports\n\n- https://github.com/python/cpython/issues/124878: since its fix, the new thread holds its own reference, so the state isn't freed too early. The bind after the clear remains.\n- https://github.com/python/cpython/issues/104690 (2023): the same assertion in `bind_tstate()`. Here a debug build with the probe trips it in 4,773 of 5,058 exits, without the probe in 0 of 2,485.\n- https://github.com/python/cpython/issues/108987: where the \"must exit\" check in `thread_run()` comes from.\n- https://github.com/python/cpython/issues/124619: threads not started by Python that create a thread state only after finalization has taken the others out. A different window, which the C extension also hits.\n\n### A sketch that worked here\n\nI don't know this code well enough to say whether this is the right place for a fix, but this sketch removed the crash in my tests. A small field in each thread state decides who goes first. The thread claims it before binding and marks it as bound afterwards (in `_PyThreadState_Bind()` and `_PyThreadState_NewBound()`); finalization claims it before clearing (in `_PyThreadState_DeleteList()`). If finalization is first, the thread leaves the state unbound and hangs when it tries to attach, so nothing of it is in the table. If the thread is in the middle of its bind, finalization waits for it and then clears the state. After a fork there is no waiting: the binding thread doesn't exist in the child.\n\n| | exits without the sketch | crashed or aborted | exits with the sketch | crashed or aborted |\n|---|---:|---:|---:|---:|\n| the script, release build | 10,752 | 1 | 10,550 | 0 |\n| the script with the probe, release build | 12,845 | 198 | 9,954 | 0 |\n| the script with the probe, debug build | 5,058 | 4,773 | 3,596 | 0 |\n| the C extension with the pause, debug build | 3,450 | 3,450 | 3,017 | 0 |\n\nThe C extension also hits https://github.com/python/cpython/issues/124619. In the last row a second sketch for that issue was applied too; I'll describe it there. With this sketch alone, the full test suite passes on Linux (free-threaded release and debug builds, default build).\n\nIts limits:\n\n- Without the sketch the script also hangs at exit now and then (16 of the 10,752 exits). That looks like a separate problem (`PyMutex` hands a mutex directly to a thread that has waited for it more than 1 ms, and that thread may then hang), which I'm reporting separately. None of the runs with the sketch hung, but it isn't meant for that.\n- Finalization now waits for a thread that is in the middle of its bind. If that thread is stuck on a mutex of the table that was handed off to a hung thread, finalization waits forever; without the sketch it would get stuck on the same mutex a moment later, when it clears that thread state.\n\nAttempts that didn't work out:\n\n- Checking and binding while holding `HEAD_LOCK`: deadlocked once in 14,335 exits. The thread held `HEAD_LOCK`, which a stop-the-world needs, while it waited for a mutex of the table that had been handed to a thread the stop-the-world had already paused.\n- Removing the state from the table in `_PyThreadState_HangThread()`: tested clean, but every hanging thread then reads its interpreter, which may already be freed for a subinterpreter.\n- A second \"must exit\" check after the bind: no crash with the probe (0 of 21,987 exits), but the cleared state stays bound (a debug build with the probe still aborts in 2,750 of 2,756 exits), and `PyThreadState_New()` isn't covered.\n\nI may well be missing context here, and I'm glad to test other approaches.\n\n\u003cdetails>\n\u003csummary>The sketch, the probe and the C extension (against e8c0584877, 2026-11-10)\u003c/summary>\n\n```diff\n--- a/Include/internal/pycore_tstate.h\n+++ b/Include/internal/pycore_tstate.h\n@@ -39,6 +39,10 @@\n // thread state during runtime finalization.\n Py_ssize_t refcount;\n \n+ // Who goes first: the thread binding this thread state, or finalization\n+ // clearing it (_PyThreadState_DeleteList()). One of _Py_TSTATE_BIND_*.\n+ int bind_claim;\n+\n // These are addresses, but we need to convert to ints to avoid UB.\n uintptr_t c_stack_top;\n uintptr_t c_stack_soft_limit;\n--- a/Python/pystate.c\n+++ b/Python/pystate.c\n@@ -160,6 +160,46 @@\n \n static void tstate_mimalloc_bind(PyThreadState *);\n \n+#define _Py_TSTATE_BIND_NONE 0\n+#define _Py_TSTATE_BIND_BINDING 1 // the thread is binding it\n+#define _Py_TSTATE_BIND_DONE 2 // the thread has bound it\n+#define _Py_TSTATE_BIND_CLAIMED 3 // finalization took it first\n+\n+// Called by the thread before it binds tstate. Returns 0 if finalization has\n+// already claimed the thread state: then the thread must not bind it.\n+static int\n+bind_claim_begin(PyThreadState *tstate)\n+{\n+ int expected = _Py_TSTATE_BIND_NONE;\n+ return _Py_atomic_compare_exchange_int(\n+ &((_PyThreadStateImpl *)tstate)->bind_claim, &expected,\n+ _Py_TSTATE_BIND_BINDING);\n+}\n+\n+static void\n+bind_claim_end(PyThreadState *tstate)\n+{\n+ _Py_atomic_store_int(&((_PyThreadStateImpl *)tstate)->bind_claim,\n+ _Py_TSTATE_BIND_DONE);\n+}\n+\n+// Called by finalization before it clears the thread state of another thread:\n+// either the thread hasn't started to bind it (and now won't), or wait until\n+// its bind has finished.\n+static void\n+bind_claim_for_clear(PyThreadState *tstate)\n+{\n+ int *claim = &((_PyThreadStateImpl *)tstate)->bind_claim;\n+ int expected = _Py_TSTATE_BIND_NONE;\n+ if (_Py_atomic_compare_exchange_int(claim, &expected,\n+ _Py_TSTATE_BIND_CLAIMED)) {\n+ return;\n+ }\n+ while (_Py_atomic_load_int(claim) == _Py_TSTATE_BIND_BINDING) {\n+ _Py_yield();\n+ }\n+}\n+\n static void\n bind_tstate(PyThreadState *tstate)\n {\n@@ -1802,14 +1842,17 @@\n _PyThreadState_NewBound(PyInterpreterState *interp, int whence)\n {\n PyThreadState *tstate = new_threadstate(interp, whence);\n- if (tstate) {\n+ if (tstate && bind_claim_begin(tstate)) {\n bind_tstate(tstate);\n // This makes sure there's a gilstate tstate bound\n // as soon as possible.\n if (gilstate_get() == NULL) {\n bind_gilstate_tstate(tstate);\n }\n+ bind_claim_end(tstate);\n }\n+ // Otherwise finalization has taken (and cleared) the thread state before\n+ // the bind: leave it unbound; the thread hangs when it tries to attach.\n return tstate;\n }\n \n@@ -2190,6 +2233,9 @@\n PyThreadState *p, *next;\n for (p = list; p; p = next) {\n next = p->next;\n+ if (!is_after_fork) {\n+ bind_claim_for_clear(p);\n+ }\n PyThreadState_Clear(p);\n tstate_delete_common(p, /*release_gil=*/0, /*is_unlinked=*/1);\n if (is_after_fork) {\n@@ -2817,12 +2863,18 @@\n // was called, tstate becomes a dangling pointer.\n assert(_PyThreadState_CheckConsistency(tstate));\n \n+ if (!bind_claim_begin(tstate)) {\n+ // Finalization has taken the thread state before the bind: leave it\n+ // unbound; the thread hangs when it tries to attach.\n+ return;\n+ }\n bind_tstate(tstate);\n // This makes sure there's a gilstate tstate bound\n // as soon as possible.\n if (gilstate_get() == NULL) {\n bind_gilstate_tstate(tstate);\n }\n+ bind_claim_end(tstate);\n }\n \n #if defined(Py_GIL_DISABLED) && !defined(Py_LIMITED_API)\n```\n\nThe probe in `thread_run()` (not part of the sketch). The pause for the C extension is the same block in `_PyThreadState_NewBound()`, between `new_threadstate()` and `bind_tstate()`.\n\n```diff\n--- a/Modules/_threadmodule.c\n+++ b/Modules/_threadmodule.c\n@@ -2,6 +2,7 @@\n /* Interface to Sjoerd's portable C thread library */\n \n #include \"Python.h\"\n+#include \u003cunistd.h> // usleep(), PROBE\n #include \"pycore_fileutils.h\" // _PyFile_Flush\n #include \"pycore_interp.h\" // _PyInterpreterState.threads.count\n #include \"pycore_lock.h\"\n@@ -382,6 +383,12 @@ thread_run(void *boot_raw)\n goto exit;\n }\n \n+ { // PROBE: a pause at every 16th thread start\n+ static int probe_count;\n+ if ((_Py_atomic_add_int(&probe_count, 1) & 15) == 0) {\n+ usleep(3000);\n+ }\n+ }\n _PyThreadState_Bind(tstate);\n PyEval_AcquireThread(tstate);\n _Py_atomic_add_ssize(&tstate->interp->threads.count, 1);\n```\n\nThe C extension: 8 threads that loop over `PyThreadState_New()`, `PyEval_RestoreThread()`, 20 new bytes objects into a shared list (so that finalization decrefs objects they own), `PyThreadState_Clear()` and `PyThreadState_DeleteCurrent()`. It ran in the same kind of loop, with `import natthreads, time; natthreads.start(8, [None] * 256); time.sleep(0.02)` as the process.\n\n```c\n#include \u003cPython.h>\n#include \u003cpthread.h>\n\nstatic PyInterpreterState *interp;\nstatic PyObject *sink; /* a list owned by the main thread */\n\nstatic void *\nworker(void *arg)\n{\n for (Py_ssize_t i = (Py_ssize_t)arg;; i++) {\n PyThreadState *ts = PyThreadState_New(interp);\n PyEval_RestoreThread(ts);\n for (int k = 0; k \u003c 20; k++) {\n PyObject *b = PyBytes_FromStringAndSize(NULL, 64);\n if (b != NULL) {\n PyList_SetItem(sink, (i * 20 + k) % PyList_GET_SIZE(sink), b); /* steals b */\n }\n }\n PyThreadState_Clear(ts);\n PyThreadState_DeleteCurrent();\n }\n return NULL;\n}\n\nstatic PyObject *\nstart(PyObject *self, PyObject *args)\n{\n int n;\n if (!PyArg_ParseTuple(args, \"iO!\", &n, &PyList_Type, &sink)) {\n return NULL;\n }\n Py_INCREF(sink);\n interp = PyInterpreterState_Get();\n for (int i = 0; i \u003c n; i++) {\n pthread_t t;\n if (pthread_create(&t, NULL, worker, (void *)(Py_ssize_t)(i * 1000)) != 0) {\n return PyErr_SetFromErrno(PyExc_OSError);\n }\n pthread_detach(t);\n }\n Py_RETURN_NONE;\n}\n\nstatic PyMethodDef methods[] = {{\"start\", start, METH_VARARGS, NULL}, {NULL, NULL, 0, NULL}};\nstatic PyModuleDef_Slot slots[] = {{Py_mod_gil, Py_MOD_GIL_NOT_USED}, {0, NULL}};\nstatic struct PyModuleDef module = {PyModuleDef_HEAD_INIT, \"natthreads\", NULL, 0, methods, slots};\nPyMODINIT_FUNC PyInit_natthreads(void) { return PyModuleDef_Init(&module); }\n```\n\n\u003c/details>\n\n\n### CPython versions tested on:\n\nCPython main branch\n\n### Operating systems tested on:\n\nLinux\n\n### Output from running 'python -VV' on the command line:\n\nPython 3.16.0a1+dev free-threading build (main, Nov 11 2026, 08:33:39) [GCC 14.2.0]","author":{"url":"https://github.com/maikschulze","@type":"Person","name":"maikschulze"},"datePublished":"2026-11-16T11:53:56Z","url":"https://github.com/python/cpython/issues/162109"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:dec4f708-171b-8bdf-8ccf-101361228de0 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | BB40:14EB6E:15E6A47:1CD4E95:6B01F9E4 |
| html-safe-nonce | 18495c778d09291f8a7ef5b1c05bf6693adc054d892b00c52e0fdb40fea2f0e4 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCQjQwOjE0RUI2RToxNUU2QTQ3OjFDRDRFOTU6NkIwMUY5RTQiLCJ2aXNpdG9yX2lkIjoiNzk3ODQ2MDA3MzA3MTcyMzI0IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | 42020c51da746bccee2e4f1cdcdcdc60f9476d51db14eb202fe304a954c3a10d |
| hovercard-subject-tag | issue:6420621093 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/162109/issue_layout |
| twitter:image | https://opengraph.githubassets.com/8e01e767b273cd1bf2194ab08e6b616f13db037d0ccb18443c0e54125cd69efe/python/cpython/issues/162109 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/8e01e767b273cd1bf2194ab08e6b616f13db037d0ccb18443c0e54125cd69efe/python/cpython/issues/162109 |
| og:image:alt | Crash report I was exploring peak RSS gaps of the free-threaded build with mimalloc v3 (#159718) with overnight stress tests. With daemon threads, the process sometimes crashed at exit. The cause l... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | maikschulze |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width