René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"`_interpreters` is not thread safe on the free-threaded build","articleBody":"# Crash report\r\n\r\n### What happened?\r\n\r\nFirst off, sorry for not being able to offer code that is more reduced and certain to trigger a repro.\r\n\r\nThe code below non-deterministically triggers `python: Python/index_pool.c:92: heap_pop: Assertion 'heap-\u003esize \u003e 0' failed.` in a free-threading build with `PYTHON_GIL=0`.\r\n\r\n```python\r\nimport _interpreters\r\nfrom threading import Thread\r\n\r\ndef f(): pass\r\n\r\nints = []\r\nfor x in range(300):\r\n    ints.append(_interpreters.create())\r\n\r\nthreads = []\r\nfor interpr in ints:\r\n    threads.append(Thread(target=_interpreters.run_string, args=(interpr, \"f()\",)))\r\n    threads.append(Thread(target=_interpreters.get_current, args=()))\r\n    threads.append(Thread(target=_interpreters.destroy, args=(interpr,)))\r\n    threads.append(Thread(target=_interpreters.list_all, args=()))\r\n    threads.append(Thread(target=_interpreters.destroy, args=(interpr,)))\r\n    threads.append(Thread(target=_interpreters.get_current, args=()))\r\n    threads.append(Thread(target=_interpreters.get_main, args=()))\r\n    threads.append(Thread(target=_interpreters.destroy, args=(interpr,)))\r\n    threads.append(Thread(target=_interpreters.run_string, args=(interpr, \"f()\",)))\r\n\r\nfor thread in threads:\r\n    try:\r\n        print(\"START\", thread)\r\n        thread.start()\r\n    except Exception:\r\n        pass\r\n\r\nfor thread in threads:\r\n    try:\r\n        print(\"JOIN\", thread)\r\n        thread.join()\r\n    except Exception:\r\n        pass\r\n```\r\n\r\nBacktrace:\r\n```gdb\r\n#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140729519150656) at ./nptl/pthread_kill.c:44\r\n#1  __pthread_kill_internal (signo=6, threadid=140729519150656) at ./nptl/pthread_kill.c:78\r\n#2  __GI___pthread_kill (threadid=140729519150656, signo=signo@entry=6) at ./nptl/pthread_kill.c:89\r\n#3  0x00007ffff7ce0476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26\r\n#4  0x00007ffff7cc67f3 in __GI_abort () at ./stdlib/abort.c:79\r\n#5  0x00007ffff7cc671b in __assert_fail_base (fmt=0x7ffff7e7b130 \"%s%s%s:%u: %s%sAssertion `%s' failed.\\n%n\",\r\n    assertion=0x555555aceaf4 \"heap-\u003esize \u003e 0\", file=0x555555aceae0 \"Python/index_pool.c\", line=92, function=\u003coptimized out\u003e)\r\n    at ./assert/assert.c:92\r\n#6  0x00007ffff7cd7e96 in __GI___assert_fail (assertion=assertion@entry=0x555555aceaf4 \"heap-\u003esize \u003e 0\",\r\n    file=file@entry=0x555555aceae0 \"Python/index_pool.c\", line=line@entry=92,\r\n    function=function@entry=0x555555aceb40 \u003c__PRETTY_FUNCTION__.1\u003e \"heap_pop\") at ./assert/assert.c:101\r\n#7  0x00005555558caa1c in heap_pop (heap=\u003coptimized out\u003e) at Python/index_pool.c:92\r\n#8  0x00005555558cac33 in _PyIndexPool_AllocIndex (pool=pool@entry=0x7ffff780f498) at Python/index_pool.c:173\r\n#9  0x000055555568d9f0 in _Py_ReserveTLBCIndex (interp=interp@entry=0x7ffff780b020) at Objects/codeobject.c:2752\r\n#10 0x0000555555951c33 in new_threadstate (interp=interp@entry=0x7ffff780b020, whence=whence@entry=5) at Python/pystate.c:1516\r\n#11 0x0000555555953994 in _PyThreadState_NewBound (interp=interp@entry=0x7ffff780b020, whence=whence@entry=5)\r\n    at Python/pystate.c:1578\r\n#12 0x0000555555896ee9 in _enter_session (session=session@entry=0x7ffe24ff8740, interp=interp@entry=0x7ffff780b020)\r\n    at Python/crossinterp.c:1548\r\n#13 0x000055555589b9fb in _PyXI_Enter (session=session@entry=0x7ffe24ff8740, interp=interp@entry=0x7ffff780b020,\r\n    nsupdates=nsupdates@entry=0x0) at Python/crossinterp.c:1711\r\n#14 0x00007ffff7c3d217 in _run_in_interpreter (interp=interp@entry=0x7ffff780b020, codestr=0x200006182c8 \"f()\", codestrlen=3,\r\n    shareables=shareables@entry=0x0, flags=1, p_excinfo=p_excinfo@entry=0x7ffe24ff8870) at ./Modules/_interpretersmodule.c:461\r\n#15 0x00007ffff7c3d3ef in _interp_exec (self=self@entry=\u003cmodule at remote 0x20000778b30\u003e, interp=interp@entry=0x7ffff780b020,\r\n    code_arg=\u003coptimized out\u003e, shared_arg=0x0, p_excinfo=p_excinfo@entry=0x7ffe24ff8870) at ./Modules/_interpretersmodule.c:950\r\n#16 0x00007ffff7c3dc6b in interp_run_string (self=\u003cmodule at remote 0x20000778b30\u003e, args=args@entry=(9, 'f()'), kwds=kwds@entry={})\r\n    at ./Modules/_interpretersmodule.c:1110\r\n#17 0x000055555570145a in cfunction_call (func=func@entry=\u003cbuilt-in method run_string of module object at remote 0x20000778b30\u003e,\r\n    args=args@entry=(9, 'f()'), kwargs=kwargs@entry={}) at Objects/methodobject.c:551\r\n#18 0x000055555567f69a in _PyObject_Call (tstate=0x5555560c09d0,\r\n    callable=callable@entry=\u003cbuilt-in method run_string of module object at remote 0x20000778b30\u003e, args=args@entry=(9, 'f()'),\r\n    kwargs=kwargs@entry={}) at Objects/call.c:361\r\n```\r\n\r\nThis code most usually results in one of the following errors, from most common to rarest:\r\n\r\n1. tstate_delete_common assertion, which seems to indicate a thread creation failed, see https://github.com/python/cpython/issues/109746.\r\n```\r\npython: Python/pystate.c:1739: tstate_delete_common: Assertion `tstate-\u003e_status.cleared \u0026\u0026 !tstate-\u003e_status.finalized' failed.\r\n```\r\n\r\n2. Fatal Python error, seems to be https://github.com/python/cpython/issues/113148.\r\n\r\n```\r\nFatal Python error: Py_EndInterpreter: not the last thread\r\nPython runtime state: initialized\r\n\r\nCurrent thread 0x00007f09af37e640 (most recent call first):\r\n  \u003cno Python frame\u003e\r\n\r\nThread 0x00007f09b0380640 (most recent call first):\r\n  \u003cno Python frame\u003e\r\nAborted\r\n```\r\n\r\nGiven that, running it multiple times seems necessary to trigger the correct abort in `heap_pop`. The affected code seems to have been included in https://github.com/python/cpython/pull/123926, so cc @mpage.\r\n\r\nFound using fusil by @vstinner.\r\n\r\n### CPython versions tested on:\r\n\r\nCPython main branch\r\n\r\n### Operating systems tested on:\r\n\r\nLinux\r\n\r\n### Output from running 'python -VV' on the command line:\r\n\r\nPython 3.14.0a1+ experimental free-threading build (heads/main-dirty:54c63a32d0, Nov  8 2024, 20:16:36) [GCC 11.4.0]\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-126696\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/devdanzin","@type":"Person","name":"devdanzin"},"datePublished":"2024-11-10T15:09:04.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":6},"url":"https://github.com/126644/cpython/issues/126644"}