René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"CPython 3.12 embedded in WeeChat causes segfault on subsequent calls to Py_EndInterpreter","articleBody":"# Crash report\r\n\r\n### What happened?\r\n\r\nWeeChat embeds CPython in order to run Python scripts inside WeeChat. It can load multiple scripts and they each get their own interpreter. When a script is loaded `Py_NewInterpreter` is called, and when it's unloaded `Py_EndInterpreter` is called.\r\n\r\nWith CPython 3.12 loading two scripts and then unloading them in the same order causes a segmentation fault. Interestingly, the segmentation fault doesn't happen if the script that was loaded last is unloaded first.\r\n\r\nI bisected this and found it was introduced in commit de64e7561680fdc5358001e9488091e75d4174a3. I also noticed that the crash doesn't occur in the main branch, and did another bisect and found it was fixed in commit 7a7bce5a0ab249407e866a1e955d21fa2b0c8506.\r\n\r\nThis issue seems similar to the one reported in #115649 which is also introduced by the same commit, but that one still crashes on the main branch (commit 735fc2cbbcf875c359021b5b2af7f4c29f4cf66d).\r\n\r\nI haven't been able to reproduce this outside of WeeChat unfortunately, but here is a backtrace from the crash with WeeChat, with commit de64e7561680fdc5358001e9488091e75d4174a3 of CPython and commit ec56a1103f47b15a641ff93528fd6f50025dd524 of WeeChat.\r\n\r\n```\r\nProgram terminated with signal SIGSEGV, Segmentation fault.\r\n#0  0x000074b5e5fd2700 in ?? ()\r\n[Current thread is 1 (Thread 0x74b5e7bec940 (LWP 1881931))]\r\n(gdb) bt\r\n#0  0x000074b5e5fd2700 in ?? ()\r\n#1  \u003csignal handler called\u003e\r\n#2  0x000074b5e64f9ddd in _PyGCHead_SET_PREV (prev=\u003coptimized out\u003e, gc=\u003coptimized out\u003e) at ./Include/internal/pycore_gc.h:74\r\n#3  _PyObject_GC_UNTRACK (op=0x74b5dfb746d0) at ./Include/internal/pycore_object.h:228\r\n#4  PyObject_GC_UnTrack (op_raw=op_raw@entry=0x74b5dfb746d0) at Modules/gcmodule.c:2241\r\n#5  0x000074b5e63c430c in module_dealloc (m=0x74b5dfb746d0) at Objects/moduleobject.c:672\r\n#6  0x000074b5e63c393d in Py_DECREF (op=\u003coptimized out\u003e) at ./Include/object.h:681\r\n#7  Py_XDECREF (op=\u003coptimized out\u003e) at ./Include/object.h:777\r\n#8  meth_dealloc (m=0x74b5dfb81210) at Objects/methodobject.c:170\r\n#9  0x000074b5e63b5d00 in Py_DECREF (op=0x74b5dfb81210) at ./Include/object.h:681\r\n#10 Py_XDECREF (op=0x74b5dfb81210) at ./Include/object.h:777\r\n#11 insertdict (interp=0x74b5e0e42010, mp=mp@entry=0x74b5df1d3d40, key=0x74b5dfb7db30, hash=\u003coptimized out\u003e, value=value@entry=0x74b5e6750240 \u003c_Py_NoneStruct\u003e) at Objects/dictobject.c:1304\r\n#12 0x000074b5e63b6107 in _PyDict_SetItem_Take2 (value=0x74b5e6750240 \u003c_Py_NoneStruct\u003e, key=\u003coptimized out\u003e, mp=0x74b5df1d3d40) at Objects/dictobject.c:1854\r\n#13 0x000074b5e63c5684 in _PyModule_ClearDict (d=0x74b5df1d3d40) at Objects/moduleobject.c:619\r\n#14 0x000074b5e63c5a6e in _PyModule_Clear (m=m@entry=0x74b5df1e12b0) at Objects/moduleobject.c:567\r\n#15 0x000074b5e64c884e in finalize_modules_clear_weaklist (verbose=0, weaklist=0x74b5dfbed080, interp=0x74b5e0e42010) at Python/pylifecycle.c:1491\r\n#16 finalize_modules (tstate=tstate@entry=0x74b5e0ea0400) at Python/pylifecycle.c:1574\r\n#17 0x000074b5e64cc476 in Py_EndInterpreter (tstate=0x74b5e0ea0400) at Python/pylifecycle.c:2137\r\n#18 0x000074b5e6a5bce6 in weechat_python_unload (script=0x5d9a7b80ee60) at /home/trygve/dev/weechat/src/plugins/python/weechat-python.c:947\r\n#19 0x000074b5e6a5bea6 in weechat_python_unload_all () at /home/trygve/dev/weechat/src/plugins/python/weechat-python.c:996\r\n#20 0x000074b5e7b81cea in plugin_script_end (weechat_plugin=0x5d9a7b29ad50, plugin_data=0x74b5e6a9e640 \u003cpython_data\u003e) at /home/trygve/dev/weechat/src/plugins/plugin-script.c:1841\r\n#21 0x000074b5e6a5d8c4 in weechat_plugin_end (plugin=0x5d9a7b29ad50) at /home/trygve/dev/weechat/src/plugins/python/weechat-python.c:1634\r\n#22 0x00005d9a78ed4c38 in plugin_unload (plugin=0x5d9a7b29ad50) at /home/trygve/dev/weechat/src/plugins/plugin.c:1261\r\n#23 0x00005d9a78ed4d9f in plugin_unload_all () at /home/trygve/dev/weechat/src/plugins/plugin.c:1313\r\n#24 0x00005d9a78ed50f8 in plugin_end () at /home/trygve/dev/weechat/src/plugins/plugin.c:1433\r\n#25 0x00005d9a78e04340 in weechat_end (gui_end_cb=0x5d9a78ecb8ae \u003cgui_main_end\u003e) at /home/trygve/dev/weechat/src/core/weechat.c:709\r\n#26 0x00005d9a78e03163 in main (argc=4, argv=0x7ffdb171c178) at /home/trygve/dev/weechat/src/gui/curses/normal/main.c:45\r\n```\r\n\r\nThis was produced by creating these two python scripts:\r\n\r\n`dummy1.py`:\r\n```python\r\nimport weechat\r\n\r\nif weechat.register(\"dummy1\", \"trygveaa\", \"0.1\", \"MIT\", \"Dummy script 1\", \"\", \"\"):\r\n    weechat.prnt(\"\", \"Loaded dummy script 1\")\r\n```\r\n\r\n`dummy2.py`\r\n```python\r\nimport weechat\r\n\r\nif weechat.register(\"dummy2\", \"trygveaa\", \"0.1\", \"MIT\", \"Dummy script 2\", \"\", \"\"):\r\n    weechat.prnt(\"\", \"Loaded dummy script 2\")\r\n```\r\n\r\nAnd then running `weechat -t -r '/script load dummy1.py; /script load dummy2.py; /quit'`.\r\n\r\nAlso, here is the issue report for WeeChat: https://github.com/weechat/weechat/issues/2046\r\n\r\nSince it's fixed in main it seems there won't be a problem with 3.13, but I wonder if the fix can be backported to 3.12?\r\n\r\n### CPython versions tested on:\r\n\r\n3.12, CPython main branch\r\n\r\n### Operating systems tested on:\r\n\r\nLinux\r\n\r\n### Output from running 'python -VV' on the command line:\r\n\r\nPython 3.12.0a7+ (tags/v3.12.0a7-340-gde64e75616:de64e75616, Mar  8 2024, 19:43:39) [GCC 13.2.1 20230801]\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-124536\n* gh-124541\n* gh-124645\n* gh-124646\n* gh-124648\n* gh-124649\n* gh-124865\n* gh-125204\n* gh-125205\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/trygveaa","@type":"Person","name":"trygveaa"},"datePublished":"2024-03-08T19:12:21.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":17},"url":"https://github.com/116510/cpython/issues/116510"}