René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"3.12: \"Python memory allocator called without holding the GIL\" with `PyMem_Free` as `Py_AtExit`","articleBody":"When importing PyQt6 with Python 3.12.0b2 (as well as on the current git 3.12, 0b305e8d36c5ac5dc83f0c09e7e2b3360b992eea), it segfaults on exit.\r\n\r\n```c\r\nProgram received signal SIGSEGV, Segmentation fault.\r\n0x000055555572563a in _PyInterpreterState_GET () at ./Include/internal/pycore_pystate.h:126\r\n126\t   return tstate-\u003einterp;\r\n(gdb) bt\r\n#0  0x000055555572563a in _PyInterpreterState_GET () at ./Include/internal/pycore_pystate.h:126\r\n#1  get_state () at Objects/obmalloc.c:866\r\n#2  _PyObject_Free (ctx=\u003coptimized out\u003e, p=0x555555d527a0) at Objects/obmalloc.c:1850\r\n#3  _PyObject_Free (ctx=0x0, p=0x555555d527a0) at Objects/obmalloc.c:1843\r\n#4  0x00007ffff75e5351 in finalise () at sip_core.c:1807\r\n#5  0x0000555555835235 in call_ll_exitfuncs (runtime=0x555555ba6380 \u003c_PyRuntime\u003e) at Python/pylifecycle.c:3002\r\n#6  Py_FinalizeEx () at Python/pylifecycle.c:1966\r\n#7  0x0000555555838075 in Py_FinalizeEx () at Python/pylifecycle.c:1970\r\n#8  0x00005555558693cf in Py_RunMain () at Modules/main.c:691\r\n[...]\r\n```\r\n\r\nA `--with-pydebug` build instead prints:\r\n\r\n```\r\nFatal Python error: _PyMem_DebugFree: Python memory allocator called without holding the GIL                                                                                                                                                  \r\nPython runtime state: finalizing (tstate=0x0000555555c76490)\r\n```\r\n\r\nand aborts in:\r\n\r\n```c\r\n[...]\r\n#4  0x0000555555847cc2 in fatal_error_exit (status=\u003coptimized out\u003e) at Python/pylifecycle.c:2693\r\n#5  0x000055555584957c in fatal_error (fd=2, header=header@entry=1, prefix=prefix@entry=0x55555591fa10 \u003c__func__.3\u003e \"_PyMem_DebugFree\", msg=msg@entry=0x5555559202d8 \"Python memory allocator called without holding the GIL\", \r\n    status=status@entry=-1) at Python/pylifecycle.c:2874\r\n#6  0x00005555558495e6 in _Py_FatalErrorFunc (func=func@entry=0x55555591fa10 \u003c__func__.3\u003e \"_PyMem_DebugFree\", msg=msg@entry=0x5555559202d8 \"Python memory allocator called without holding the GIL\") at Python/pylifecycle.c:2890\r\n#7  0x000055555573f9b3 in _PyMem_DebugCheckGIL (func=func@entry=0x55555591fa10 \u003c__func__.3\u003e \"_PyMem_DebugFree\") at Objects/obmalloc.c:2271\r\n#8  0x000055555573fa30 in _PyMem_DebugFree (ctx=0x555555c024d0 \u003c_PyRuntime+208\u003e, ptr=0x555555dd67b0) at Objects/obmalloc.c:2295\r\n#9  0x00005555557408e7 in PyMem_Free (ptr=\u003coptimized out\u003e) at Objects/obmalloc.c:732\r\n#10 0x00007ffff74e5296 in sip_api_free (mem=\u003coptimized out\u003e) at sip_core.c:1905\r\n#11 0x00007ffff74f529a in sipOMFinalise (om=om@entry=0x7ffff74ff820 \u003ccppPyMap\u003e) at sip_object_map.c:69\r\n#12 0x00007ffff74e5351 in finalise () at sip_core.c:1807\r\n#13 0x00005555558479d0 in call_ll_exitfuncs (runtime=runtime@entry=0x555555c02400 \u003c_PyRuntime\u003e) at Python/pylifecycle.c:3002\r\n#14 0x0000555555849153 in Py_FinalizeEx () at Python/pylifecycle.c:1966\r\n#15 0x0000555555875aa7 in Py_RunMain () at Modules/main.c:691\r\n[...]\r\n```\r\n\r\nTo reproduce, `pip install PyQt6` followed by e.g. `python3 -c \"import PyQt6.QtCore\"`.\r\n\r\n---\r\n\r\nIt's unclear to me whether this is an intended/expected regression or not, given that the behavior of `sip` (the C++ bindings behind PyQt) seems somewhat unkosher here. Its [maintainer says](https://www.riverbankcomputing.com/pipermail/pyqt/2023-June/045332.html):\r\n\r\n\u003e `finalise()` is registered with `Py_AtExit()` so that it is called by `Py_FinalizeEx()`...\r\n\u003e \r\n\u003e https://docs.python.org/3/c-api/sys.html#c.Py_AtExit\r\n\u003e \r\n\u003e The docs say nothing about the state of the GIL when a cleanup function is called. The docs also state that the cleanup handler should not make calls to the Python API, so I suppose the state of the GIL shouldn't matter.\r\n\u003e \r\n\u003e However `finalise()` does make calls to the Python API - but only to free memory (one of the stated purposes of `Py_FinalizeEx()`). It seems a little bit contradictory to me to disallow cleanup functions calling `Py_MemFree()`.\r\n\u003e \r\n\u003e If this new behaviour were to remain then sip would have to stop using the Python memory allocator and just use `malloc()` and `free()`.\r\n\r\nSo from what I understand, it sounds like this was \"if it breaks, you got to keep both pieces\" territory to begin with?\r\n\r\nHowever, I was able to bisect this to https://github.com/python/cpython/commit/6036c3e856f033bf13e929536e7bf127fdd921c9 as part of #101161 (\"Clarify GILState-related Code\"), which in the commit message claims to be a refactor only:\r\n\r\n\u003e The objective of this change is to help make the GILState-related code easier to understand.  This mostly involves moving code around and some semantically equivalent refactors.  However, there are a also a small number of slight changes in structure and behavior:\r\n\u003e \r\n\u003e * tstate_current is moved out of _PyRuntimeState.gilstate\r\n\u003e * autoTSSkey is moved out of _PyRuntimeState.gilstate\r\n\u003e * autoTSSkey is initialized earlier\r\n\u003e * autoTSSkey is re-initialized (after fork) earlier\r\n\r\nThus, I'm erring on the side of caution here, and decided to open an issue anyways - if this is something sip definitely shouldn't be doing that way, feel free to close!\r\n\r\nThe relevant sip sources can be found [on their Mercurial server](https://www.riverbankcomputing.com/hg/sip/file/tip/sipbuild/module/source/13) or in the PyPI sdist (`sipbuild/module/source/13/`). \r\n\r\ncc @philthompson10 @ericsnowcurrently ","author":{"url":"https://github.com/The-Compiler","@type":"Person","name":"The-Compiler"},"datePublished":"2023-06-12T13:08:50.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":5},"url":"https://github.com/105690/cpython/issues/105690"}