René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"destroy_gil() is not called over multiple Py_Initialize() / Py_Finalize() calls","articleBody":"The following simple balanced calls to `Py_Initialize()` / `Py_FinalizeEx()` trigger an error when running under Microsoft's Application Verifier on Windows:\r\n\r\n```c\r\n  Py_Initialize();\r\n  Py_FinalizeEx();\r\n  Py_Initialize();\r\n  Py_FinalizeEx();\r\n```\r\n\r\nThe above sequence causes there to be two calls to `create_gil()` for the main GIL, without an intervening call to `destroy_gil()`: the mismatched use of `PyMUTEX_INIT()`/`PyMUTEX_FINI()` (`create_gil()` calls `PyMUTEX_INIT()` and `destroy_gil()` calls `PyMUTEX_FINI()`) translates into mismatched calls to Win32 APIs [`InitializeCriticalSection()`](https://learn.microsoft.com/en-us/windows/win32/api/synchapi/nf-synchapi-initializecriticalsection) and [`DeleteCriticalSection()`](https://learn.microsoft.com/en-us/windows/win32/api/synchapi/nf-synchapi-deletecriticalsection), which are then detected by Application Verifier.\r\n\r\n\\[Note, however, that this error is not specific to Windows and the GIL is being incorrectly re-initialised without a call to `PyMUTEX_FINI()` on all platforms.\\]\r\n\r\nThe first `Py_Initialize()` causes a call to `create_gil()` on `interp-\u003eceval.gil` for the main interpreter (`_PyRuntime._main_interpreter`), but as a result of bpo-9901, `finalize_interp_delete()` (as called from `Py_FinalizeEx()`) currently defers calling `_PyEval_FiniGIL()` (and thereby `destroy_gil()`) until the next `Py_Initialize()`:\r\n\r\n```c\r\nstatic void\r\nfinalize_interp_delete(PyInterpreterState *interp)\r\n{\r\n    /* Cleanup auto-thread-state */\r\n    _PyGILState_Fini(interp);\r\n\r\n    /* We can't call _PyEval_FiniGIL() here because destroying the GIL lock can\r\n       fail when it is being awaited by another running daemon thread (see\r\n       bpo-9901). Instead pycore_create_interpreter() destroys the previously\r\n       created GIL, which ensures that Py_Initialize / Py_FinalizeEx can be\r\n       called multiple times. */\r\n```\r\nhttps://github.com/python/cpython/blame/41aff464cef83d2655029ddd180a51110e8d7f8e/Python/pylifecycle.c#L1740\r\n\r\nUnfortunately, the second call `Py_Initialize()` doesn't call `destroy_gil()` before calling `create_gil()` a second time on the main interpreter's GIL, causing this error.\r\n\r\n@ericsnowcurrently's [comment on `init_interp_create_gil()`](https://github.com/python/cpython/blame/41aff464cef83d2655029ddd180a51110e8d7f8e/Python/pylifecycle.c#L594) ('`XXX This is broken with a per-interpreter GIL`'):\r\n\r\n```c\r\nstatic PyStatus\r\ninit_interp_create_gil(PyThreadState *tstate, int own_gil)\r\n{\r\n    PyStatus status;\r\n\r\n    /* finalize_interp_delete() comment explains why _PyEval_FiniGIL() is\r\n       only called here. */\r\n    // XXX This is broken with a per-interpreter GIL.\r\n    _PyEval_FiniGIL(tstate-\u003einterp);\r\n```\r\n\r\nnotes that the call to `_PyEval_FiniGIL()` in `init_interp_create_gil()` doesn't work with per-interpreter GIL, but it's worse than this as it's broken with just re-creating the main interpreter!\r\n\r\nThe reason for all this appears to be that the GIL is re-initialised by `_PyRuntime_Initialize()`, before `_PyEval_FiniGIL()` has a chance to get its hands on it. Prior to GH-104210 from @ericsnowcurrently landing yesterday (and on previous Python versions), `_PyEval_FiniGIL()` would detect that the GIL is reinitialised by a prior call to `_gil_initialize()` within `_PyRuntime_Initialize()`. However, now the main interpreter (`_PyRuntime._main_interpreter`) has a GIL pointer (`interp-\u003eceval.gil`) which is cleared when `_PyRuntimeState_Init()` resets the runtime to `_PyRuntimeState_INIT`:\r\n\r\n```c\r\n    if (runtime-\u003e_initialized) {\r\n        // Py_Initialize() must be running again.\r\n        // Reset to _PyRuntimeState_INIT.\r\n        memcpy(runtime, \u0026initial, sizeof(*runtime));\r\n    }\r\n```\r\n\r\nI have reproduced this with recent cpython commit [`bf89d4283a28dd00836f2c312a9255f543f93fc7`](https://github.com/python/cpython/commit/bf89d4283a28dd00836f2c312a9255f543f93fc7) and have attached a log of the callstacks \\([avrf-gil.txt](https://github.com/python/cpython/files/11431606/avrf-gil.txt)\\).\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-104325\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/adr26","@type":"Person","name":"adr26"},"datePublished":"2023-05-09T13:35:38.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":4},"url":"https://github.com/104324/cpython/issues/104324"}