René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"The Word \"Finalizing\" in C-API Function Names is Sometimes Misleading","articleBody":"(This is something I've been thinking about in the last few weeks and has before more significant as we've added `Py_IsFinalizing()` to the 3.13 public C-API[^1].)\r\n\r\n[^1]: ...and possibly to the stable ABI.  See gh-110441.\r\n\r\ntl;dr I think that either we should make the `Py_IsFinalizing()` behavior match the name more closely or we should change the name.  Internally we should think in terms of what runtime capabilities are available rather than just \"is finalizing\".\r\n\r\n----\r\n\r\nAs of recently, we have `Py_IsFinalizing()` in the public C-API, along with various similarly named (`.*[Ff]inalizing.*`) functions and runtime state fields in the internal API. [^2]  In addition, `sys.is_finalizing()` has been around since 3.5[^3].\r\n\r\n[^2]: We also have the fundamentally related `Py_Finalize()` and `Py_FinalizeEx()`, which have been with us for a long, long time.\r\n[^3]: `sys.finalizing()` is a thin wrapper around `Py_IsFinalizing()`.  Before 3.13, it wrapped `_Py_IsFinalizing()`.  Before 3.7, it wrapped `_Py_Finalizing`.\r\n\r\nThey all serve to track/inform when the current runtime/interpreter is shutting down and therefore might not be in a stable state (leading to crashes).  The intention is clear in the docs for [`Py_IsFinalizing()`](https://docs.python.org/3.13/c-api/init.html#c.Py_IsFinalizing):\r\n\r\n```\r\nint Py_IsFinalizing()\r\n    Return true (non-zero) if the main Python interpreter is shutting down. Return false (zero) otherwise.\r\n```\r\n\r\nInternally, we use that status to determine the availability of certain capabilities, most notably threading.  In fact, the whole family of functions and state started in 2011 as part of the effort to deal with daemon threads during shutdown.\r\n\r\nThis has lead to a bit of a conflict in the meaning of \"finalizing\".  IMHO, it would be worth sorting out the discrepancy.  I expect this will include either changing the behavior of `Py_IsFinalizing()` or changing the name of the function to match what it is actually reporting.\r\n\r\n(Is this a critical issue?  No.  The matter at hand is fundamentally related to daemon threads. 🤮)\r\n(Is it worth thinking through anyway?  Yes.  I expect the discussion would help bring more clarity to runtime finalization and to the runtime in general.)\r\n\r\n\r\n## Context\r\n\r\nFinalization is started for the runtime by `Py_Finalize()` and `Py_FinalizeEx()`.  For an interpreter we use `Py_EndInterpreter()`.  (For thread states we don't have a specific API, but the closest is `PyThreadState_Clear()` + `PyThreadState_Delete()`.)\r\n\r\n\u003cdetails\u003e\r\n\u003csummary\u003eOnce finalization has begun, we keep track of that fact in a number of places:\u003c/summary\u003e\r\n\r\n* `_PyRuntimeState._finalizing`\r\n* `_PyRuntimeState._finalizing_id`\r\n* `PyInterpreterState.finalizing`\r\n* `PyInterpreterState._finalizing`\r\n* `PyInterpreterState._finalizing_id`\r\n* `PyThreadState._status.finalizing`\r\n\r\n\u003c/details\u003e\r\n\r\n\u003cdetails\u003e\r\n\u003csummary\u003eWe report it through a various API:\u003c/summary\u003e\r\n\r\n* `Py_IsFinalizing()`  [3.13] (docs: \"Return true (non-zero) if the main Python interpreter is shutting down. Return false (zero) otherwise.\")\r\n* `sys.is_finalizing()`  [3.5] (docs: \"Return True if the Python interpreter is shutting down, False otherwise.\")\r\n* `_PyRuntimeState_GetFinalizing()`\r\n* `_PyRuntimeState_GetFinalizingID()`\r\n* `_PyInterpreterState_GetFinalizing()`\r\n* `_PyInterpreterState_GetFinalizingID()`\r\n\r\n\u003c/details\u003e\r\n\r\n\u003cdetails\u003e\r\n\u003csummary\u003eHere's a timeline:\u003c/summary\u003e\r\n\r\n* [2008, ~3.0] [bpo-1856](https://bugs.python.org/issue1856) (AKA [gh-46164](https://github.com/python/cpython/issues/46164)) opened about crashes with daemon threads during runtime finalization\r\n* [2011, 3.2] added `_Py_Finalizing` as part of the effort to solve bpo-1856\r\n* [3.5] added `sys.is_finalizing()`, a light wrapper around `_Py_Finalizing`\r\n* [3.7] replaced `_Py_Finalizing` with `_PyRuntimeState.finalizing`; added `_Py_IsFinalizing()` (in \"public\" C-API)\r\n* [2019] ([gh-80608](https://github.com/python/cpython/issues/80608)) `_Py_Finalizing()` recommended for use in docs for [`PyEval_RestoreThread()`](https://docs.python.org/3.13/c-api/init.html#c.PyEval_RestoreThread)\r\n* [2020, 3.9] added `_Py_GetFinalizing()`; renamed `_PyRuntimeState.finalizing` to `_PyRuntimeState._finalizing` (and made it atomic)\r\n* [July 2023, 3.13] ([gh-106400](https://github.com/python/cpython/pull/106400)) moved `_Py_IsFinalizing()` to the internal C-API\r\n* [Aug. 2023, 3.13] ([gh-108014](https://github.com/python/cpython/issues/108014)/[gh-108032](https://github.com/python/cpython/pull/108032))  added `_Py_IsFinalizing()` back to the public C-API as `Py_IsFinalizing()`\r\n* [this week] ([gh-110397](https://github.com/python/cpython/issues/110397)/[gh-110441](https://github.com/python/cpython/pull/110441)) discussion about adding `Py_IsFinalizing()` to the stable ABI\r\n\r\n\u003c/details\u003e\r\n\r\n\r\n## The Problem\r\n\r\nThe name `Py_IsFinalizing` (or `sys.is_finalizing`[^2]) implies that the function tells you if the runtime is shutting down and will soon be unavailable.  In fact, the docs actually say this.  However, that explanation is only *mostly* accurate.  When the function returns true, that is completely correct, but the same is not *completely* correct when it returns false.  So, where might returning false be incorrect?\r\n\r\nCurrently, there is actually a meaningful space of time between when `Py_Finalize()` is called and when `Py_IsFinalizing()` returns true.  In that time we do a number of things, like wait for all non-daemon threads to stop and run all global atexit hooks.  Essentially, `Py_IsFinalizing()` returns true only at the point in finalization where other threads (i.e. not the current/main thread) should no longer rely on a stable runtime or C-API.  (Perhaps the function should be called something like `Py_AreThreadsAllowed()` instead.)\r\n\r\nTracking that point in time is important for how we handle daemon threads during shutdown.  We shouldn't change that.\r\n\r\nTo resolve any confusion and ambiguity here, we should instead:\r\n\r\n* explicitly acknowledge what we're actually tracking (in name and description)\r\n* decide what we care about internally (and why)\r\n* determine what information users actually want\r\n\r\nInternally we care about two things: whether or not the runtime currently supports multithreading and whether or not the runtime (or current interpreter) is reaching the end of its life.\r\n\r\nFor users of `PyEval_RestoreThread()`, they want to know if the current thread will be terminated if they call that function to re-acquire the GIL.  That matters for extensions running in daemon threads, and probably for some embedders.  I'm not sure they care about whether or not Python is finalizing specifically.\r\n\r\nWhat about other users?  I don't know why they might want to know if the runtime is finalizing.  It would certainly only be of interest for daemon threads (and for embedded applications).\r\n\r\n\r\n## What To Do About It?\r\n\r\nIt makes sense to figure out what folks actually care about when it comes to the concept of \"finalizing\".  It would likewise make sense to ensure names and descriptions actually match what functions do (and state is for).\r\n\r\nIn the specific case of `Py_IsFinalizing()`, I see a couple options:\r\n\r\n1. change it to track the moment `Py_Finalize()` starts, before it actually does anything, so \"finalizing\" is 100% accurate\r\n2. change the name to match what it actually tracks: whether or not other threads can count on a stable runtime\r\n\r\n(Likewise for `sys.is_finalizing()`, and most internal API.)\r\n\r\nHere are some tricky things to consider:\r\n\r\n* we want to disable some capabilities as soon as possible during shutdown (e.g. disallow creating new threads *before* waiting for existing non-daemon threads to finish)\r\n* other threads are welcome to keep using the full runtime, even after shutdown begins, as long as we haven't started cleaning up state (which would then cause crashes) [^4]\r\n* in the main thread (executing `Py_Finalize()`), extension modules can (for now?) still run the risk of accessing invalid state *after* the point `Py_IsFinalizing()` returns true, up to the point that the extension module is cleaned up (see `finalize_modules()` in pylifecycle.c) and perhaps even after that\r\n\r\n[^4]: This is the fundamental basis of the original \"finalizing\" status we introduced back in 2011.\r\nThe one tricky thing is that we don't want","author":{"url":"https://github.com/ericsnowcurrently","@type":"Person","name":"ericsnowcurrently"},"datePublished":"2023-10-06T22:35:46.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":3},"url":"https://github.com/110490/cpython/issues/110490"}