René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Add a Check for Extensions that Support Running in Subinterpreters","articleBody":"Any given extension module may or may not support being loaded in multiple interpreters.  This primarily depends on whether or not the extension use any process-global state.  When imported in multiple interpreters at the same time, a module that does not isolate its state may lead to problems.\r\n\r\nGlobal state (which must be isolated) includes:\r\n\r\n* Python objects the extension uses (including internally)\r\n* other data it uses directly\r\n* indirect state, via linked libraries\r\n\r\nIf an extension properly implements multi-phase init (PEP 489) then it probably does support use in multiple interpreters (but not necessarily under per-interpreter GIL).  In fact, PEP 489 proscribes this.\r\n\r\nIt would be useful to add a check to the `ExtensionFileLoader` to this effect.\r\n\r\n## Proposal\r\n\r\n* if an extension is imported in a sub-interpreter and it does not implement multi-phase init then we should raise `ImportError`\r\n* this should be configurable on a per-interpreter basis\r\n* the default is debatable\r\n* we may want to allow multi-phase init extension to opt in to multiple interpreter support\r\n* (there are some additional minor considerations if we factor in a per-interpreter GIL)\r\n\r\n### Configuration\r\n\r\nGiven a `_PyInterpreterConfig` (see gh-98608), we would add something like `_PyInterpreterConfig.check_multi_interp_extensions` to indicate that the interpreter created via `_PyInterpreterNewFromConfig()` should do the check or not.\r\n\r\nFor backward compatibility, we'd use the following defaults:\r\n\r\n* \"false\" for the main interpreter\r\n* **(maybe)** \"false\" for the legacy `Py_NewInterpreter()`\r\n* \"true\" for \"isolated\" interpreters (see the `_xxsubinterpreters` module, PEPs 554, 684)\r\n\r\nFor `Py_NewInterpreter()` it may make sense to default to \"true\", to avoid causing folks headaches (see https://github.com/pyca/cryptography/issues/2299).  However, we'll probably want to stick with \"false\" under the assumption that most extensions work well enough in multiple interpreters to allow us to preserve compatibility.\r\n\r\n## A Module Def Slot\r\n\r\nPer PEP 684 discussions, there is some uncertainty about how confident extension authors may be about how isolated their extensions really are.  If this is a genuine concern then we should consider also adding a PEP 489 module def slot, e.g. `Py_mod_subinterpreters`, to allow extension authors to opt in to supporting multiple interpreters.  It would probably also make sense to offer a slot to opt out of multiple interpreter support, e.g. `Py_mod_no_subinterpreters`, for when support is more wide-spread.\r\n\r\nAgain, this only makes sense if we aren't confident that multi-phase init support is an adequate indicator.\r\n\r\n## Per-Interpreter GIL (PEP 684)\r\n\r\nThere are additional static variables, e.g. temporary buffers, that factor in to extension \"isolation\" if you drop the thread safety provided by the GIL.  This is real concern relative to PEP 684, where we would make the GIL per-interpreter.  The difference between isolation is relatively small, but enough to warrant consideration.\r\n\r\nMost notably, a per-interpreter GIL would strengthen the value of a dedicated module def slot.\n\n\u003c!-- gh-pr-number: gh-99039 --\u003e\n* PR: gh-99039\n\u003c!-- /gh-pr-number --\u003e\n\n\n\u003c!-- gh-pr-number: gh-99040 --\u003e\n* PR: gh-99040\n\u003c!-- /gh-pr-number --\u003e\n","author":{"url":"https://github.com/ericsnowcurrently","@type":"Person","name":"ericsnowcurrently"},"datePublished":"2022-10-24T21:59:52.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":4},"url":"https://github.com/98627/cpython/issues/98627"}