René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Extension Module Handles Are Never Closed","articleBody":"We load extension modules by loading the shared library file (`dlopen()`, `LoadLibraryExW()`).  However we never unload/release it.  In each case we get a handle when load it, but we throw the handle away.  We should consider holding on to the handle (maybe in something like `PyModuleObject.md_handle`).\r\n\r\nIf we save the handle, we could use it to release the loaded shared library (`dlclose()`, `FreeLibrary()`).  `module_dealloc()` would probably be a good place to do so.  In every case. releasing it drops an internal refcount, and when it reaches 0 the lib is actually unloaded.\r\n\r\nAside from not leaking, there's another real benefit.  Extension modules can (and often do, e.g. static types) store objects in C global variables.  Single-phase init modules do not have an opportunity to clean up after themselves (which was part of the motivation for PEP 489).  So they don't have a chance to decref any objects they may have stashed away in globals.  This is problematic when such stashed objects survive runtime init/fini cycles. [^1]  Closing the lib means all the extensions C globals are discarded and thus importing the module in a later cycle will result in a clean set of globals.  That may mean a few objects get leaked, but that's better than segfaulting.\r\n\r\n[^1]: The same goes for when an extension is loaded for the first time in an isolated (per-interpreter GIL) subinterpreter rather than in the main interpreter, since the C globals will outlive the interpreter.  However, there are some caveats with that too.  Also, though single-phase init modules are not supported in isolated interpreters, their init functions still get run.\r\n\r\nThere are some complexities though.  Using `dlclose()`, etc. must be done carefully.  While not critical, it's best to do it in the thread where the \"open\" was done.  Also, once the lib is closed, trying to use any variables or functions defined in it will segfault.  That means you can't do it if there's any thread running C code from the module (incl. daemon threads).  Likewise if the module registered a callback somewhere or if a reference to one of its static types (or other global objects) is held outside the module.  That may even impact GC.\r\n\r\nWe might consider at least closing the lib if `_PyImport_CheckSubinterpIncompatibleExtensionAllowed()` fails in `_PyImport_LoadDynamicModuleWithSpec()`.  The risk there is smaller.\r\n\r\nFWIW, in practice this isn't a big concern because a loaded extension module typically doesn't get destroyed until not long before the process terminates (though it might be a different story for embedders).  I'm bringing it up because it something I noticed and looked into and thought we might at least consider the problem.","author":{"url":"https://github.com/ericsnowcurrently","@type":"Person","name":"ericsnowcurrently"},"datePublished":"2024-01-24T22:11:39.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":8},"url":"https://github.com/114538/cpython/issues/114538"}