René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Fix Refleak in test_import","articleBody":"Commit 096d0097a09e (for gh-101758) resulted in refleak failures.  It's unclear if the leak is new or if the test simply exposed it.  In 984f8ab018f847fe8d66768af962f69ec0e81849, I skipped the leaking tests until we can fix the leak.\r\n\r\nUPDATE:  I've updates the tests to only skip when checking for refleaks.  I've also added a few more tests.\r\n\r\n----\r\n\r\n### Context\r\n\r\n\u003cdetails\u003e\r\n\u003csummary\u003e(expand)\u003c/summary\u003e\r\n\r\n\u003cBR/\u003e\r\n(Also see gh-101758.)\r\n\u003cBR/\u003e\u003cBR/\u003e\r\n\r\nLoading an extension module involves meaningfully different code paths depending on the content of the `PyModuleDef`.  There's the difference between legacy (single-phase-init, PEP 3121) and modern (multi-phase-init, PEP 489) modules.  For single-phase init, there are those that support being loaded more than once (`m_size` \u003e= 0) and those that can't (`m_size` == -1).  I've added several long comments in import.c explaining about single-phase init modules in detail.  I also added some tests to verify the behavior of the single-phase-init cases more thoroughly.  Those are the leaking tests.\r\n\r\nRelevant state:\r\n\r\n* `PyModuleDef.m_size` - the size of the module's per-interpreter state (`PyModuleObject.md_state`)\r\n   * single-phase init and multi-phase init modules can have a value of 0 or greater\r\n   * such modules must not have any process-global state\r\n   * only single-phase init modules can have a value of -1, which means the module does not support being loaded more than once\r\n   * such modules may have process-global state\r\n* `PyModuleDef.m_base.m_index` - the index into `PyInterpreterState.imports.modules_by_index`  (same index used for every interpreter)\r\n   * set by `PyModuleDef_Init()`  (e.g. when the module is created)\r\n* `PyModuleDef.m_base.m_copy` - a copy of the `__dict__` of the last time a module was *loaded* using this def  (only single-phase init where `m_size` is -1)\r\n   * set exclusively by `fix_up_extension()`\r\n   * used exclusively by `import_find_extension()`\r\n   * cleared by (replaced in) `fix_up_extension()` if `m_copy` was already set  (e.g. multiple interpreters, multiple modules in same file using same def)\r\n   * cleared by `_PyImport_ClearModulesByIndex()` during interpreter finalization\r\n* `_PyRuntime.imports.extensions` - a dict mapping `(filename, name)` to `PyModuleDef`\r\n   * entry set exclusively by `fix_up_extension()`  (only for single-phase init modules)\r\n   * entry used exclusively by `import_find_extension()`\r\n   * entry cleared by `_PyImport_Fini()` at runtime finalization\r\n* `interp-\u003eimports.modules_by_index` - a list of single-phase-init modules loaded in this interpreter; lookups (e.g. `PyState_FindModule()`) use `PyModuleDef.m_base.m_index`\r\n   * entry set normally only by `fix_up_extension()` and `import_find_extension()`  (only single-phase init modules)\r\n   * (entry also set by `PyState_AddModule()`)\r\n   * used exclusively by `PyState_FindModule()`\r\n   * entry cleared normally by `_PyImport_ClearModulesByIndex()` during interpreter finalization\r\n   * (entry also cleared by `PyState_RemoveModule()`)\r\n\r\nCode path when loading a single-phase-init module for the first time (in import.c, unless otherwise noted):\r\n\r\n* `imp.load_dynamic()`\r\n   * `importlib._boostrap._load()`  (using a spec with `ExtensionFileLoader`)\r\n      * `ExtensionFileLoader.create_module()`  (in _boostrap_external.py)\r\n         * `_imp.create_dynamic()`  (`_imp_create_dynamic_impl()`)\r\n            * `import_find_extension()`  (not found in `_PyRuntime.imports.extensions`)\r\n            * `_PyImport_LoadDynamicModuleWithSpec()`  (importdl.c)\r\n               * `_PyImport_FindSharedFuncptr()`\r\n               * `\u003cmodule init func from loaded binary\u003e()`\r\n                  * sets process-global state  (only where `m_size == -1`)\r\n                  * `PyModule_Create()`\r\n                  * populates per-interpreter module state  (only where `m_size \u003e 0`)\r\n                  * sets module attrs  (on `__dict__`)\r\n               * sets `def-\u003em_base.m_init`  (only needed for single-phase-init where `m_size \u003e=0`)\r\n               * `_PyImport_FixupExtensionObject()`\r\n                  * sets `sys.modules[spec.name]`\r\n                  * `fix_up_extension()`\r\n                     * set `interp-\u003eimports.modules_by_index[def-\u003em_base.m_index]`\r\n                     * clear `def-\u003em_base.m_copy`  (only if set and only if `m_size == -1`)\r\n                     * set `def-\u003em_base.m_copy` to a copy of the module's `__dict__`  (only if `m_size == -1`)\r\n                     * set `_PyRuntime.imports.extensions[(filename, name)]`\r\n      * sets missing module attrs (e.g. `__file__`)\r\n\r\nDuring testing we use a helper to erase (nearly) any evidence of the module having been imported before.  That means clearing the state described above.\r\n\r\nHere's the code path:\r\n\r\n* `_testinternalcapi.clear_extension()`  (Modules/_testinternalcapi.c)\r\n   * `_PyImport_ClearExtension()`\r\n      * `clear_singlephase_extension()`\r\n         * (only if there's a `_PyRuntime.imports.extensions` entry)\r\n            * clear the module def's `m_copy`\r\n            * replace the `interp-\u003eimports.modules_by_index` entry with `Py_None`\r\n            * delete the `_PyRuntime.imports.extensions` entry\r\n\r\n\u003c/details\u003e\r\n\r\n\u003c!-- gh-linked-prs --\u003e\r\n### Linked PRs\r\n* gh-102254\r\n* gh-104796\n* gh-105082\n* gh-105083\n* gh-105085\n* gh-105170\n* gh-106013\n* gh-109540\n\u003c!-- /gh-linked-prs --\u003e\r\n","author":{"url":"https://github.com/ericsnowcurrently","@type":"Person","name":"ericsnowcurrently"},"datePublished":"2023-02-25T18:19:13.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":9},"url":"https://github.com/102251/cpython/issues/102251"}