René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"sub-interpreters: significant memory leak on shutdown","articleBody":"# Bug report\r\n\r\n### Bug description:\r\n\r\nHi all, congrats on the new Python release, very keen on the direction that sub-interpreters are taking!\r\n\r\nI noticed that constructing and destructing sub-interpreters now causes memory to be leaked. This was not a problem in 3.11, but is as of 3.12 and occurs on latest main branch too (as of 8c071373f12f325c54591fe990ec026184e48f8f). I figure it must relate to the changes made to split up the interpreter states.\r\n\r\nI can consistently replicate the issue using simply:\r\n```python\r\nimport _xxsubinterpreters\r\nwhile 1:\r\n    interp = _xxsubinterpreters.create()\r\n    _xxsubinterpreters.destroy(interp)\r\n```\r\nYou'll notice the memory usage climb quite quickly if you look at top/htop.\r\n\r\nvalgrind/massif think the culprit are interned strings, specifically here:\r\n```\r\n==219777==    at 0x4848A13: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)\r\n==219777==    by 0x2DA59E: PyMem_RawCalloc (obmalloc.c:671)\r\n==219777==    by 0x2DA59E: arena_map_get (obmalloc.c:1021)\r\n==219777==    by 0x2DA59E: arena_map_mark_used (obmalloc.c:1076)\r\n==219777==    by 0x2DAA71: new_arena (obmalloc.c:1209)\r\n==219777==    by 0x2DAA71: allocate_from_new_pool (obmalloc.c:1389)\r\n==219777==    by 0x2DB00B: pymalloc_alloc (obmalloc.c:1553)\r\n==219777==    by 0x2DB00B: _PyObject_Malloc (obmalloc.c:1564)\r\n==219777==    by 0x424AF5: gc_alloc (gcmodule.c:2304)\r\n==219777==    by 0x424AF5: _PyObject_GC_New (gcmodule.c:2319)\r\n==219777==    by 0x2BDF3B: new_dict (dictobject.c:748)\r\n==219777==    by 0x2BDF3B: PyDict_New (dictobject.c:851)\r\n==219777==    by 0x34CF06: init_interned_dict (unicodeobject.c:248)\r\n==219777==    by 0x34CF06: _PyUnicode_InitGlobalObjects (unicodeobject.c:14670)\r\n==219777==    by 0x3EDE18: pycore_init_global_objects (pylifecycle.c:678)\r\n==219777==    by 0x3EDE18: pycore_interp_init (pylifecycle.c:826)\r\n==219777==    by 0x3F07A9: new_interpreter (pylifecycle.c:2079)\r\n==219777==    by 0x3F07A9: Py_NewInterpreterFromConfig (pylifecycle.c:2114)\r\n==219777==    by 0x4867A52: interp_create (_xxsubinterpretersmodule.c:516)\r\n==219777==    by 0x2D0C22: cfunction_call (methodobject.c:537)\r\n==219777==    by 0x2716A7: _PyObject_MakeTpCall (call.c:240)\r\n```\r\n\r\nI had a look around that code but I can't spot any obvious bug. I did verify that the interned strings dict created and destroyed in `clear_interned_dict()` is the same for each sub-interpreter, just in case.\r\n\r\nI think every interned string is now also automatically immortal. Does that mean it wouldn't be cleaned up even if the refcount goes down to zero?\r\n\r\n### CPython versions tested on:\r\n\r\n3.11, 3.12, CPython main branch\r\n\r\n### Operating systems tested on:\r\n\r\nLinux","author":{"url":"https://github.com/costasgambit","@type":"Person","name":"costasgambit"},"datePublished":"2023-10-05T17:09:40.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":8},"url":"https://github.com/110411/cpython/issues/110411"}