René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Isolate the tracemalloc Module Between Interpreters","articleBody":"(See https://github.com/python/cpython/issues/100227.)\r\n\r\nCurrently the tracemalloc module has some state in `_PyRuntimeState`, including objects, which is shared by all interpreters.  Interpreters should be isolated from each other, for a variety of reasons (including the possibility of a per-interpreter GIL).  Isolating the module will involve moving some of the state to `PyInterpreterState` (and, under per-interpreter GIL, guarding other state with a global lock).\r\n\r\n----\r\n\r\nAnalysis:\r\n\r\n\u003cdetails\u003e\r\n\u003csummary\u003e(expand)\u003c/summary\u003e\r\n\r\n## Allocators\r\n\r\nThe module installs a custom allocator (using the PEP 445 API\r\n ([docs](https://docs.python.org/3.12/c-api/memory.html#customize-memory-allocators)).\r\n\r\n\u003cdetails\u003e\r\n\u003csummary\u003e(expand)\u003c/summary\u003e\r\n\r\n\u003cBR/\u003e\r\n\r\nthe allocator functions:\r\n\r\n| domain | func | wraps | wraps (reentrant) | actually wraps |\r\n| ---- | ---- | ---- | ---- | ---- |\r\n| \u003cp align=\"center\"\u003e-\u003c/p\u003e | `tracemalloc_alloc()` | \\\u003coriginal `malloc()` or `calloc()`\\\u003e | \\\u003c--- |  |\r\n| \u003cp align=\"center\"\u003e-\u003c/p\u003e | `tracemalloc_realloc()` | \\\u003coriginal `realloc()` or `free()`\\\u003e | \\\u003c--- |  |\r\n| \u003cp align=\"center\"\u003e-\u003c/p\u003e | `tracemalloc_raw_alloc()` | `tracemalloc_alloc()` **\\*** | \\\u003coriginal `malloc()` or `calloc()`\\\u003e |  |\r\n| \u003cp align=\"center\"\u003e-\u003c/p\u003e | `tracemalloc_alloc_gil()` | `tracemalloc_alloc()` | \\\u003coriginal `malloc()` or `calloc()`\\\u003e |  |\r\n| raw |  |  |  |  |\r\n|  | `tracemalloc_raw_malloc()` | `tracemalloc_alloc()` | \u003coriginal `malloc()`\u003e | `tracemalloc_raw_alloc()` |\r\n|  | `tracemalloc_raw_calloc()` | `tracemalloc_alloc()` | \u003coriginal `calloc()`\u003e | `tracemalloc_raw_alloc()` |\r\n|  | `tracemalloc_raw_realloc()` | `tracemalloc_realloc()` **\\*** | \\\u003coriginal `realloc()`\u003e |  |\r\n|  | `tracemalloc_free()` | \u003coriginal `free()`\u003e | \\\u003c--- |  |\r\n| mem |  |  |  |  |\r\n| obj |  |  |  |  |\r\n|  | `tracemalloc_malloc_gil()` | `tracemalloc_alloc_gil()` | \\\u003c--- |  |\r\n|  | `tracemalloc_calloc_gil()` | `tracemalloc_alloc_gil()` | \\\u003c--- |  |\r\n|  | `tracemalloc_realloc_gil()` | `tracemalloc_realloc()` | \\\u003coriginal `realloc()`\\\u003e |  |\r\n|  | `tracemalloc_free()` | \u003coriginal `free()`\u003e | \\\u003c--- |  |\r\n\r\n**\\*** Note that `tracemalloc_raw_alloc()` wraps the `tracemalloc_alloc()` call\r\nwith `PyGILState_Ensure()`/`PyGILState_Release()`.\r\nLikewise for `tracemalloc_raw_realloc()` where it calls `tracemalloc_realloc()`.\r\nIn no other case does an allocator function use the GILState API for any calls.\r\n\r\n\u003c/details\u003e\r\n\r\n\r\n## State\r\n\r\n### Fields\r\n\r\n\u003cdetails\u003e\r\n\u003csummary\u003e(expand)\u003c/summary\u003e\r\n\r\n\u003cBR/\u003e\r\n\r\nhttps://github.com/python/cpython/blob/main/Include/internal/pycore_tracemalloc.h#L57-L107\r\nhttps://github.com/python/cpython/blob/main/Include/internal/pycore_runtime.h#L147\r\n\r\n\u003cdetails\u003e\r\n\u003csummary\u003eraw\u003c/summary\u003e\r\n\r\n\u003cBR/\u003e\r\n\r\nhttps://github.com/python/cpython/blob/0675b8f032c69d265468b31d5cadac6a7ce4bd9c/Include/internal/pycore_tracemalloc.h#L43-L55\r\nhttps://github.com/python/cpython/blob/0675b8f032c69d265468b31d5cadac6a7ce4bd9c/Include/internal/pycore_tracemalloc.h#L57-L64\r\nhttps://github.com/python/cpython/blob/0675b8f032c69d265468b31d5cadac6a7ce4bd9c/Include/internal/pycore_tracemalloc.h#L57-L107\r\n\r\nhttps://github.com/python/cpython/blob/0675b8f032c69d265468b31d5cadac6a7ce4bd9c/Modules/_tracemalloc.c#L54-L55\r\nhttps://github.com/python/cpython/blob/0675b8f032c69d265468b31d5cadac6a7ce4bd9c/Modules/_tracemalloc.c#L69-L76\r\n\r\n\u003c/details\u003e\r\n\r\n| name | type | protected by | \\#ifdef | notes |\r\n| ---- | ---- | ---- | ---- | ---- |\r\n| `config` | `struct _PyTraceMalloc_Config` |  |  |  |\r\n| \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;. `initialized` | `enum {}` | GIL |  |  |\r\n| \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;. `tracing` | `bool` | GIL |  |  |\r\n| \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;. `max_nframe` | `int` | GIL |  |  |\r\n| `allocators` |  |  |  | see PEP 445 ([docs](https://docs.python.org/3.12/c-api/memory.html#customize-memory-allocators)) |\r\n| \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;. `mem` | `PyMemAllocatorEx` | GIL |  |  |\r\n| \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;. `raw` | `PyMemAllocatorEx` | GIL |  |  |\r\n| \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;. `obj` | `PyMemAllocatorEx` | GIL |  |  |\r\n| `tables_lock` | `PyThread_type_lock` | GIL | `TRACE_RAW_MALLOC` |  |\r\n| `traced_memory` | `size_t` | `tables_lock` |  |  |\r\n| `peak_traced_memory` | `size_t` | `tables_lock` |  |  |\r\n| `filenames` | `_Py_hashtable_t *` | GIL |  | interned; effectively a `set` of objects |\r\n| `traceback` | `struct tracemalloc_traceback *` | GIL |  | a temporary buffer |\r\n| `tracebacks` | `_Py_hashtable_t *` | GIL |  | interned; effectively a `set` of `traceback_t` |\r\n| `traces` | `_Py_hashtable_t *` | `tables_lock` |  | void-ptr -\u003e `trace_t`  |\r\n| `domains` | `_Py_hashtable_t *` |`tables_lock`  |  | domain -\u003e `_Py_hashtable_t *` (per-domain `traces`) |\r\n| `empty_traceback` | `struct tracemalloc_traceback` | ??? |  |  |\r\n| `reentrant_key` | `Py_tss_t` | ??? |  |  |\r\n\r\nnotes:\r\n* each frame in `struct tracemalloc_traceback` holds a filename object\r\n* `traceback_t` is a typedef for `struct tracemalloc_traceback`\r\n* `frame_t` is a typedef for `struct tracemalloc_frame`\r\n\r\nhold objects:\r\n\r\n* `filenames`\r\n* `traceback` (filename in each frame)\r\n* `tracebacks` (filename in each frame of each traceback)\r\n* `traces` (filename in each frame of each traceback)\r\n* `domains` (filename in each frame of each traceback in each domain)\r\n\r\n\u003c/details\u003e\r\n\r\n\r\n### Usage\r\n\r\n\u003cdetails\u003e\r\n\u003csummary\u003e(expand)\u003c/summary\u003e\r\n\r\n\u003cBR/\u003e\r\n\r\nsimple:\r\n\r\n| name | context | get | set |\r\n| ---- | ---- | ---- | ---- |\r\n| `config` |  |  |  |\r\n| \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;. `initialized` | lifecycle | `tracemalloc_init()`\u003cBR/\u003e`tracemalloc_deinit()` | `tracemalloc_init()`\u003cBR/\u003e`tracemalloc_deinit()` |\r\n| \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;. `tracing` | module | `_tracemalloc_is_tracing_impl()`\u003cBR/\u003e`_tracemalloc__get_traces_impl()`\u003cBR/\u003e`_tracemalloc_clear_traces_impl()`\u003cBR/\u003e`_tracemalloc_get_traceback_limit_impl()`\u003cBR/\u003e`_tracemalloc_get_traced_memory_impl()`\u003cBR/\u003e`_tracemalloc_reset_peak_impl()` |  |\r\n|  | C-API | `PyTraceMalloc_Track()`\u003cBR/\u003e`PyTraceMalloc_Untrack()`\u003cBR/\u003e`_PyTraceMalloc_NewReference()`\u003cBR/\u003e`_PyMem_DumpTraceback()` |  |\r\n|  | lifecycle | `tracemalloc_start()`\u003cBR/\u003e`tracemalloc_stop()` | `tracemalloc_start()`\u003cBR/\u003e`tracemalloc_stop()` |\r\n|  | internal | `tracemalloc_get_traceback()` |  |\r\n| \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;. `max_nframe` | module | `_tracemalloc_get_traceback_limit_impl()` |  |\r\n|  | lifecycle |  | `tracemalloc_start()` |\r\n|  | internal | `traceback_get_frames()` |  |\r\n| `allocators` |  |  |  |  |\r\n| \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;. `mem` | lifecycle | `tracemalloc_start()` \\+\u003cBR/\u003e`tracemalloc_stop()` |  |\r\n| \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;. `raw` | lifecycle | `tracemalloc_init()` \\+\u003cBR/\u003e`tracemalloc_start()` \\+\u003cBR/\u003e`tracemalloc_stop()` |  |\r\n|  | internal | `raw_malloc()`\u003cBR/\u003e `raw_free()` |  |\r\n| \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;. `obj` | lifecycle | `tracemalloc_start()` \\+\u003cBR/\u003e`tracemalloc_stop()` |  |\r\n| `tables_lock` | module | `_tracemalloc__get_traces_impl()` \\+\u003cBR/\u003e`_tracemalloc_get_tracemalloc_memory_impl()` \\+\u003cBR/\u003e`_tracemalloc_get_traced_memory_impl()` \\+\u003cBR/\u003e`_tracemalloc_reset_peak_impl()` \\+ |  |\r\n|  | C-API | `PyTraceMalloc_Track()` \\+\u003cBR/\u003e`PyTraceMalloc_Untrack()` \\+\u003cBR/\u003e`_PyTraceMalloc_NewReference()` \\+ |  |\r\n|  | lifecycle | `tracemalloc_init()`\u003cBR/\u003e`tracemalloc_deinit()` | `tracemalloc_init()` \\*\u003cBR/\u003e`tracemalloc_deinit()` \\* |\r\n|  | allocator | `tracemalloc_alloc()` \\+\u003cBR/\u003e`tracemalloc_realloc()` \\+\u003cBR/\u003e`tracemalloc_free()` \\+\u003cBR/\u003e`tracemalloc_realloc_gil()` \\+\u003cBR/\u003e`tracemalloc_raw_realloc()` \\+ |  |\r\n|  | internal | `tracemalloc_get_traceback()` \\+\u003cBR/\u003e`tracemalloc_clear_traces()` \\+ |  |\r\n| `traced_memory` | module | `_tracemalloc_get_traced_memory_impl()`\u003cBR/\u003e`_tracemalloc_reset_peak_impl()` |  |\r\n|  | internal | `tracemalloc_add_trace()` | `tracemalloc_add_trace()`\u003cBR/\u003e`tracemalloc_remove_trace()`\u003cBR/\u003e`tracemalloc_clear_traces()` |\r\n| `peak_traced_memory` | module | `_tracemalloc_get_traced_memory_impl()` | `_tracemalloc_reset_peak_impl()` |\r\n|  | internal | `tracemalloc_add_trace()` | `tracemalloc_add_trace()`\u003cBR/\u003e`tracemalloc_clear_traces()` |\r\n| `filenames` | module | `_tracemalloc_get_tracemalloc_memory_impl()` |  |\r\n|  | lifecycle |  | `tracemalloc_init()` \\* |\r\n|  | internal | `tracemalloc_get_frame()`\u003cBR/\u003e`tracemalloc_clear_traces()` \\+ |  |\r\n| `traceback` | lifecycle | `tracemalloc_stop()` | `tracemalloc_start()` \\*\u003cBR/\u003e`tracemalloc_stop()` \\* |\r\n|  | internal | `traceback_new()` \\+ |  |\r\n| `tracebacks` | module | `_tracemalloc_get_tracemalloc_memory_impl()` |  |\r\n|  | lifecycle |  | `tracemalloc_init()` \\*\u003cBR/\u003e`tracemalloc_deinit()` \\+ |\r\n|  | internal | `traceback_new()` \\+\u003cBR/\u003e`tracemalloc_clear_traces()` \\+ |  |\r\n| `traces` | module | `_tracemalloc__get_traces_impl()`\u003cBR/\u003e`_tracemalloc_get_tracemalloc_memory_impl()` |  |\r\n|  | C-API | `_PyTraceMalloc_NewReference()` |  |\r\n|  | lifecycle | `tracemalloc_deinit()` \\+ | `tracemalloc_init()` \\* |\r\n|  | internal | `tracemalloc_get_traces_table()`\u003cBR/\u003e`tracemalloc_add_trace()` (indirect)\u003cBR/\u003e`tracemalloc_remove_trace()` (indirect)\u003cBR/\u003e`tracemalloc_get_traceback()` (indirect)\u003cBR/\u003e`tracemalloc_clear_traces()` \\+ |  |\r\n| `domains` | module | `_tracemalloc__get_traces_impl()`\u003cBR/\u003e`_tracemalloc_get_tracemalloc_memory_impl()` |  |\r\n|  | lifecycle | `tracemalloc_deinit()` \\+ | `tracemalloc_init()` \\* |\r\n|  | internal | `tracemalloc_get_traces_table()`\u003cBR/\u003e`tracemalloc_remove_trace()` (indirect)\u003cBR/\u003e`tracemalloc_get_traceback()` (indirect)\u003cBR/\u003e`tracemalloc_clear_traces()` \\+\u003cBR/\u003e`tracemalloc_add_trace()` \\+ |\r\n| `empty_traceback` | lifecycle | `tracemalloc_init()` \\+ |  |\r\n|  | internal | `traceback_new()` |  |\r\n| `reentrant_key` | lifecycle | `tracemalloc_init()` \\+\u003cBR/\u003e`tracemalloc_deinit()` \\+ |  |\r\n|  | allocator | `tracemalloc_alloc_gil()` (indirect) \\+\u003cBR/\u003e`tracemalloc_realloc_gil()` (indirect) \\+\u003cBR/\u003e`tracemalloc_raw_alloc()` (indirect) \\+\u003cBR/\u003e`tracemalloc_raw_realloc()` (indirect) \\+ |  |\r\n|  | internal | `get_reentrant()`\u003cBR/\u003e`set_reentrant()` \\+ |  |\r\n\r\n\\* the function allocates/deallocates the value (see below)\r\n\\+ the function mutates the value (see below)\r\n\r\nsimple (extraneous):\r\n\r\n| name | context | allocate/deallocate | get (assert-only) |\r\n| ---- | ---- | ---- | ---- |\r\n| `config.tracing` | internal |  | `tracemalloc_add_trace()`\u003cBR/\u003e`tracemalloc_remove_trace()` |\r\n| `tables_lock` | lifecycle | `tracemalloc_init()`\u003cBR/\u003e`tracemalloc_deinit()` |  |\r\n| `traced_memory` | internal |  | `tracemalloc_remove_trace()` |\r\n| `filenames` | lifecycle | `tracemalloc_init()` |  |\r\n| `traceback` | lifecycle | `tracemalloc_start()`\u003cBR/\u003e`tracemalloc_stop()` | `tracemalloc_start()` |\r\n| `tracebacks` | lifecycle | `tracemalloc_init()` |  |\r\n| `traces` | lifecycle | `tracemalloc_init()` |  |\r\n| `domains` | lifecycle | `tracemalloc_init()` |  |\r\n\r\nmutation of complex fields:\r\n\r\n| name | context | initialize | finalize | clear | modify |\r\n| ---- | ---- | ---- | ---- | ---- | ---- |\r\n| `allocators.mem` | lifecycle | `tracemalloc_start()` |  |  |  |\r\n| `allocators.raw` | lifecycle | `tracemalloc_init()`\u003cBR/\u003e`tracemalloc_start()` |  |  |  |\r\n| `allocators.obj` | lifecycle | `tracemalloc_start()` |  |  |  |\r\n| `tables_lock` | module |  |  |  | `_tracemalloc__get_traces_impl()`\u003cBR/\u003e`_tracemalloc_get_tracemalloc_memory_impl()`\u003cBR/\u003e`_tracemalloc_get_traced_memory_impl()`\u003cBR/\u003e`_tracemalloc_reset_peak_impl()` |\r\n|  | C-API |  |  |  | `PyTraceMalloc_Track()`\u003cBR/\u003e`PyTraceMalloc_Untrack()`\u003cBR/\u003e`_PyTraceMalloc_NewReference()` |\r\n|  | allocator | `tracemalloc_alloc()` |  |  | `tracemalloc_alloc()`\u003cBR/\u003e`tracemalloc_realloc()`\u003cBR/\u003e`tracemalloc_free()`\u003cBR/\u003e`tracemalloc_realloc_gil()`\u003cBR/\u003e`tracemalloc_raw_realloc()` |\r\n|  | internal |  |  |  | `tracemalloc_clear_traces()`\u003cBR/\u003e`tracemalloc_get_traceback()` |\r\n| `filenames` | internal |  |  | `tracemalloc_clear_traces()` | `tracemalloc_get_frame()`\u003cBR/\u003e`tracemalloc_clear_traces()` |\r\n|  | lifecycle |  | `tracemalloc_deinit()` |  |  |\r\n| `traceback` | internal | `traceback_new()` |  |  |  |\r\n| `tracebacks` | lifecycle |  | `tracemalloc_deinit()` |  |  |\r\n|  | internal |  |  | `tracemalloc_clear_traces()` | `traceback_new()` |\r\n| `traces` | lifecycle |  | `tracemalloc_deinit()` |  |  |\r\n|  | internal |  |  | `tracemalloc_clear_traces()` |  |\r\n| `domains` | lifecycle|  | `tracemalloc_deinit()` |  |  |\r\n|  | internal |  |  | `tracemalloc_clear_traces()` | `tracemalloc_add_trace()` |\r\n| `reentrant_key` | lifecycle | `tracemalloc_init()` | `tracemalloc_deinit()` |  |  |\r\n|  | internal |  |  |  | `set_reentrant()` |\r\n\r\nindirection:\r\n\r\n| name | context | direct | indirect |\r\n| ---- | ---- | ---- | ---- |\r\n| `allocators.raw` | lifecycle | `tracemalloc_start()`\u003cBR/\u003e`tracemalloc_copy_trace()` | `raw_malloc()` |\r\n|  |  | `tracemalloc_stop()` | `raw_free()` |\r\n|  | internal | `traceback_new()`\u003cBR/\u003e`tracemalloc_add_trace()`\u003cBR/\u003e`tracemalloc_copy_trace()` | `raw_malloc()` |\r\n|  |  | `traceback_new()`\u003cBR/\u003e`tracemalloc_add_trace()`\u003cBR/\u003e`tracemalloc_remove_trace()` | `raw_free()` |\r\n| `traces` | internal | `tracemalloc_add_trace()`\u003cBR/\u003e`tracemalloc_remove_trace()`\u003cBR/\u003e`tracemalloc_get_traceback()` | `tracemalloc_get_traces_table()` |\r\n| `domains` | internal | `tracemalloc_add_trace()`\u003cBR/\u003e`tracemalloc_remove_trace()`\u003cBR/\u003e`tracemalloc_get_traceback()` | `tracemalloc_get_traces_table()` |\r\n| `reentrant_key` | allocator | `tracemalloc_alloc_gil()`\u003cBR/\u003e`tracemalloc_realloc_gil()`\u003cBR/\u003e`tracemalloc_raw_alloc()`\u003cBR/\u003e`tracemalloc_raw_realloc()` | `get_reentrant()` |\r\n|  |  | `tracemalloc_alloc_gil()`\u003cBR/\u003e`tracemalloc_realloc_gil()`\u003cBR/\u003e`tracemalloc_raw_alloc()`\u003cBR/\u003e`tracemalloc_raw_realloc()` | `set_reentrant()` |\r\n\r\n\u003c/details\u003e\r\n\r\n\u003c/details\u003e\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-104508\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/ericsnowcurrently","@type":"Person","name":"ericsnowcurrently"},"datePublished":"2023-02-02T21:41:41.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/101520/cpython/issues/101520"}