René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"mimalloc: additional integration and changes for `--disable-gil` builds","articleBody":"# Feature or enhancement\r\n\r\nMimalloc was added as an allocator in https://github.com/python/cpython/issues/90815. The `--disable-gil` builds need further integration with mimalloc, as well as some modifications to mimalloc to support thread-safe garbage collection in `--disable-gil` builds and the dictionary accesses that mostly avoid locking.\r\n\r\nThese changes can be split up across multiple PRs.\r\n\r\n* Currently, when mimalloc is enabled, all allocations go to the default heap. This is fine for `PyMem_Malloc` calls, but we need separate heaps for `PyObject_Malloc` and `PyObject_GC_New`. We should associate some `mi_heap_t`s with each `PyThreadState`. Every PyThreadState needs four heaps: one for PyMem_Malloc, one for non-GC objects (via PyObject_Malloc), one for GC objects with managed dicts (extra pre-header) and one for GC objects without a managed dict. We need some way to know which heap to use in `_PyObject_MiMalloc`. There's not a great way to do this, but I suggest adding something like a [\"current pyobject heap\"](https://github.com/colesbury/nogil-3.12/blob/cedde4f5ec3759ad723c89d44738776f362df564/Include/cpython/pystate.h#L127) variable to PyThreadState. It should generally point to the `PyObject_Malloc` heap, but `PyObject_GC_New` should temporarily override it to point to the correct GC heap when called\r\n* `--disable-gil` should imply `--with-mimalloc` and require mimalloc (i.e., disallow changing the allocator with `PYTHONMALLOC`).\r\n* We should tag each mi_heap_t and mi_page_t with a number identifying which type of allocation it's associated with. This is important for when pages are abandoned (i.e., when a thread exits with live blocks remaining) and the page is no longer associated with a heap. The GC still needs to identify which of those pages store GC-enabled objects. (see https://github.com/colesbury/nogil-3.12/commit/d447b6980856df7e0050ecaba4fd6cf21747d4f2)\r\n* When claiming a page from an abandoned segment, mimalloc should associate it with the correct heap from the current thread. In other words, pages that store GC-enabled objects should only be put back in the correct GC heap.\r\n\r\ncc @DinoV \n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-112883\n* gh-113263\n* gh-113492\n* gh-113717\n* gh-113742\n* gh-113995\n* gh-114133\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/colesbury","@type":"Person","name":"colesbury"},"datePublished":"2023-11-29T18:22:21.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":10},"url":"https://github.com/112532/cpython/issues/112532"}