Title: Free-threaded 3.14.10 and 3.15.2 keep memory freed by other threads: 10 GB peaks since the mimalloc 2.5.2 update · Issue #161420 · python/cpython · GitHub
Open Graph Title: Free-threaded 3.14.10 and 3.15.2 keep memory freed by other threads: 10 GB peaks since the mimalloc 2.5.2 update · Issue #161420 · python/cpython
X Title: Free-threaded 3.14.10 and 3.15.2 keep memory freed by other threads: 10 GB peaks since the mimalloc 2.5.2 update · Issue #161420 · python/cpython
Description: Bug report I've been looking into the performance of streaming decompression with compression.zstd (archive extraction in a package manager), first on 3.14 and 3.15, then on the free-threaded builds. On 3.14.10t, extracting an archive wi...
Open Graph Description: Bug report I've been looking into the performance of streaming decompression with compression.zstd (archive extraction in a package manager), first on 3.14 and 3.15, then on the free-threaded build...
X Description: Bug report I've been looking into the performance of streaming decompression with compression.zstd (archive extraction in a package manager), first on 3.14 and 3.15, then on the free-threaded b...
Opengraph URL: https://github.com/python/cpython/issues/161420
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Free-threaded 3.14.10 and 3.15.2 keep memory freed by other threads: 10 GB peaks since the mimalloc 2.5.2 update","articleBody":"### Bug report\n\nI've been looking into the performance of streaming decompression with `compression.zstd` (archive extraction in a package manager), first on 3.14 and 3.15, then on the free-threaded builds. On 3.14.10t, extracting an archive with 1.1 GiB of files peaked at 1.3 GB, against 170 MB on 3.14.\n\nThe extraction decompresses on worker threads and hands the decompressed `bytes` to one thread that writes them out. Boiled down: worker threads make large `bytes` objects, the main thread consumes and drops them, with a bounded number in flight. That needs about 150 MiB at any time, but on 3.14.10t, 3.15.2t and current `main` the peak grows to 9–10 GB:\n\n```python\nimport resource\nimport sys\nfrom collections import deque\nfrom concurrent.futures import ThreadPoolExecutor\n\nBLOCK, COUNT, THREADS = 16 \u003c\u003c 20, 1536, 8 # 24 GiB in total\npattern = bytes(range(256)) * (BLOCK // 256)\n\ndef produce(i):\n return memoryview(pattern).tobytes() # a new bytes object, as decompress() returns\n\nwith ThreadPoolExecutor(THREADS) as pool:\n pending = deque()\n for i in range(COUNT):\n pending.append(pool.submit(produce, i))\n if len(pending) >= THREADS:\n pending.popleft().result() # freed here, on the main thread\n for future in pending:\n future.result()\n\npeak = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss\npeak = peak if sys.platform == \"darwin\" else peak * 1024\nprint(f\"{sys.version.split()[0]}{'' if sys._is_gil_enabled() else 't'}: peak RSS {peak >> 20} MiB\")\n```\n\nmacOS 26 (arm64), python-build-standalone builds, and `main` built from source:\n\n| Python | vendored mimalloc | peak RSS |\n|---|---|---:|\n| 3.14.10 (default allocator) | — | 167 MiB |\n| 3.13.18t | 2.1.2 | 392 MiB |\n| 3.14.10t | 2.5.2 | **9,933 MiB** |\n| 3.15.2t | 2.5.2 | **10,266 MiB** |\n| 3.15.2, `PYTHONMALLOC=mimalloc` | 2.5.2 | **10,453 MiB** |\n| 3.16.0a1t, tagged before the update | 2.1.2 | 430 MiB |\n| `main` (aaaf61c), free-threaded | 2.5.2 | **8,994 MiB** |\n\nLinux (aarch64) and Windows (x64) show the same: 9.2–10.5 GiB on 3.14.10t and 3.15.2t. Builds with the GIL are affected under `PYTHONMALLOC=mimalloc`; free-threaded builds always use mimalloc. The memory is not leaked: the peak levels off (see below).\n\n### Bisected to the mimalloc 2.5.2 update\n\n3.14.10 and 3.15.2 carry the vendored mimalloc 2.5.2 from gh-159111 and its backports; 3.13.18 and 3.16.0a1 still have 2.1.2. Free-threaded builds of `main` with the script above, macOS:\n\n| commit | mimalloc | peak RSS |\n|---|---|---:|\n| ab5187c, before gh-159111 | 2.1.2 | 443 MiB |\n| aff7a2e, gh-159111 | 2.5.2 | 9,477 MiB |\n| aaaf61c, `main` on 2026-11-05 | 2.5.2 | 8,994 MiB |\n| aaaf61c, with option 1 below | 2.5.2 | 426 MiB |\n\n### Cause\n\n`_mi_malloc_generic()` in `Objects/mimalloc/page.c` used to process the heap's delayed frees (blocks other threads freed back to it) on every call. Since 2.5.2 it does so only every 100 calls:\n\n```c\n // do administrative tasks every N generic mallocs\n if mi_unlikely(++heap->generic_count >= 100) {\n ...\n // free delayed frees from other threads (but skip contended ones)\n _mi_heap_delayed_free_partial(heap);\n```\n\nEvery block over `MI_MEDIUM_OBJ_SIZE_MAX` (64 KiB in 2.5.2) gets a page of its own, and 2.5.2 moves such a page to the full queue right away. A free of that block from another thread goes onto the owning heap's delayed-free list, and allocating such a block always takes this slow path. So each allocating thread keeps up to about 100 freed blocks out of reach. The peak follows threads × 100 × block size (free-threaded `main` at aff7a2e, macOS, 24 GiB in total):\n\n| threads × block | predicted | measured |\n|---|---:|---:|\n| 2 × 8 MiB | 1,600 MiB | 1,673 MiB |\n| 4 × 8 MiB | 3,200 MiB | 3,218 MiB |\n| 8 × 8 MiB | 6,400 MiB | 6,434 MiB |\n| 4 × 16 MiB | 6,400 MiB | 6,269 MiB |\n\nThat also explains why `MIMALLOC_GENERIC_COLLECT=1` doesn't help: it's only read inside the same branch. `MIMALLOC_PURGE_DELAY=0` doesn't either: the blocks aren't freed until that list is processed, so there is nothing to purge. `MIMALLOC_TARGET_SEGMENTS_PER_THREAD=2` works around it (4 GiB down to about 600 MiB).\n\nThe batching came with upstream commit [58d13f6a](https://github.com/microsoft/mimalloc/commit/58d13f6a4fa7390e198a01dd4cc4a935724eb348) (\"collect every N generic calls\", in mimalloc 2.2.2), which added the periodic heap collection (`generic_collect`) and moved the per-call housekeeping into the same every-100 block. That works when generic mallocs are frequent, as for small objects, but every allocation over 64 KiB is one, and 100 of those can be gigabytes. Upstream's `main` and `dev` branches still have it, and microsoft/mimalloc#1117 reports the same symptom (large blocks freed by other threads that aren't collected), without a cause so far; I'll add this there.\n\n### Possible fixes\n\nI'm not sure what the right fix is. The every-100 batching probably exists for a good reason: it amortizes the housekeeping over workloads with many small allocations, which I haven't measured beyond a quick benchmark. Two options came to mind, and I'm happy to help with either:\n\n1. **A small, targeted revert** for `main`, 3.14 and 3.15: process the delayed frees on every generic malloc again, as 2.1.2 did, and keep the deferred-free callback and the periodic collection batched. It costs an atomic load per generic malloc when the list is empty, which I couldn't measure. It would be one more CPython-only change to the vendored mimalloc (gh-113141). Patch and measurements below.\n2. **A better strategy** that keeps the batching for small objects but bounds what large blocks can hold back, for example a byte budget next to the call count (housekeeping every 100 generic mallocs or every few MiB allocated through the generic path). That needs larger changes, a new heap field and a tuning constant, and is probably better discussed with upstream mimalloc, where it would apply too.\n\n\u003cdetails>\n\u003csummary>Option 1, against \u003ccode>main\u003c/code>\u003c/summary>\n\n```diff\ndiff --git a/Objects/mimalloc/page.c b/Objects/mimalloc/page.c\n--- a/Objects/mimalloc/page.c\n+++ b/Objects/mimalloc/page.c\n@@ -1071,6 +1071,12 @@ void* _mi_malloc_generic(mi_heap_t* heap, size_t size, bool zero, size_t huge_al\n }\n mi_assert_internal(mi_heap_is_initialized(heap));\n \n+ // free delayed frees from other threads (but skip contended ones)\n+ // CPython: on every generic malloc, as before mimalloc 2.5: a block of a singleton page (any\n+ // block over MI_MEDIUM_OBJ_SIZE_MAX) freed by another thread is a delayed free, and up to 100\n+ // generic mallocs of such blocks per heap kept that many freed blocks out of reach.\n+ _mi_heap_delayed_free_partial(heap);\n+\n // do administrative tasks every N generic mallocs\n if mi_unlikely(++heap->generic_count >= 100) {\n heap->generic_collect_count += heap->generic_count;\n@@ -1078,9 +1084,6 @@ void* _mi_malloc_generic(mi_heap_t* heap, size_t size, bool zero, size_t huge_al\n // call potential deferred free routines\n _mi_deferred_free(heap, false);\n \n- // free delayed frees from other threads (but skip contended ones)\n- _mi_heap_delayed_free_partial(heap);\n-\n // collect every once in a while (10000 by default)\n const long generic_collect = mi_option_get_clamp(mi_option_generic_collect, 1, 1000000L);\n if (heap->generic_collect_count >= generic_collect) {\n```\n\n\u003c/details>\n\nWith option 1, the peak is back at 2.1.2 levels on macOS, Linux (GCC 8.5) and Windows (MSVC), and the full test suite passes on a free-threaded build of `main` (macOS). I couldn't measure a cost: in a quick allocation benchmark (small `bytes`, dicts of lists, 4 MiB blocks) the patched and unpatched builds of `main` are within run-to-run noise of each other.\n\nA middle ground I also tried: process the delayed frees only right before a large allocation in `mi_find_page()`. That keeps the batching for small objects and also fixes this, but it was less steady: in some runs (16 threads × 8 MiB, 8 threads × 16 MiB) it peaked at about 700 MiB, where option 1 stayed at 300–350 MiB.\n\n### Related\n\n- #158972 (memory not returned to the OS on macOS) is a different problem: small objects, pages waiting for QSBR, and it predates 2.5.2. Its fix in progress, #160150, doesn't change this.\n- #159718 (mimalloc v3) makes this worse: the script keeps nearly everything, because v3 does the same housekeeping every 1,000 generic mallocs and the PR sets `page_full_retain` to -1.\n\n\n### CPython versions tested on:\n\n3.13, 3.14, 3.15, CPython main branch\n\n### Operating systems tested on:\n\nWindows, macOS, Linux\n\n\u003c!-- gh-linked-prs -->\n### Linked PRs\n* gh-161533\n* gh-161537\n* gh-161538\n\u003c!-- /gh-linked-prs -->\n","author":{"url":"https://github.com/maikschulze","@type":"Person","name":"maikschulze"},"datePublished":"2026-11-05T12:25:28Z","url":"https://github.com/python/cpython/issues/161420"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:1becee25-96b3-8025-df92-cfd96ae39084 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | 841C:23DB5F:433F13:54A10B:6B01FE96 |
| html-safe-nonce | ef915471386b732e750b71418212e3559022e4a403f8a1a2c36199970319297a |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI4NDFDOjIzREI1Rjo0MzNGMTM6NTRBMTBCOjZCMDFGRTk2IiwidmlzaXRvcl9pZCI6Ijc3ODk0MTE1OTEwMDIzOTAxNjYiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | 3c37eec829c91298eaf2c5ec3f8f1d2176902886764de2d8907d57cd9fa7a227 |
| hovercard-subject-tag | issue:6223137446 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/161420/issue_layout |
| twitter:image | https://opengraph.githubassets.com/6b9cbc10345b132bc8938b62c1a2b06292b6469d2b515030d8fe81ffbf5d3713/python/cpython/issues/161420 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/6b9cbc10345b132bc8938b62c1a2b06292b6469d2b515030d8fe81ffbf5d3713/python/cpython/issues/161420 |
| og:image:alt | Bug report I've been looking into the performance of streaming decompression with compression.zstd (archive extraction in a package manager), first on 3.14 and 3.15, then on the free-threaded build... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | maikschulze |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | canary-2 |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width