René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Free-threaded 3.15: a `functools.partial` shared between threads no longer scales (it slows down as you add threads)","articleBody":"### Bug report\n\nI was migrating some of my \"massively threaded\" 3.14t code to 3.15t and my performance agent saw everything collapse under concurrency after the python version change.\n\nTurns out a new 3.15-only fix (?) for a `partial` application is now over-locking everything. For my own use I just migrated away from using `functools.partial` everywhere now, but ... FYI i guess.\n\nagent report:\n\n=========================\n\nFree-threaded 3.15: a `functools.partial` shared between threads no longer scales (it slows down as you add threads)\n\n## Summary\n\nOn the free-threaded build, CPython 3.15 calls to a `functools.partial` stop\nscaling across threads when the partial (or just its target function) is shared\nbetween threads. On a 10-core Apple M1 Max, 24 threads calling one module-level\n`partial(f, 1)` reach **0.20x** the single-thread rate, so total throughput drops\nwhen threads are added. 3.14.3t reaches **6.37x** on the same machine. The\nslowdown shows up at 2 threads (0.34x). The cause is the use-after-free fix for\ngh-154189 (PR gh-154508, 3.15 backport gh-154652, first shipped in 3.15.0rc1). It\nmade `partial_vectorcall` take and drop strong references to `pto->fn`,\n`pto->args` and `pto->kw` on every call. In the free-threaded build an\nincref/decref from a thread that does not own the object is an atomic\nread-modify-write on that object's `ob_ref_shared`. Every calling thread now writes\nthe same three cache lines (the target function, the bound-args tuple, and the\nkeywords dict, which partial allocates even when empty) six times per call. In\n3.14 the call path takes no references on these objects: it reads them borrowed,\nand the callee frame uses stackrefs that skip deferred-refcount and immortal\nobjects. Giving each thread its own partial does **not** help when the target\nfunction is shared (0.24x). Making the partial, its function and its args tuple\nimmortal also does not help, because the per-partial empty keywords dict is still\nrefcounted (0.20x). Scaling comes back only when all three (fn, args, keywords)\nare immortal (5.43x). The source of `partial_vectorcall` is the same in v3.15.0\n(tagged 2026-10-09) and `main`. The 3.14 branch did not take the backport.\n\n## Environment\n\n|               |                                                                                                                           |\n| ------------- | ------------------------------------------------------------------------------------------------------------------------- |\n| Interpreter A | `3.14.3 free-threading build (main, Mar 21 2026, 17:04:12) [Clang 22.1.1 ]` (pyenv source build)                          |\n| Interpreter B | `3.15.0rc3 free-threading build (main, Oct  3 2026, 00:55:19) [Clang 22.1.3 ]` (python-build-standalone, installed by uv) |\n| GIL           | disabled in both (`sys._is_gil_enabled() == False`)                                                                       |\n| Platform      | `macOS-14.8.2-arm64-arm-64bit-Mach-O`                                                                                     |\n| CPU           | Apple M1 Max, 10 cores (8 performance + 2 efficiency), `os.cpu_count() == 10`, 128-byte cache line                        |\n| Load          | other work was running on the host, and probes ran under `nice -n 10`                                                     |\n\n24 threads on 10 cores is deliberate oversubscription (it matches a 24-worker\nserver). The no-shared-state control loop therefore tops out at about 5.5-6.7x on\nthis machine, not 24x.\n\n## Results\n\nThe minimal repro is below. Each cell is aggregate calls/s over 1 s per\nmeasurement. Speedup = calls/s at N threads ÷ calls/s at 1 thread, for the same\nshape.\n\n| shape                                               | 3.14.3t 1T | 3.14.3t 24T | 3.14.3t speedup | 3.15.0rc3t 1T | 3.15.0rc3t 24T | 3.15.0rc3t speedup |\n| --------------------------------------------------- | ---------: | ----------: | --------------: | ------------: | -------------: | -----------------: |\n| control (a function private to each thread)         |     31.3 M |     208.8 M |           6.67x |        38.4 M |        216.5 M |              5.63x |\n| **shared**: one `partial(f, 1)` for all threads     |     19.1 M |     121.8 M |       **6.37x** |        18.6 M |          3.7 M |          **0.20x** |\n| **own**: one `partial(f, 1)` per thread, `f` shared |     19.1 M |     103.2 M |           5.40x |        19.9 M |          4.7 M |          **0.24x** |\n\nThread sweep (0.5 s per point):\n\n| threads | 3.14.3t shared | 3.15.0rc3t shared | 3.15.0rc3t own (shared `f`) | 3.15.0rc3t control |\n| ------: | -------------: | ----------------: | --------------------------: | -----------------: |\n|       1 | 1.00x (19.1 M) |    1.00x (17.9 M) |              1.00x (20.4 M) |     1.00x (38.8 M) |\n|       2 |          1.57x |         **0.34x** |                       0.48x |              1.89x |\n|       4 |          3.49x |             0.30x |                       0.29x |              2.85x |\n|       8 |          5.40x |             0.20x |                       0.20x |              3.42x |\n|      24 |          4.95x |             0.21x |                       0.22x |              4.50x |\n\nOn 3.15, 1-thread cost is unchanged. An uncontended atomic is cheap. The cost is\nall in contention.\n\n### Variants that isolate the mechanism\n\nThese come from `probe_variants.py` (Appendix B), with 1 and 24 threads at 1 s\neach. \"shared\" means one object built by the main thread and called by all\nthreads. \"own\" means each worker builds its own object from the same recipe.\nEvery thread passes its own argument object. Pinning is done with the exported\n`_Py_SetImmortal` through `ctypes`.\n\n| variant                                    | what is shared / pinned      | 3.14.3t shared | 3.14.3t own | 3.15.0rc3t shared | 3.15.0rc3t own |\n| ------------------------------------------ | ---------------------------- | -------------: | ----------: | ----------------: | -------------: |\n| `partial(f, 1)(2)`                         |                              |          6.21x |       6.22x |         **0.17x** |      **0.19x** |\n| own partial over a **per-thread** function | nothing shared in \"own\"      |          6.08x |       6.16x |             0.18x |      **6.42x** |\n| `partial(max, 1)(2)` (builtin target)      | `max` shared in both shapes  |          6.45x |       6.26x |             0.19x |          0.20x |\n| partial subclass `(f, 1)(2)`               |                              |          6.24x |       6.29x |             0.17x |          0.17x |\n| `operator.call(p, 2)`                      |                              |          5.97x |       6.02x |             0.19x |          0.18x |\n| pin fn only                                | `p.func` immortal            |          5.53x |           — |             0.21x |         5.72x¹ |\n| pin keywords dict only                     | `p.keywords` immortal        |          5.54x |           — |             0.17x |          0.19x |\n| pin partial + fn + args                    | keywords dict **not** pinned |          5.46x |           — |         **0.20x** |         5.55x¹ |\n| pin partial + fn + args + keywords dict    | everything the call touches  |          5.77x |           — |         **5.43x** |         5.50x¹ |\n| `partial(fk, a=1)(2)` (stored keyword)     |                              |      **1.00x** |       6.76x |             0.45x |          0.29x |\n| `partial(fk, a=1)`, everything pinned      |                              |          0.83x |      6.15x¹ |             0.67x |         5.65x¹ |\n\n¹ In the pinning rows, the \"own\" partials are built after the shared run has\nalready made `f` (or `fk`) immortal, so they show that a private partial over an\nimmortal target scales.\n\nWhat the table shows:\n\n- The partial type, its vectorcall path and the partial object's own refcount are\n  not the problem. Own partials over own functions scale on 3.15 (6.42x).\n- The problem is a write to every object the call touches: the target function,\n  the args tuple and the keywords dict. All three have to be immortal before 3.15\n  scales again. Pinning any one or two of them still leaves 0.17-0.21x.\n- Keyword-bound partials serialise on **both** versions even with everything\n  immortal (0.83x / 0.67x). That is a separate, older bottleneck: a per-call lock\n  on the shared keywords dict. See [Keyword partials](#keyword-partials-a-separate-older-bottleneck).\n\n## Minimal reproducer\n\nStandard library only. Save it as `repro_partial_shared.py` and run it with a\nfree-threaded interpreter:\n\n```\npython3.14t repro_partial_shared.py            # 1 and 24 threads, 1 s each\npython3.15t repro_partial_shared.py            # 1 and 24 threads, 1 s each\npython3.15t repro_partial_shared.py 1 2 4 8 24 0.5   # sweep, 0.5 s per point\n```\n\n```python\n\"\"\"Free-threaded scaling of ONE shared functools.partial vs one partial per thread.\n\nStandard library only. Run on a free-threaded build (python3.14t / python3.15t):\n\n    python3.15t repro_partial_shared.py            # 1 and 24 threads, 1 s each\n    python3.15t repro_partial_shared.py 1 8 24 2.0 # thread counts..., seconds\n\nEach thread calls its target in a tight loop for SECONDS and counts calls.\n\"speedup\" = aggregate calls/s at N threads / calls/s at 1 thread.\n\"\"\"\n\nimport functools\nimport os\nimport platform\nimport sys\nimport threading\nimport time\n\n\ndef f(a, b):\n    return b\n\n\nSHARED = functools.partial(f, 1)  # one object, called by every thread\n\n\ndef control_target(b):  # the no-shared-state control: each thread defines its own below\n    return b\n\n\ndef worker(mode, seconds, barrier, rates, slot):\n    if mode == \"shared\":\n        target = SHARED\n    elif mode == \"own\":\n        target = functools.partial(f, 1)  # one partial per thread\n    else:  # \"control\": a function object private to this thread\n        def target(b):\n            return b\n    clock = time.perf_counter\n    barrier.wait()\n    calls = 0\n    start = clock()\n    end = start + seconds\n    while True:\n        for _ in range(1000):\n            target(2)\n        calls += 1000\n        now = clock()\n        if now >= end:\n            break\n    rates[slot] = calls / (now - start)\n\n\ndef measure(mode, threads, seconds):\n    rates = [0.0] * threads\n    barrier = threading.Barrier(threads)\n    ts = [\n        threading.Thread(target=worker, args=(mode, seconds, barrier, rates, i))\n        for i in range(threads)\n    ]\n    for t in ts:\n        t.start()\n    for t in ts:\n        t.join()\n    return sum(rates)\n\n\ndef main():\n    nums = [a for a in sys.argv[1:] if \".\" not in a]\n    secs = [a for a in sys.argv[1:] if \".\" in a]\n    counts = [int(a) for a in nums] or [1, 24]\n    seconds = float(secs[0]) if secs else 1.0\n    gil = sys._is_gil_enabled()\n    print(sys.version.replace(\"\\n\", \" \"))\n    print(platform.platform(), \"| cpus:\", os.cpu_count(), \"| GIL enabled:\", gil)\n    base = {}\n    print(f\"{'mode':8} {'threads':>7} {'calls/s':>14} {'speedup':>8}\")\n    for mode in (\"control\", \"shared\", \"own\"):\n        for n in counts:\n            rate = measure(mode, n, seconds)\n            if n == counts[0]:\n                base[mode] = rate\n            speed = rate / base[mode]\n            print(f\"{mode:8} {n:7d} {rate:14,.0f} {speed:7.2f}x\")\n\n\nif __name__ == \"__main__\":\n    main()\n```\n\nOutput on the machine above:\n\n```\n3.14.3 free-threading build (main, Mar 21 2026, 17:04:12) [Clang 22.1.1 ]\nmacOS-14.8.2-arm64-arm-64bit-Mach-O | cpus: 10 | GIL enabled: False\nmode     threads        calls/s  speedup\ncontrol        1     31,330,215    1.00x\ncontrol       24    208,830,981    6.67x\nshared         1     19,134,731    1.00x\nshared        24    121,813,373    6.37x\nown            1     19,124,673    1.00x\nown           24    103,228,171    5.40x\n\n3.15.0rc3 free-threading build (main, Oct  3 2026, 00:55:19) [Clang 22.1.3 ]\nmacOS-14.8.2-arm64-arm-64bit-Mach-O | cpus: 10 | GIL enabled: False\nmode     threads        calls/s  speedup\ncontrol        1     38,416,566    1.00x\ncontrol       24    216,453,219    5.63x\nshared         1     18,561,162    1.00x\nshared        24      3,729,330    0.20x\nown            1     19,941,575    1.00x\nown           24      4,743,980    0.24x\n```\n\nThe probe keeps the shared object in a local variable, so only the call is\nmeasured. It does not measure a `LOAD_GLOBAL` of a non-deferred object, which\ncarries its own refcount traffic.\n\n## Root cause\n\n### The change\n\n[gh-154189](https://github.com/python/cpython/issues/154189), \"use-after-free:\nfunctools partial\", was reported 2026-07-19 against `main` (free-threading debug\nTSan build). It is a **single-threaded re-entrancy** bug. `partial_vectorcall`\nheld a raw `PyObject **` into `pto->args`. In the keyword-merge path, a call\nkeyword's `__hash__` could call `partial.__setstate__` on the same partial, which\nfreed the old args tuple while the C frame was still using it.\n\nThe fix is [PR gh-154508](https://github.com/python/cpython/pull/154508), merged to\n`main` 2026-07-24 as\n[`1a742d4`](https://github.com/python/cpython/commit/1a742d403243ac9915ffcc54e29516e58c44dc42),\nand backported to 3.15 by [gh-154652](https://github.com/python/cpython/pull/154652)\n([`6246342`](https://github.com/python/cpython/commit/624634242965e50d587a6a01b9faba27de0b2344),\n2026-07-27). The NEWS entry reads:\n\n> Fixed a potential use-after-free when calling `functools.partial`. Now, when\n> invoking a `partial()` object, the stored function, positional arguments, and\n> keyword arguments are preserved for the duration of the call in case of\n> reentrancy.\n\nWhich releases have it:\n\n- 3.15: first in **v3.15.0rc1**. v3.15.0b4 does not contain the commit; rc1 does.\n  Present in rc2, rc3 and **v3.15.0** final (tagged 2026-10-09; the\n  `partial_vectorcall` source is byte-identical to rc3) and in `main`.\n- 3.14: not backported. The automatic cherry-pick conflicted, and the author and\n  reviewer agreed a backport was not worth the effort (\"I don't think PR needs to\n  be backported to 3.14.\" / \"I agree; it doesn't look worth the effort.\"). v3.14.8\n  has the old code.\n\nThe issue and PR do not discuss free-threaded performance, and the PR includes no\nbenchmark.\n\n### Before: v3.14.3 `Modules/_functoolsmodule.c`, `partial_vectorcall` (positional path)\n\n```c\n    /* pto->kw is mutable, so need to check every time */\n    if (PyDict_GET_SIZE(pto->kw)) {\n        return partial_vectorcall_fallback(tstate, pto, args, nargsf, kwnames);\n    }\n    ...\n    PyObject **pto_args = _PyTuple_ITEMS(pto->args);\n    Py_ssize_t pto_nargs = PyTuple_GET_SIZE(pto->args);\n    ...\n        if (pto_nargs == 1 && (nargsf & PY_VECTORCALL_ARGUMENTS_OFFSET)) {\n            PyObject **newargs = (PyObject **)args - 1;\n            PyObject *tmp = newargs[0];\n            newargs[0] = pto_args[0];\n            PyObject *ret = _PyObject_VectorcallTstate(tstate, pto->fn, newargs,\n                                                       nargs + 1, kwnames);\n            newargs[0] = tmp;\n            return ret;\n        }\n```\n\nAll reads are borrowed. The call path performs no refcount writes on `pto->fn`,\n`pto->args` or `pto->kw`.\n\n### After: v3.15.0rc3 / v3.15.0 / main, `partial_vectorcall`\n\n```c\n    PyObject *result = NULL;\n    PyObject *partial_function = Py_NewRef(pto->fn);\n    PyObject *partial_args = Py_NewRef(pto->args);\n    PyObject *partial_keywords = Py_NewRef(pto->kw);\n\n    PyObject **pto_args = _PyTuple_ITEMS(partial_args);\n    Py_ssize_t pto_nargs = PyTuple_GET_SIZE(partial_args);\n    Py_ssize_t pto_nkwds = PyDict_GET_SIZE(partial_keywords);\n    ...\n            result = _PyObject_VectorcallTstate(tstate, partial_function, newargs,\n                                                nargs + 1, kwnames);\n            newargs[0] = tmp;\n            goto done;\n    ...\n done:\n    Py_DECREF(partial_function);\n    Py_DECREF(partial_args);\n    Py_DECREF(partial_keywords);\n    return result;\n```\n\nThe three `Py_NewRef` / `Py_DECREF` pairs run on **every** call, including the\npositional-only fast paths, where no user code runs between reading the fields and\nmaking the call. Keyword merging is the only place the re-entrancy in gh-154189\ncan happen.\n\n## Mechanism\n\nIn the free-threaded build each object has an owning thread (`ob_tid`), a\nnon-atomic `ob_ref_local` that only the owner writes, and an atomic\n`ob_ref_shared` that every other thread writes (`Include/refcount.h`, 3.15):\n\n```c\nstatic inline Py_ALWAYS_INLINE void Py_INCREF(PyObject *op)\n{\n    uint32_t local = _Py_atomic_load_uint32_relaxed(&op->ob_ref_local);\n    uint32_t new_local = local + 1;\n    if (new_local == 0) {\n        // local is equal to _Py_IMMORTAL_REFCNT_LOCAL: do nothing\n        return;\n    }\n    if (_Py_IsOwnedByCurrentThread(op)) {\n        _Py_atomic_store_uint32_relaxed(&op->ob_ref_local, new_local);\n    }\n    else {\n        _Py_atomic_add_ssize(&op->ob_ref_shared, (1 \u003c\u003c _Py_REF_SHARED_SHIFT));\n    }\n```\n\n`Py_DECREF` from a non-owner goes to `_Py_DecRefShared` (`Objects/object.c`). That\nfunction runs a compare-and-swap loop on `ob_ref_shared`, and the loop retries\nunder contention.\n\nA long-lived partial and its `fn`, `args` and `kw` are owned by the thread that\ncreated them, usually the importing or main thread. Every worker thread is\ntherefore a non-owner. So each call on 3.15 does 3 atomic adds and 3 CAS loops on\nthree objects that every thread touches. Each read-modify-write needs exclusive\nownership of the cache line, so the line holding each object header moves between\ncores on every call. The header also holds `ob_type`, `ob_tid` and\n`ob_ref_local`, which every caller reads, so those reads miss too. Throughput is\nthen limited by how fast the coherence fabric can move those lines, not by the\nnumber of cores. The aggregate settles around 3-5 M calls/s at any thread count:\n0.34x at 2 threads, about 0.2x from 8 threads up.\n\n**Why own partials over a shared function still collapse.** `fn` is shared, so\n`Py_NewRef(pto->fn)` writes the same line from every thread. The per-thread args\ntuple and keywords dict are private, so the slowdown is a little smaller (0.24x\nvs 0.20x), but it is the same effect. Own partials over **per-thread** functions\ntouch only private objects and scale (6.42x).\n\n**Why 3.14 did not have this.** The callee side never writes the shared refcounts.\n`_PyEval_Vector` takes its references with `PyStackRef_FromPyObjectNew`:\n\n```c\n    for (size_t i = 0; i \u003c argcount; i++) {\n        arguments[i] = PyStackRef_FromPyObjectNew(args[i]);\n    }\n    ...\n    _PyInterpreterFrame *frame = _PyEvalFramePushAndInit(\n        tstate, PyStackRef_FromPyObjectNew(func), locals,\n```\n\nIn the free-threaded build that is a no-op for immortal objects (small ints) and\nfor deferred-refcount objects. Top-level functions and methods are created with\ndeferred refcounting (`Objects/funcobject.c`: `_PyObject_SetDeferredRefcount` when\nthe code is not `CO_NESTED`, or is `CO_METHOD`). `Py_NewRef` does not check the\ndeferred bit, so on 3.15 the same deferred `f` gets an atomic increment on every\ncall. That is why a plain shared `f(…)` call scales on both versions (6.0x /\n6.7x) while `partial(f, 1)` does not on 3.15.\n\n**Why making partial + fn + args immortal did not help.** `partial_new` always\nstores a dict in `pto->kw`. With no keywords it is a fresh, empty dict per partial,\nowned by the creating thread. 3.15 increfs and decrefs that dict on every call even\nthough it is empty. Pinning the partial object itself does nothing, because the\ncall path never touches the partial's own refcount. Only pinning **fn, args and\nthe keywords dict** brings scaling back (5.43x). The same reasoning covers a\npartial of a builtin (`max`): `PyCFunction` objects are neither immortal nor\ndeferred, and the 3.15 path increfs them.\n\n## Is it intended?\n\nThe correctness fix is intended: a real use-after-free reachable from pure Python.\nThe free-threaded cost looks unintended. Nothing in the issue, the PR review or the\nNEWS entry mentions it. The PR has no benchmark, and the 3.14 backport was skipped\nfor effort, not because of any trade-off. The fix also guards less than its cost\nsuggests:\n\n- It covers the same-thread re-entrancy case. It does **not** make concurrent\n  `__setstate__` safe. `Py_NewRef(pto->args)` loads the field and increments it\n  later, so another thread can free the tuple in between. gh-157841 (open,\n  \"concurrent functools.partial.**setstate** crashes with repr(), calls, or\n  another **setstate**\") reports call+`__setstate__` still crashing on `main` with\n  this fix in place.\n- A proposed fix for gh-157841, PR gh-157979 (closed unmerged 2026-09-30 because\n  the CLA was not signed), took `Py_BEGIN_CRITICAL_SECTION(self)` in\n  `partial_vectorcall` and `partial_call` on every call. That design would replace\n  three contended atomics with a contended per-object mutex. For a shared partial\n  it would be at least as serialising. The existing per-call critical section on\n  the keywords dict already holds keyword partials to 0.67x with everything\n  immortal, below.\n\nNo open issue tracks the performance regression. Searches of the cpython tracker\nfor partial plus free-threading, scaling or contention since 2026-07 found only the\ncrash reports above.\n\n## Suggested upstream fixes\n\n`__setstate__` is the only writer of `fn`, `args` and `kw`. It is called by pickle\nand `copy` on a freshly built object, almost never on a live, shared one. The fix\nshould make that rare writer pay and keep the per-call reader free of shared\nwrites. In rough order of preference:\n\n1. **Take the per-call snapshot as GC-visible C stack references instead of\n   `Py_NewRef`.** CPython already has the machinery: `_PyCStackRef` with\n   `_PyThreadState_PushCStackRefNew` / `_PyThreadState_PopCStackRef`\n   (`Include/internal/pycore_stackref.h`). `typeobject.c`'s `call_method` uses it\n   for exactly this shape: it holds the looked-up `__call__` function for the\n   duration of a call without touching that shared function's refcount. That is\n   why a per-thread instance of a class with `__call__` scales on 3.15 even though\n   every thread calls the same `__call__` function. `PyStackRef_FromPyObjectNew`\n   costs nothing for deferred and immortal objects, and the GC sees C stack refs,\n   so a deferred object cannot be collected under the call. For `fn` this is\n   enough on its own when `fn` is a top-level function (already deferred). For\n   `args` and `kw`, `partial_new` / `partial_setstate` could enable deferred\n   refcounting on the tuple and dict they store, and stop allocating a fresh empty\n   dict when there are no keywords (keep `NULL`, or share one immortal empty\n   sentinel internally). Result: correctness against same-thread re-entrancy is\n   preserved, and the call does no shared writes.\n2. **Keep the positional fast paths borrowed, as in 3.14, and take strong\n   references only on the keyword-merge path.** That path is where user code\n   (`__hash__` / `__eq__` of call keywords, `PyDict_Copy`, `PyDict_SetItem`) runs\n   before the call. This is the narrowest fix for the reported reproducer. It\n   should be paired with (3) to cover a callee that re-enters `__setstate__` on its\n   own partial.\n3. **Make the displaced state outlive any call in progress, in the writer.** In\n   `__setstate__`, retire the old `fn`, `args` and `kw` with\n   `_PyObject_XDecRefDelayed` (QSBR) in the free-threaded build. That handles\n   cross-thread readers, the gh-157841 shape. It does not fully cover same-thread\n   re-entrancy, because the calling thread can report a quiescent state while its\n   call is still running (for example, when it detaches for blocking I/O).\n   Alternatively, keep the displaced objects alive on the partial until\n   `tp_dealloc`. That is simple, costs nothing per call, and only matters for the\n   rare object that is re-stated.\n4. **For gh-157841, do not add a per-call lock on the partial.** Lock in the\n   writer, and publish a coherent state the reader can load without writing a\n   shared line: an immutable state record holding `fn`, `args`, `kw` and\n   `phcount`, read with an acquire load into a C stack ref and retired via QSBR.\n   An atomic pointer plus `_Py_TryIncrefCompare` on a non-deferred state object\n   would bring back the same contention this report measures.\n\n### Keyword partials: a separate, older bottleneck\n\nA partial with stored keywords serialises when shared, on **both** versions, even\nwith everything immortal: 0.83x on 3.14.3t, 0.67x on 3.15.0rc3t.\n\n- In 3.14 the call falls back to `partial_call`, which does\n  `PyDict_Copy(pto->kw)`. That copy locks the shared source dict.\n- In 3.15 the vectorcall keyword path iterates `pto->kw` inside\n  `Py_BEGIN_CRITICAL_SECTION(keyword_dict)` on every call. That critical section\n  was added for `PyDict_Next` safety by gh-145446.\n\nBoth are per-call acquisitions of the same per-object mutex. The lock exists\nbecause `p.keywords` returns the live dict and user code can mutate it. A\ncontention-free design would snapshot the keywords into immutable\n`(kwnames tuple, values)` at construction and in `__setstate__`. That is a\nbehaviour change for code that mutates `p.keywords` in place, which is probably\nrare and undocumented.\n\n## Workarounds for users (until fixed)\n\nMeasured on 3.15.0rc3t, shared object called by 24 threads:\n\n| replacement for a shared `partial(f, x)`                         | 3.15.0rc3t | 3.14.3t | notes                                                                                                                |\n| ---------------------------------------------------------------- | ---------: | ------: | -------------------------------------------------------------------------------------------------------------------- |\n| top-level `def p(b): return f(CONST, b)`                         |      6.71x |   5.96x | Best with no C code, when the bound value is a constant or immortal. Module-level functions are deferred-refcounted. |\n| closure over the bound value, **cells pinned** (immortal)        |      6.33x |   6.01x | Pin with `PyUnstable_SetImmortal` (3.15 public unstable C API) or `_Py_SetImmortal`; leaks by design.                |\n| closure over the bound value, cells **not** pinned               |      0.18x |   0.14x | `COPY_FREE_VARS` increfs each shared cell per call, on both versions.                                                |\n| instance of a slotted class with `__call__`, **instance pinned** |      5.55x |   6.05x |                                                                                                                      |\n| instance of a slotted class with `__call__`, not pinned          |      0.21x |   0.31x | `self` is increfed into the callee frame per call, on both versions.                                                 |\n| partial with `fn`, `args` **and** `keywords` all pinned          |      5.43x |   5.77x | Pinning only partial + fn + args is not enough (0.20x).                                                              |\n| one partial per thread **over a per-thread function**            |      6.42x |   6.16x | One partial per thread over a shared function still collapses (0.19x).                                               |\n\nGeneral rule on free-threaded CPython: a long-lived callable shared by many\nthreads scales only if every object the call path increfs is immortal or deferred\nand reached through stackrefs. Bound `self`, closure cells and, on 3.15, partial\nfields are increfed per call.\n\n## Appendix A: other common callables (shared vs own, 1 → 24 threads)\n\n| callable                                      | 3.14.3t shared | 3.14.3t own | 3.15.0rc3t shared | 3.15.0rc3t own | verdict                                                        |\n| --------------------------------------------- | -------------: | ----------: | ----------------: | -------------: | -------------------------------------------------------------- |\n| plain top-level function                      |          5.96x |       6.23x |             6.71x |          6.07x | scales on both                                                 |\n| `operator.itemgetter(1)(tup)`                 |          6.11x |       6.13x |             5.85x |          5.87x | scales on both                                                 |\n| `operator.attrgetter(\"a\")(obj)`               |          6.16x |       6.06x |             5.82x |          5.92x | scales on both                                                 |\n| `operator.methodcaller(\"count\", 1)(tup)`      |          0.31x |       0.30x |             5.15x |          5.23x | serialises on 3.14 even per thread; fixed in 3.15              |\n| `operator.methodcaller(\"meth\", b=1)(obj)`     |          0.25x |       0.28x |             5.43x |          5.70x | same as above                                                  |\n| bound builtin method `{}.get`                 |          5.58x |       5.46x |             5.36x |          5.70x | scales on both                                                 |\n| bound Python method `obj.meth`, not pinned    |          0.17x |       5.96x |             0.14x |          5.82x | shared serialises on both (incref of `self`)                   |\n| bound Python method, instance + method pinned |          5.93x |       5.76x |             5.17x |          5.99x | scales                                                         |\n| slotted instance `__call__`, not pinned       |          0.31x |       6.72x |             0.21x |          6.20x | shared serialises on both (incref of `self`)                   |\n| closure, cells not pinned                     |          0.14x |       5.94x |             0.18x |          6.36x | shared serialises on both (cell incref)                        |\n| `functools.partial` (positional)              |          6.21x |       6.22x |             0.17x |          0.19x | **regressed in 3.15**                                          |\n| `functools.partial` (keyword)                 |          1.00x |       6.76x |             0.45x |          0.29x | shared serialises on both (dict lock); 3.15 adds the fn incref |\n\nOnly `functools.partial` regressed between 3.14 and 3.15. `methodcaller` improved\nin 3.15. The 3.14 collapse even with per-thread objects is probably the per-call\nreference to the shared method descriptor found by attribute lookup; that was not\ninvestigated further. The bound-method, slotted `__call__` and closure collapses\nare older and the same on both versions. Each comes from a per-call incref of a\nshared, non-immortal, non-deferred object (`self` or a cell), and pinning that\nobject fixes it.\n\n## Appendix B: the variant probe\n\nRun it as `python3.15t probe_variants.py \u003cvariant ...>` (`LIST` prints the names).\nPinning variants permanently immortalise the target function, so run each one in a\nfresh process, as was done for the tables above.\n\n```python\n\"\"\"Variant probe: which callables stop scaling when one instance is shared by N threads.\n\nStandard library only (ctypes is used solely to call the exported\n`_Py_SetImmortal` so a variant can pin an object). Free-threaded build required.\n\n    python3.15t probe_variants.py LIST                 # list variant names\n    python3.15t probe_variants.py partial_pos closure  # run some variants\n    python3.15t probe_variants.py --seconds 0.5 --threads 1,24 all\n\nFor every variant two shapes are measured at each thread count:\n  shared  one object built once by the main thread, called by every thread\n  own     each thread builds its own object (by the same recipe) before the start\nEvery thread passes its own argument object, so only the callable is shared.\nspeedup = aggregate calls/s at N threads / calls/s at 1 thread (same shape).\n\"\"\"\n\nimport ctypes\nimport functools\nimport json\nimport operator\nimport os\nimport sys\nimport threading\nimport time\n\n_set_immortal = ctypes.pythonapi._Py_SetImmortal\n_set_immortal.argtypes = [ctypes.py_object]\n_set_immortal.restype = None\n\n\ndef pin(*objs):\n    for o in objs:\n        _set_immortal(o)\n\n\ndef f(a, b):\n    return b\n\n\ndef fk(b, a=0):\n    return b\n\n\ndef g(b):\n    return b\n\n\nclass Slotted:\n    __slots__ = (\"a\",)\n\n    def __init__(self, a):\n        self.a = a\n\n    def __call__(self, b):\n        return b\n\n    def meth(self, b):\n        return b\n\n\nclass PartialSub(functools.partial):\n    pass\n\n\ndef make_closure(a):\n    def inner(b):\n        return b if a else a\n    return inner\n\n\n# Each variant: name -> (factory, argument factory, description).\n# factory(shared: bool) returns the callable; for \"shared\" it runs once in the\n# main thread, for \"own\" once per worker thread.\ndef _partial_pinned(which):\n    def factory(shared):\n        p = functools.partial(f, 1)\n        if shared:\n            objs = {\"all\": (p, p.func, p.args, p.keywords),\n                    \"fn_args\": (p, p.func, p.args),\n                    \"kw\": (p.keywords,),\n                    \"fn\": (p.func,),\n                    \"args\": (p.args,)}[which]\n            pin(*objs)\n        return p\n    return factory\n\n\ndef _partial_kw_pinned(shared):\n    p = functools.partial(fk, a=1)\n    if shared:\n        pin(p, p.func, p.args, p.keywords)\n    return p\n\n\ndef _partial_own_fn(shared):\n    if shared:\n        return functools.partial(f, 1)\n    def own_f(a, b):\n        return b\n    return functools.partial(own_f, 1)\n\n\ndef _closure_pinned(shared):\n    c = make_closure(1)\n    if shared:\n        pin(*c.__closure__)\n    return c\n\n\ndef _bound_py(shared):\n    return Slotted(1).meth\n\n\ndef _slotted_pinned(shared):\n    s = Slotted(1)\n    if shared:\n        pin(s)\n    return s\n\n\ndef _bound_py_pinned(shared):\n    s = Slotted(1)\n    m = s.meth\n    if shared:\n        pin(s, m)\n    return m\n\n\ndef _bound_c(shared):\n    return {}.get  # dict.get bound to a dict; call d.get(2) -> None\n\n\nVARIANTS = {\n    \"func\": (lambda s: g, lambda: 2, \"plain module-level Python function g(b)\"),\n    \"partial_pos\": (lambda s: functools.partial(f, 1), lambda: 2, \"partial(f, 1)(2)\"),\n    \"partial_ownfn\": (_partial_own_fn, lambda: 2,\n                      \"own = partial over a per-thread function (shared = partial_pos)\"),\n    \"partial_pin_all\": (_partial_pinned(\"all\"), lambda: 2,\n                        \"shared partial; partial, func, args tuple, keywords dict immortal\"),\n    \"partial_pin_fn_args\": (_partial_pinned(\"fn_args\"), lambda: 2,\n                            \"shared partial; partial, func, args immortal (keywords dict NOT)\"),\n    \"partial_pin_kw\": (_partial_pinned(\"kw\"), lambda: 2,\n                       \"shared partial; only the empty keywords dict immortal\"),\n    \"partial_pin_fn\": (_partial_pinned(\"fn\"), lambda: 2,\n                       \"shared partial; only func immortal\"),\n    \"partial_kw\": (lambda s: functools.partial(fk, a=1), lambda: 2, \"partial(fk, a=1)(2)\"),\n    \"partial_kw_pin_all\": (_partial_kw_pinned, lambda: 2,\n                           \"shared partial(fk, a=1); partial, func, args, keywords immortal\"),\n    \"partial_builtin\": (lambda s: functools.partial(max, 1), lambda: 2,\n                        \"partial(max, 1)(2) -- builtin target\"),\n    \"partial_sub\": (lambda s: PartialSub(f, 1), lambda: 2, \"partial subclass(f, 1)(2)\"),\n    \"partial_opcall\": (lambda s: functools.partial(f, 1), lambda: 2,\n                       \"operator.call(p, 2) [loop uses operator.call]\"),\n    \"closure\": (lambda s: make_closure(1), lambda: 2, \"closure over a cell, not pinned\"),\n    \"closure_pin\": (_closure_pinned, lambda: 2, \"closure, cells immortal when shared\"),\n    \"slotted_call\": (lambda s: Slotted(1), lambda: 2, \"slotted instance with __call__\"),\n    \"itemgetter\": (lambda s: operator.itemgetter(1), lambda: (1, 2, 3), \"itemgetter(1)(tup)\"),\n    \"attrgetter\": (lambda s: operator.attrgetter(\"a\"), lambda: Slotted(2), \"attrgetter('a')(obj)\"),\n    \"methodcaller\": (lambda s: operator.methodcaller(\"count\", 1), lambda: (1, 2, 3),\n                     \"methodcaller('count', 1)(tup)\"),\n    \"methodcaller_kw\": (lambda s: operator.methodcaller(\"meth\", b=1), lambda: Slotted(2),\n                        \"methodcaller('meth', b=1)(obj)\"),\n    \"bound_py\": (_bound_py, lambda: 2, \"bound method of a Python class instance\"),\n    \"slotted_call_pin\": (_slotted_pinned, lambda: 2, \"slotted __call__, instance immortal when shared\"),\n    \"bound_py_pin\": (_bound_py_pinned, lambda: 2, \"bound Python method, instance + method immortal when shared\"),\n    \"bound_c\": (_bound_c, lambda: 2, \"bound builtin method {}.get\"),\n}\n\n\ndef worker(name, shared_obj, seconds, barrier, rates, slot):\n    factory, argf, _ = VARIANTS[name]\n    target = shared_obj if shared_obj is not None else factory(False)\n    arg = argf()\n    clock = time.perf_counter\n    opcall = operator.call\n    barrier.wait()\n    calls = 0\n    start = clock()\n    end = start + seconds\n    if name == \"partial_opcall\":\n        while True:\n            for _ in range(1000):\n                opcall(target, arg)\n            calls += 1000\n            now = clock()\n            if now >= end:\n                break\n    else:\n        while True:\n            for _ in range(1000):\n                target(arg)\n            calls += 1000\n            now = clock()\n            if now >= end:\n                break\n    rates[slot] = calls / (now - start)\n\n\ndef measure(name, shared, threads, seconds):\n    factory = VARIANTS[name][0]\n    shared_obj = factory(True) if shared else None\n    rates = [0.0] * threads\n    barrier = threading.Barrier(threads)\n    ts = [threading.Thread(target=worker, args=(name, shared_obj, seconds, barrier, rates, i))\n          for i in range(threads)]\n    for t in ts:\n        t.start()\n    for t in ts:\n        t.join()\n    return sum(rates)\n\n\ndef main():\n    args = sys.argv[1:]\n    seconds, threads = 1.0, [1, 24]\n    if \"--seconds\" in args:\n        i = args.index(\"--seconds\")\n        seconds = float(args[i + 1])\n        del args[i:i + 2]\n    if \"--threads\" in args:\n        i = args.index(\"--threads\")\n        threads = [int(x) for x in args[i + 1].split(\",\")]\n        del args[i:i + 2]\n    if args == [\"LIST\"]:\n        for k, v in VARIANTS.items():\n            print(f\"{k:20} {v[2]}\")\n        return\n    names = list(VARIANTS) if args == [\"all\"] else args\n    ver = f\"{sys.version_info.major}.{sys.version_info.minor}\"\n    print(sys.version.replace(\"\\n\", \" \"), \"| cpus:\", os.cpu_count(),\n          \"| GIL:\", sys._is_gil_enabled())\n    print(f\"{'variant':20} {'shape':6} \" + \" \".join(f\"{n:>3}T calls/s\" for n in threads)\n          + \"  speedup\")\n    out_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), f\"results_{ver}.jsonl\")\n    with open(out_path, \"a\") as out:\n        for name in names:\n            for shape in (\"shared\", \"own\"):\n                rates = [measure(name, shape == \"shared\", n, seconds) for n in threads]\n                speed = rates[-1] / rates[0]\n                print(f\"{name:20} {shape:6} \" + \" \".join(f\"{r:14,.0f}\" for r in rates)\n                      + f\"  {speed:6.2f}x\", flush=True)\n                out.write(json.dumps({\"py\": ver, \"variant\": name, \"shape\": shape,\n                                      \"threads\": threads, \"rates\": rates,\n                                      \"speedup\": speed, \"seconds\": seconds}) + \"\\n\")\n\n\nif __name__ == \"__main__\":\n    main()\n```\n\n## How we found it\n\nA free-threaded web framework runs 24 worker threads that share module-level\ncallables. Its per-workload scale-out ledger (1/8/24 threads against a\nno-shared-state control) showed every path that called a long-lived\n`functools.partial` dropping from \"scales\" on 3.14 to below 1x on 3.15. The\nframework's workaround was to replace every long-lived shared partial with a\nclosure whose cells are pinned at construction.\n\n=========================\n\n\n### CPython versions tested on:\n\n3.15\n\n### Operating systems tested on:\n\nmacOS","author":{"url":"https://github.com/mattsta","@type":"Person","name":"mattsta"},"datePublished":"2026-10-10T23:07:45Z","url":"https://github.com/python/cpython/issues/159157"}