René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Data race on frozendict reads during construction in the free-threading build","articleBody":"# Data race on `frozendict` reads during construction in the free-threading build\n\n## Summary\n\nIn the free-threading build (`--disable-gil`), several `frozendict` (PEP 814)\nread paths access the object's internal state without synchronization, on the\nassumption that a `frozendict` is immutable and therefore safe to read\nlock-free. That assumption holds once the object is fully built, but **not while\nit is being constructed from a user-defined mapping**: the half-built\n`frozendict` is already observable from other threads, and its scalar fields and\nentry table are being mutated concurrently. ThreadSanitizer reports a genuine\ndata race, and a reader can observe the object's length and contents change\nunderneath it, which violates the immutability the type is supposed to\nguarantee.\n\nAffected read paths: `len(fd)` (`frozendict_length`), `repr(fd)`\n(`frozendict_repr`), `hash(fd)` (`frozendict_hash`). All three read raw\n`ma_used` and/or walk the entry table via `_PyDict_Next` with no atomic load and\nno critical section, while the corresponding mutable-`dict` paths\n(`dict_length`, `dict_repr`) use `GET_USED()` / `Py_BEGIN_CRITICAL_SECTION`.\n\n## How a half-built `frozendict` becomes observable\n\n`frozendict(mapping)` for a non-`dict` mapping takes the generic path\n`frozendict_new` → `dict_update_common` → `dict_merge`, which calls back into\nuser Python: `PyMapping_Keys(b)`, then `PyObject_GetItem(b, key)` per key\n(`Objects/dictobject.c`, the generic branch of `dict_merge`). `dict_new` already\ncalls `_PyObject_GC_TRACK(d)` *before the first item is inserted*, so from the\nvery first callback the half-built `frozendict` is discoverable via\n`gc.get_objects()`. A `__getitem__` callback can take a second reference to it\nand hand a live alias to another thread, which then reads the same object while\n`dict_merge` keeps inserting and resizing it.\n\n`can_modify_dict()` treats a `frozendict` as mutable only while it is uniquely\nreferenced (`PyUnstable_Object_IsUniquelyReferenced(mp)`), i.e. it assumes a\nhalf-built `frozendict` is private. Once user code has run during construction,\nthat assumption is false.\n\n## ThreadSanitizer evidence (main HEAD `9688d252d330b0b586760a121ee8c8f7215176e8`)\n\nBuild: `./configure --disable-gil --with-thread-sanitizer \u0026\u0026 make`.\n\n`len` read racing the construction-time size store:\n\n```\nWARNING: ThreadSanitizer: data race\n  Read of size 8 ... by thread T1:\n    #0 frozendict_length Objects/dictobject.c\n    #1 PyObject_Size / builtin_len\n  Previous atomic write of size 8 ... by main thread:\n    #0 insertdict Objects/dictobject.c            (STORE_USED, already atomic)\n    #1 setitem_take2_lock_held\n    #2 dict_merge\n    #3 frozendict_vectorcall\nSUMMARY: ThreadSanitizer: data race in frozendict_length\n```\n\n`repr` and `hash` additionally race the entry table while a concurrent\n`dictresize` reallocates it:\n\n```\nWARNING: ThreadSanitizer: data race\n  ... by main thread:\n    #0 dictresize Objects/dictobject.c\n    #1 insert_combined_dict\n    #2 insertdict\n    #3 dict_merge -\u003e frozendict_vectorcall\n  Previous read of size 8 ... by thread T1:\n    #0 _PyDict_Next          (reached from frozendict_repr / frozendict_hash)\nSUMMARY: ThreadSanitizer: data race in dictresize\n```\n\nBeyond the sanitizer report, the violation is directly observable: a reader\nwatching `len()` of the half-built object sees it grow `0 → 499 → 999 → 1899`,\nand `gc.is_tracked()` is `True` at every step.\n\nThe construction (write) side is already atomic — `insertdict`'s size update\nuses `STORE_USED` (`FT_ATOMIC_STORE_SSIZE_RELAXED`) and the inserts run under\n`Py_BEGIN_CRITICAL_SECTION(a)` on the target in `dict_merge`. The defect is on\nthe read side: the custom `frozendict` read functions deliberately dropped both\nthe atomic load and the critical section.\n\n## Reproduction\n\n```python\nimport threading, gc\n\nstop = threading.Event()\nbox = []\n\ndef reader():\n    while not stop.is_set():\n        if box:\n            try:\n                len(box[0])     # or repr(box[0]) / hash(box[0])\n            except Exception:\n                pass\n\nclass Evil:\n    def keys(self):\n        return [f\"k{i}\" for i in range(8192)]\n    def __getitem__(self, k):\n        if not box:                      # leak the half-built frozendict\n            for o in gc.get_objects():\n                if isinstance(o, frozendict) and len(o) \u003e= 1:\n                    box.append(o)\n                    break\n        return 1\n\nt = threading.Thread(target=reader, daemon=True)\nt.start()\nfor _ in range(60):\n    box.clear()\n    frozendict(Evil())\nstop.set()\nt.join(timeout=2)\n```\n\nRun under a free-threading + TSan build; the race fires for `len`, `repr` and\n`hash`.\n\n## Fix\n\n`frozendict` reads should follow the same free-threading conventions the rest of\n`dictobject.c` already uses:\n\n1. Read scalar `ma_used` through `GET_USED()`\n   (`FT_ATOMIC_LOAD_SSIZE_RELAXED`), exactly like `dict_length` and like\n   `set_len` for `frozenset`.\n2. Wrap the entry-table walk in `frozendict_repr` and `frozendict_hash` in a\n   per-object critical section, exactly like `dict_repr` already wraps\n   `anydict_repr_impl`, and like `set_repr` wraps `set_repr_lock_held` (which is\n   the `tp_repr` of both `set` and `frozenset`). The construction side already\n   holds `Py_BEGIN_CRITICAL_SECTION(a)` on the target in `dict_merge`, so the\n   reader's critical section serializes against construction and the race goes\n   away. The cached `ma_hash` short-circuit stays lock-free (atomic load),\n   matching `frozenset_hash`.\n\nWith this change, ThreadSanitizer is clean for `len`, `repr` and `hash` against\na concurrently-constructing `frozendict`, and the existing `test_dict`,\n`test_free_threading` and `test_set` suites still pass on the free-threading +\nTSan build.\n\nA PR implementing the above is ready.\n\n### One related path: lock-free `frozendict` copy (PR gh-141510 / #145920)\n\n`copy_lock_held` was deliberately made lock-free for `frozendict` sources in\n#145920 (\"A frozendict mapping is immutable so a critical section is not needed\nto ensure that a copy is consistent\"), giving a large speedup for the normal\n(finished) case. That path reads the source's `ma_used` / `ma_keys` and walks\nthe entry table, so it has the **same** latent race during construction\n(`fd.copy()` / `dict(fd)` / `fd | m` on a half-built source still trips TSan\nafter the read-side fix above). Because re-adding a critical section there would\nwalk back a deliberate, merged performance decision, I left that path as-is and\napplied only the scalar-atomic read. The clean way to also close the copy window\nwithout giving up #145920's lock-free copy is to make a half-built `frozendict`\nnon-observable in the first place — defer GC-tracking / publication until the\nconstructor returns, which would make `can_modify_dict`'s unique-reference\nassumption actually hold and keep every lock-free read correct. That is a\nbroader design decision (it would apply to `frozenset` too, which is also\nGC-tracked and observably grows during construction), so I am raising it here\nrather than bundling it into the read-path PR. Happy to follow whichever\ndirection you prefer.\n\n## Environment\n\n- CPython `main`, free-threading build\n  (`--disable-gil --with-thread-sanitizer`), reproduced on HEAD\n  `9688d252d330b0b586760a121ee8c8f7215176e8` (`3.16.0a0`).\n- Same code is present on the 3.15 branch (the `frozendict` read paths are\n  byte-for-byte the same logic).\n\n## Linked\n\n- gh-141510 (PEP 814 `frozendict`).\n- #144913 split `frozendict_length` off `dict_length` and replaced the atomic\n  read with a plain `ma_used` load.\n- #145920 made the `frozendict` copy path lock-free (`copy_lock_held`).\n\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-151724\n* gh-151740\n* gh-151954\n* gh-151967\n* gh-152021\n* gh-152067\n* gh-152225\n* gh-152230\n* gh-152271\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/tonghuaroot","@type":"Person","name":"tonghuaroot"},"datePublished":"2026-06-19T09:36:22.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/151722/cpython/issues/151722"}