René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Free-threaded (no-GIL): dumps() crashes when another thread mutates the dict being serialized (PyDict_Next iterator invalidation)","articleBody":"Ahead of the free-threading work in #225: `dumps()` walks the object being\nserialized with `PyDict_Next`, which is documented as unsafe against concurrent\nmutation. On a free-threaded interpreter, another thread structurally mutating that\ndict while `dumps()` is walking it corrupts the walk and crashes.\n\n**Latent today, not live.** The extension does not declare `Py_MOD_GIL_NOT_USED`,\nso importing it on a free-threaded build re-enables the GIL and this is safe as\nshipped. It becomes reachable when the GIL is forced off (`PYTHON_GIL=0`) — i.e.\nthe moment free-threading support (#225) lands. So this is a concrete, reproducible\ninstance of what #225 needs to address, not a bug in current default behaviour.\n\n### Site (current `master`, `rapidjson.cpp`)\n\n```cpp\nstatic inline bool all_keys_are_string(PyObject* dict) {   // 2323\n    Py_ssize_t pos = 0; PyObject* key;\n    while (PyDict_Next(dict, &pos, &key, NULL))             // walks the user dict, no copy, no lock\n        if (!PyUnicode_Check(key)) return false;\n```\n\n`dumps(d)` walks `d` here and again in `dumps_internal` (2556 / 2596). `pos` indexes\nthe dict's internal table; a structural mutation from another thread (an insert that\ntriggers a resize, or a delete) reallocates or reorders that table, so `pos` then\npoints into freed / relocated memory.\n\n### Reproducer\n\n`python3.14.0rc1t`, python-rapidjson 1.23 built from source, `PYTHON_GIL=0`. 6\nthreads call `dumps(shared)` while 3 threads structurally mutate `shared`. 10\nrounds per arm:\n\n| arm | result |\n|---|---|\n| mutate, GIL off (`PYTHON_GIL=0`) | **10/10 SIGSEGV** |\n| control — no mutator thread | clean 10/10 |\n| control — mutate a *different* dict `dumps()` never sees | clean 10/10 |\n| control — same mutation, GIL left on | clean 10/10 |\n\nThe GIL-on control staying clean is the whole point: this is a free-threading\nfault, not a generic race, and it is safe under the GIL that ships today.\n\n\u003cdetails>\u003csummary>ft_rapidjson.py\u003c/summary>\n\n```python\n\"\"\"python-rapidjson 1.23 — iterator-invalidation in dumps() over a shared dict.\n\n`rapidjson.cpp:2320` all_keys_are_string() and dumps_internal() (2553/2593) walk\nthe user dict with `PyDict_Next`, no copy and no lock:\n\n    while (PyDict_Next(dict, &pos, &key, NULL))   // 2320\n        if (!PyUnicode_Check(key)) return false;\n\n`PyDict_Next` is documented unsafe against concurrent mutation: `pos` indexes the\ndict's internal table, and a structural mutation (insert that triggers a resize,\nor a delete) reallocates or reorders that table, so a `pos` captured before the\nmutation now points into freed/relocated memory.\n\nUnder the GIL, `dumps()` never yields mid-walk, so it is safe. Under\n`Py_GIL_DISABLED`, another thread structurally mutating the same dict while\n`dumps()` walks it invalidates the iteration → crash or garbage read.\n\nThis is the iterator-invalidation class, not borrow-then-use: the corrupted thing\nis the *walk's cursor over the container*, not a single borrowed item.\n\nRun with PYTHON_GIL=0 (the module does not declare Py_MOD_GIL_NOT_USED, so import\nre-enables the GIL otherwise).\n\"\"\"\nimport os\nimport sys\nimport threading\nimport time\n\nimport rapidjson\n\nN_DUMPERS = int(os.environ.get(\"N_DUMPERS\", 6))\nN_MUTATORS = int(os.environ.get(\"N_MUTATORS\", 3))\nSIZE = int(os.environ.get(\"SIZE\", 64))\nSECONDS = float(os.environ.get(\"SECONDS\", 6))\nMUTATE = os.environ.get(\"MUTATE\", \"1\") == \"1\"\nDECOY = os.environ.get(\"DECOY\", \"0\") == \"1\"\n\n\ndef fresh(n):\n    return {f\"k{i}\": i for i in range(n)}\n\n\ndef main():\n    gil = getattr(sys, \"_is_gil_enabled\", lambda: True)()\n    print(f\"py={sys.version.split()[0]} gil={gil} \"\n          f\"dumpers={N_DUMPERS} mutators={N_MUTATORS if MUTATE else 0} \"\n          f\"decoy={DECOY}\", flush=True)\n\n    shared = fresh(SIZE)\n    decoy = fresh(SIZE)\n    stop = threading.Event()\n    barrier = threading.Barrier(N_DUMPERS + (N_MUTATORS if MUTATE else 0) + 1)\n    dumped = [0] * N_DUMPERS\n\n    def dumper(t):\n        barrier.wait()\n        n = 0\n        while not stop.is_set():\n            try:\n                rapidjson.dumps(shared)\n            except Exception:\n                pass          # value/type errors from racing data are irrelevant\n            n += 1\n        dumped[t] = n\n\n    def mutator():\n        barrier.wait()\n        target = decoy if DECOY else shared\n        i = 0\n        while not stop.is_set():\n            # STRUCTURAL mutation: grow (insert → resize) and shrink (delete),\n            # which is what invalidates the PyDict_Next cursor.\n            target[f\"x{i}\"] = i\n            if i % 3 == 0 and len(target) > SIZE:\n                try:\n                    target.pop(next(iter(target)))\n                except (KeyError, StopIteration, RuntimeError):\n                    pass\n            i += 1\n\n    threads = [threading.Thread(target=dumper, args=(t,)) for t in range(N_DUMPERS)]\n    if MUTATE:\n        threads += [threading.Thread(target=mutator) for _ in range(N_MUTATORS)]\n    for th in threads:\n        th.start()\n    barrier.wait()\n    time.sleep(SECONDS)\n    stop.set()\n    for th in threads:\n        th.join()\n    print(f\"clean: {sum(dumped)} dumps survived\", flush=True)\n    return 0\n\n\nif __name__ == \"__main__\":\n    raise SystemExit(main())\n```\n\u003c/details>\n\n### A possible fix (when #225 is worked)\n\nTake a critical section on the dict for the duration of the walk\n(`Py_BEGIN_CRITICAL_SECTION(dict)` around the `PyDict_Next` loops), or snapshot the\nkeys/items before serializing. `all_keys_are_string` and the two `dumps_internal`\nwalks are the sites.\n\n### Not claimed\n\nNo severity — reachable only with the GIL disabled, which is not the default today.\nNo exploitability — the observed faults are consistent with the mechanism; I did not\nbuild a primitive.\n","author":{"url":"https://github.com/espressolee","@type":"Person","name":"espressolee"},"datePublished":"2026-08-05T15:00:48Z","url":"https://github.com/python-rapidjson/python-rapidjson/issues/234"}