René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"`itertools._grouper.__next__` has the same re-entrant use-after-free that was fixed in `groupby.__next__`","articleBody":"# Crash report\n\n### What happened?\n\nAI Disclosure: this issue, including the reproducer code, has been drafted by an LLM.\n\n## Bug description\n\ngh-143543 / commit a91b5c3fb5a fixed a re-entrant use-after-free in `groupby_next` by snapshotting `gbo-\u003etgtkey` and `gbo-\u003ecurrkey` with `Py_INCREF` before calling `PyObject_RichCompareBool`. However, the sibling function `_grouper_next` (the inner group iterator) has the exact same unprotected comparison at `Modules/itertoolsmodule.c:681`:\n\n```c\nrcmp = PyObject_RichCompareBool(igo-\u003etgtkey, gbo-\u003ecurrkey, Py_EQ);\n```\n\nNeither `igo-\u003etgtkey` nor `gbo-\u003ecurrkey` is held with a strong reference during the comparison. A user-defined `__eq__` that re-enters the `_grouper` iterator can trigger `groupby_step`, which calls `Py_XSETREF(gbo-\u003ecurrkey, newkey)` — freeing the old `currkey` while `PyObject_RichCompareBool` still holds a dangling pointer to it as local variable `w`. When `__eq__` returns `NotImplemented`, `do_richcompare` tries the reverse comparison `w.__eq__(v)` on the freed object.\n\n## Reproducer\n\nCrashes with a segfault:\n\n```python\nimport itertools\n\ngrouper_iter = None\n\nclass Key:\n    __hash__ = None\n\n    def __init__(self, do_advance):\n        self.do_advance = do_advance\n        self.payload = bytearray(256)\n\n    def __eq__(self, other):\n        if self.do_advance:\n            self.do_advance = False\n            if grouper_iter is not None:\n                try:\n                    next(grouper_iter)\n                except StopIteration:\n                    pass\n            for _ in range(50):\n                bytearray(256)\n            return NotImplemented\n        return True\n\ndef keyfunc(element):\n    if element == 0:\n        return Key(do_advance=True)\n    return Key(do_advance=False)\n\ng = itertools.groupby(range(4), keyfunc)\nkey, grouper_iter = next(g)\nitems = list(grouper_iter)  # segfault\n```\n\n## Backtrace\n\n```\nProgram received signal SIGSEGV, Segmentation fault.\ndo_richcompare (tstate=0x555555d6cfd8 \u003c_PyRuntime+405272\u003e, v=0x20002542ba0, w=0x200025428a0, op=2) at Objects/object.c:1065\nwarning: Source file is more recent than executable.\n1065            Py_DECREF(res);\n\n#0  do_richcompare (tstate=0x555555d6cfd8 \u003c_PyRuntime+405272\u003e, v=0x20002542ba0, w=0x200025428a0, op=2) at Objects/object.c:1065\n#1  PyObject_RichCompare (v=v@entry=0x20002542ba0, w=w@entry=0x200025428a0, op=op@entry=2) at Objects/object.c:1109\n#2  0x000055555573deca in PyObject_RichCompareBool (v=0x20002542ba0, w=0x200025428a0, op=op@entry=2) at Objects/object.c:1131\n#3  0x0000555555a32c03 in _grouper_next (op=0x20002151c80) at ./Modules/itertoolsmodule.c:681\n#4  0x00005555556f2805 in list_extend_iter_lock_held (self=self@entry=0x200029bd4b0, iterable=iterable@entry=0x20002151c80) at Objects/listobject.c:1318\n#5  0x00005555556ed858 in _list_extend (self=self@entry=0x200029bd4b0, iterable=iterable@entry=0x20002151c80) at Objects/listobject.c:1507\n#6  0x00005555556f7ba8 in list___init___impl (self=self@entry=0x200029bd4b0, iterable=0x20002151c80) at Objects/listobject.c:3541\n#7  0x00005555556f0c46 in list_vectorcall (type=type@entry=0x555555cd8ef8 \u003cPyList_Type\u003e, args=args@entry=0x7fffffffa6c8, nargsf=nargsf@entry=9223372036854775809, kwnames=kwnames@entry=0x0)\n    at Objects/listobject.c:3565\n#8  0x000055555569f1de in _PyObject_VectorcallTstate (tstate=0x555555d6cfd8 \u003c_PyRuntime+405272\u003e, callable=0x555555cd8ef8 \u003cPyList_Type\u003e, args=0x7fffffffa6c8, nargsf=9223372036854775809,\n    kwnames=0x0) at ./Include/internal/pycore_call.h:136\n#9  0x0000555555869fc1 in _Py_VectorCallInstrumentation_StackRefSteal (callable=..., arguments=0x555555da57d0, total_args=total_args@entry=1, kwnames=kwnames@entry=...,\n    call_instrumentation=false, frame=0x555555da5770, this_instr=0x200023e8f3a, tstate=0x555555d6cfd8 \u003c_PyRuntime+405272\u003e) at Python/ceval.c:770\n#10 0x0000555555874b23 in _PyEval_EvalFrameDefault (tstate=0x555555d6cfd8 \u003c_PyRuntime+405272\u003e, frame=\u003coptimized out\u003e, throwflag=0) at Python/generated_cases.c.h:1842\n#11 0x0000555555869779 in _PyEval_EvalFrame (tstate=0x555555d6cfd8 \u003c_PyRuntime+405272\u003e, frame=0x555555da5770, throwflag=0) at ./Include/internal/pycore_ceval.h:118\n#12 _PyEval_Vector (tstate=tstate@entry=0x555555d6cfd8 \u003c_PyRuntime+405272\u003e, func=func@entry=0x200024c3e30, locals=locals@entry=0x2000270cbb0, args=args@entry=0x0,\n    argcount=argcount@entry=0, kwnames=kwnames@entry=0x0) at Python/ceval.c:2133\n#13 0x0000555555869292 in PyEval_EvalCode (co=co@entry=0x200023e8dd0, globals=globals@entry=0x2000270cbb0, locals=locals@entry=0x2000270cbb0) at Python/ceval.c:681\n#14 0x0000555555977e7a in run_eval_code_obj (tstate=tstate@entry=0x555555d6cfd8 \u003c_PyRuntime+405272\u003e, co=co@entry=0x200023e8dd0, globals=globals@entry=0x2000270cbb0,\n    locals=locals@entry=0x2000270cbb0) at Python/pythonrun.c:1366\n#15 0x0000555555977b39 in run_mod (mod=mod@entry=0x200027e6508, filename=filename@entry=0x20002235290, globals=globals@entry=0x2000270cbb0, locals=locals@entry=0x2000270cbb0,\n    flags=0x7fffffffb9f0, arena=arena@entry=0x200020508b0, interactive_src=0x0, generate_new_source=0) at Python/pythonrun.c:1469\n```\n\n## Root cause\n\n`_grouper_next` at line 681 passes `igo-\u003etgtkey` and `gbo-\u003ecurrkey` to `PyObject_RichCompareBool` without incrementing their refcounts. `PyObject_RichCompareBool` → `PyObject_RichCompare` → `do_richcompare` does **not** INCREF the arguments. When `__eq__` re-enters the `_grouper` iterator:\n\n1. Inner `_grouper_next` calls `groupby_step`\n2. `groupby_step` calls `Py_XSETREF(gbo-\u003ecurrkey, newkey)` — this decrefs the old `currkey` to refcount 0, freeing it\n3. `__eq__` returns `NotImplemented`\n4. `do_richcompare` tries the reverse: `w.__eq__(v)` where `w` is the freed `currkey` — **use-after-free**\n\n## Suggested fix\n\nApply the same fix from `groupby_next` (commit a91b5c3fb5a) to `_grouper_next`:\n\n```c\n// Before (vulnerable):\nrcmp = PyObject_RichCompareBool(igo-\u003etgtkey, gbo-\u003ecurrkey, Py_EQ);\n\n// After (safe):\nPyObject *tgtkey = igo-\u003etgtkey;\nPyObject *currkey = gbo-\u003ecurrkey;\nPy_INCREF(tgtkey);\nPy_INCREF(currkey);\nint rcmp = PyObject_RichCompareBool(tgtkey, currkey, Py_EQ);\nPy_DECREF(tgtkey);\nPy_DECREF(currkey);\n```\n\n## Versions affected\n\nSame as gh-143543 — all versions with `itertools.groupby`, regardless of GIL or JIT being enabled. The fix for `groupby_next` was applied to 3.13 and 3.14; this sibling function needs the same treatment, besides also hitting 3.15.\n\n**Found using [cpython-review-toolkit](https://github.com/devdanzin/cpython-review-toolkit).**\n\n### CPython versions tested on:\n\nCPython main branch, 3.15, 3.14\n\n### Operating systems tested on:\n\nLinux\n\n### Output from running 'python -VV' on the command line:\n\nPython 3.15.0a6+ free-threading build (heads/main:592e8f0865c, Mar  2 2026, 16:35:36) [Clang 21.1.2 (2ubuntu6)]\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-147962\n* gh-148010\n* gh-148011\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/devdanzin","@type":"Person","name":"devdanzin"},"datePublished":"2026-03-29T22:03:12.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":4},"url":"https://github.com/146613/cpython/issues/146613"}