René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Double free / use-after-free in `list.append()` when the list grows under `MemoryError` (`_CALL_LIST_APPEND`)","articleBody":"# Crash report\n\n### What happened?\n\nWhen `list.append(x)` has to grow the list's backing array and that allocation fails (i.e.\nunder `MemoryError`), the appended item `x` is decref'd **twice**. If `x` is referenced\nelsewhere this is a use-after-free: the interpreter aborts with `_Py_NegativeRefcount` on a\ndebug build, or **segfaults** on a release build, instead of raising a recoverable\n`MemoryError`.\n\nThis is reachable under genuine memory pressure (a real `RLIMIT_AS` reproducer with no test\nAPI is included below), so a program that correctly catches `MemoryError` can still be left\nwith a corrupted interpreter.\n\n### Reproducer\n\nDeterministic, pure Python, using `_testcapi.set_nomemory` to fail the grow allocation at a\ncontrolled point:\n\n```python\nimport _testcapi\n\nclass C:\n    __slots__ = (\"ref\",)\n    def __init__(self, ref):\n        self.ref = ref\n\ndef fill():\n    items = [C(str(i) + \"_unique\") for i in range(200)]\n    out = []\n    for it in items:\n        out.append(it.ref)          # CALL_LIST_APPEND; it.ref is also held by the C instance\n\nfill()                              # warm up: specialize out.append(...) to CALL_LIST_APPEND\nfor start in range(1500):\n    _testcapi.set_nomemory(start, start + 1)   # fail one allocation, then resume\n    try:\n        try:\n            fill()\n        finally:\n            _testcapi.remove_mem_hooks()\n    except BaseException:\n        pass\n```\n\nOn a `--with-pydebug` build this aborts:\n\n```\n./Include/refcount.h:520: _Py_NegativeRefcount: Assertion failed: object has negative ref count\nFatal Python error: _PyObject_AssertFailed\n```\n\nOn a release build it segfaults.\n\n### Without any test API (real `MemoryError`)\n\nThe same double-free fires under a genuine allocation failure. With an `RLIMIT_AS` cap so the\nlist's grow allocation returns NULL naturally (run on a non-ASan build):\n\n```python\nimport resource\n\npool = [object() for _ in range(8_000_000)]      # uniquely-referenced items, built before the cap\nwarm = []\nfor i in range(3000):\n    warm.append(pool[i])                          # specialize CALL_LIST_APPEND\ndel warm\n\ncur = int(open(\"/proc/self/statm\").read().split()[0]) * 4096   # current virtual size\nresource.setrlimit(resource.RLIMIT_AS, (cur + 24 * 1024 * 1024,) * 2)\n\nout = []\nfor x in pool:\n    out.append(x)                                 # real list_resize failure -\u003e double-free -\u003e SIGSEGV\n```\n\n```\n$ ./python natural.py\nSegmentation fault            # faulthandler pins the crash to the `out.append(x)` line\n```\n\nUnder the same cap, when the failing allocation is *not* a list-append grow (e.g. appending\nlarge `bytes`), Python raises a clean, catchable `MemoryError` and does not crash — so the\nsegfault is specific to the buggy append path.\n\n## Root cause\n\nIn the specialized append bytecode `_CALL_LIST_APPEND` (`Python/bytecodes.c`):\n\n```c\nop(_CALL_LIST_APPEND, (callable, self, arg -- none, c, s)) {\n    ...\n    int err = _PyList_AppendTakeRef((PyListObject *)self_o, PyStackRef_AsPyObjectSteal(arg));\n    UNLOCK_OBJECT(self_o);\n    if (err) {\n        ERROR_NO_POP();\n    }\n    ...\n}\n```\n\n`arg` is **stolen** via `PyStackRef_AsPyObjectSteal` and handed to `_PyList_AppendTakeRef`,\nwhich consumes the reference on every path — including decref'ing the item when the grow\nfails (`_PyList_AppendTakeRefListResize` → `if (list_resize(...) \u003c 0) { Py_DECREF(newitem);\nreturn -1; }`, `Objects/listobject.c`).\n\nBut on that failure the uop takes `ERROR_NO_POP()`, which jumps to exception handling\n**without removing `arg` from the value stack**. Since `arg`'s reference was already\nconsumed, the stale `arg` stackref is now dangling. The eval loop's `exception_unwind` then\npops the frame's value stack and `PyStackRef_XCLOSE`s every slot, closing the stale `arg`\nslot — a **second** decref of the item.\n\n(Confirmed with ASan on a `--with-pymalloc` build: the item is freed by `PyStackRef_XCLOSE`\n← `_PyEval_EvalFrameDefault` (the `exception_unwind` handler); both the item's allocation and\nthe second, use-after-free decref are visible in the report.)\n\n## Suggested fix\n\n`_CALL_LIST_APPEND` must account for the consumed `arg` on the error path — once\n`_PyList_AppendTakeRef` has taken the reference, the `arg` stackref is dead and must not be\nleft on the value stack for `exception_unwind` to close. The sibling ops already show the two\ncorrect idioms:\n\n- the comprehension element-adds (`LIST_APPEND`, `SET_ADD`, `MAP_ADD`) call the same kind of\n  steal/`*TakeRef` helper but use `ERROR_IF(...)`, so the codegen drops the consumed input on\n  the error path. Concretely, `[x for x in ...]` (`LIST_APPEND`, the *same*\n  `_PyList_AppendTakeRef` helper) does **not** crash where `lst.append(x)`\n  (`_CALL_LIST_APPEND`) does;\n- the consuming call ops (`_DO_CALL_FUNCTION_EX`, `_PY_FRAME_EX`) call `INPUTS_DEAD();\n  SYNC_SP();` before `ERROR_NO_POP()`.\n\nI audited the other specialized ops: `_CALL_LIST_APPEND` is the only one that steals a\nstack input and then takes a bare `ERROR_NO_POP()` without either form of accounting, which\nis why it is the lone op affected.\n\n## Environment\n\n- CPython `main` (3.16.0a0); reproduced on `--with-pydebug` builds (abort) and release builds\n  (segfault), both free-threaded and default GIL.\n\n---\n\n*This report and the reduced reproducers were drafted with the assistance of Claude Code; I\nhave reviewed and reproduced them.*\n\n### CPython versions tested on:\n\nCPython main branch, 3.16\n\n### Operating systems tested on:\n\nLinux\n\n### Output from running 'python -VV' on the command line:\n\nPython 3.16.0a0 (heads/main:aec0aed1978, Jun 20 2026, 17:44:00) [Clang 22.1.2 (1ubuntu1)]\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-151826\n* gh-152114\n* gh-152117\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/devdanzin","@type":"Person","name":"devdanzin"},"datePublished":"2026-06-20T22:13:16.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":6},"url":"https://github.com/151818/cpython/issues/151818"}