René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Use-after-free when a decode hook re-enters `feed()`","articleBody":"Reported by @oakkaya\n\n---\n\nIf a decode hook (`object_hook` / `object_pairs_hook` / `list_hook` / `ext_hook`) calls `.feed()`\non the **same** `Unpacker` while it is still unpacking, `append_buffer()` may reallocate\n(`PyMem_Free`) the internal buffer that the in-progress `unpack_execute()` is still reading from.\nWhen the hook returns, the parser keeps reading the remaining bytes from the freed buffer →\nuse-after-free. On a stock build this is a hard crash (SIGSEGV).\n\nThe application supplies the (re-entrant) hook; the attacker controls the bytes, which decide when\nthe hook fires and how large the re-entrant feed grows the buffer.\n\n## Reproduction\n```python\nimport struct\nfrom msgpack import Unpacker\n\nup = None\ndef ext_hook(code, data):\n    # re-entrant feed on the SAME unpacker, large enough to force a buffer realloc\n    up.feed(b\"\\xc0\" * 8_000_000)\n    return 0\n\nup = Unpacker(ext_hook=ext_hook, max_buffer_size=64 * 1024 * 1024)\n# array(200): [ ext (fires the re-entrant hook), then 199 more elements ]\nup.feed(b\"\\xdc\" + struct.pack(\"\u003eH\", 200) + b\"\\xd4\\x05A\" + b\"\\x2a\" * 199)\nfor _ in up:           # SIGSEGV\n    pass\n```\nUnder ASan:\n```\nERROR: AddressSanitizer: heap-use-after-free  READ of size 1\n    #0 unpack_execute msgpack/unpack_template.h:162\nfreed by: PyMem_Free  \u003c-  Unpacker.feed -\u003e append_buffer\n```\n\n## Root cause\n`Unpacker._unpack` runs the parser over the internal buffer:\n```python\n    ret = execute(\u0026self.ctx, self.buf, self.buf_tail, \u0026self.buf_head)\n```\n`unpack_execute` keeps local `p` / `pe` pointers into `self.buf`. A decode hook is invoked from\n*inside* `execute` (at map/array end or for ext). If the hook calls `up.feed(...)`,\n`append_buffer` (`_unpacker.pyx`) reallocates the buffer:\n```c\n    new_buf = \u003cchar*\u003ePyMem_Malloc(new_size)\n    ...\n    memcpy(new_buf, buf + head, tail - head)\n    PyMem_Free(buf)            # \u003c-- frees the buffer the outer execute() is reading\n```\nAfter the hook returns, `unpack_execute` continues reading from the now-freed `p`/`pe`.\n\n(The same applies to a `file_like.read()` that re-enters `feed()`/`unpack()`; the unpacker is not\nre-entrant but does not guard against it.)\n\n## Suggested fix (verified)\nAdd a re-entrancy guard so a buffer-mutating call during an active parse fails cleanly instead of\ncorrupting memory. Set a flag around the `execute(...)` call and reject `feed()` while it is set:\n```python\n    # field:  cdef bint _in_exec      (init False in __init__)\n\n    def feed(self, next_bytes):\n        ...\n        if self._in_exec:\n            raise RuntimeError(\"Unpacker.feed() called re-entrantly during unpacking\")\n        ...\n\n    # in _unpack, around the execute call:\n                self._in_exec = True\n                try:\n                    ret = execute(\u0026self.ctx, self.buf, self.buf_tail, \u0026self.buf_head)\n                finally:\n                    self._in_exec = False\n```\nVerified: the PoC now raises `RuntimeError` instead of crashing (clean under ASan), and normal\nstreaming (`feed()` between objects), iteration, and the `file_like` path are unaffected\n(`read_from_file` calls `append_buffer` *between* `execute()` calls, where the flag is not set, so\nthere is no false positive). A broader guard could also reject re-entrant `unpack()`/`skip()`.\n","author":{"url":"https://github.com/methane","@type":"Person","name":"methane"},"datePublished":"2026-06-16T11:34:28.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/695/msgpack-python/issues/695"}