René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Segfault from c stack overflow when under memory pressure","articleBody":"# Crash report\n\n### What happened?\n\nOk, this is hard to reproduce reliably, and may not even be fixable, but I'm reporting for completeness here.\n\nIf the kernel tries to allocate more python c stack space when there is extreme memory pressure, the allocation may fail, but the python c stack guard still believes there's plenty of space, resulting in stack overflows.\n\nIt's a bit tricky to reliably reproduce, because you need to avoid normal allocation failures, which unwind the stack, and avoid the python stack overflows, so it's triggered where you have a high c frame to python frame ratio, and only small object allocations to dodge. (I found this with fuzzing deeply nested imports, but this reproducer below is more reliable)\n\nI have code that _should_ reproduce, but you may need to tweak the  PRESSURE_START/STOP_KIB constants depending on the exact stack/memory setup:\n\n\u003cdetails\u003e\n\u003csummary\u003erepro2.py\u003c/summary\u003e\n\n```python\nimport faulthandler\nimport os\nimport resource\n\n\nAS_LIMIT_MIB = int(os.environ.get(\"AS_LIMIT_MIB\", \"512\"))\nPRESSURE_START_KIB = int(os.environ.get(\"PRESSURE_START_KIB\", \"506817\"))\nPRESSURE_STOP_KIB = int(os.environ.get(\"PRESSURE_STOP_KIB\", \"506832\"))\nSPARE_KIB = 256\nTUPLE_DEPTH = 5000\n\nPRESSURE_CHUNKS = (\n    16 * 1024 * 1024,\n    4 * 1024 * 1024,\n    1024 * 1024,\n    256 * 1024,\n    64 * 1024,\n    16 * 1024,\n    4096,\n)\n\n\ndef nested(depth):\n    obj = 1\n    for _ in range(depth):\n        obj = (obj,)\n    return obj\n\n\nfaulthandler.enable(all_threads=True)\n\n# Build the objects before applying memory pressure, so the comparison is the\n# first deep native-stack operation after the spare address space is released.\nleft = nested(TUPLE_DEPTH)\nright = nested(TUPLE_DEPTH)\n\nas_limit = AS_LIMIT_MIB * 1024 * 1024\nresource.setrlimit(resource.RLIMIT_AS, (as_limit, as_limit))\n\nfor pressure_kib in range(PRESSURE_START_KIB, PRESSURE_STOP_KIB + 1, 1):\n    spare = bytearray(SPARE_KIB * 1024)\n    pressure = []\n    remaining = pressure_kib * 1024\n\n    try:\n        for size in PRESSURE_CHUNKS:\n            while remaining \u003e= size:\n                pressure.append(bytearray(size))\n                remaining -= size\n    except MemoryError:\n        print(f\"{pressure_kib}KiB: MemoryError before compare\", flush=True)\n        break\n\n    del spare\n    print(f\"{pressure_kib}KiB:\", flush=True)\n    print(left == right, flush=True)\n\n```\n\u003c/details\u003e\n\nAnd a dockerfile that reproduces it on aarch64 (for me):\n\n\u003cdetails\u003e\n\u003csummary\u003eDockerfile\u003c/summary\u003e\n\n```docker\nFROM ubuntu:latest\n\nARG CPYTHON_REF=main\nARG DEBIAN_FRONTEND=noninteractive\n\nRUN apt-get update \u0026\u0026 apt-get install -y --no-install-recommends \\\n        build-essential \\\n        ca-certificates \\\n        clang \\\n        git \\\n        libbz2-dev \\\n        libffi-dev \\\n        libgdbm-dev \\\n        liblzma-dev \\\n        libncurses-dev \\\n        libreadline-dev \\\n        libsqlite3-dev \\\n        libssl-dev \\\n        libxml2-dev \\\n        libxslt1-dev \\\n        libzstd-dev \\\n        make \\\n        pkg-config \\\n        python3 \\\n        tk-dev \\\n        zlib1g-dev \u0026\u0026 \\\n    rm -rf /var/lib/apt/lists/*\n\nRUN git clone https://github.com/python/cpython /src/cpython \u0026\u0026 \\\n    cd /src/cpython \u0026\u0026 \\\n    git checkout --detach \"$CPYTHON_REF\"\n\nRUN cd /src/cpython \u0026\u0026 \\\n    ./configure --prefix=/opt/cpython --without-ensurepip \u0026\u0026 \\\n    make -j\"$(nproc)\" \u0026\u0026 \\\n    make install\n\nCOPY repro2.py /src/repro2.py\n\nENV AS_LIMIT_MIB=512\nENV PRESSURE_START_KIB=506817\nENV PRESSURE_STOP_KIB=506832\n\nENV PYTHONUNBUFFERED=1\n\n# CMD [\"/opt/cpython/bin/python3\", \"/src/repro2.py\"]\nCMD [\"/bin/sh\", \"-c\", \"set -xe \u0026\u0026 while true; do /opt/cpython/bin/python3 /src/repro2.py; sleep 0.2; done\"]\n\n```\n\n\u003c/details\u003e\n\n\nHere's LLDB indicating that the stack pointer is causing the fault (address=0xffffffeebfc0)\n, and yet the python c stack guard is still about 6MB below us: \n\n\u003cdetails\u003e\n\u003csummary\u003e LLDB session \u003c/summary\u003e\n\n```\n(lldb) thread info\nthread #1: tid = 101, 0x0000aaaaaafeb430 python3.16`PyObject_RichCompare(v=0x0000fffff79d3d40, w=0x0000fffff7a35e00, op=2) at object.c:1100, name = 'python3', stop reason = signal SIGSEGV: address not mapped to object (fault address=0xffffffeebfc0)\n\n(lldb) register read sp fp pc\n      sp = 0x0000ffffffeec020\n      fp = 0x0000ffffffeec020\n      pc = 0x0000aaaaaafeb430  python3.16`PyObject_RichCompare at object.c:1100\n(lldb) memory region $sp\n[0x0000ffffffeeb000-0x0001000000000000) rw- [stack]\n(lldb) script import lldb, os; p=lldb.debugger.GetSelectedTarget().process.GetProcessID(); print(\"\".join(l for l in open(f\"/proc/{p}/maps\") if \"[stack]\" in l))\nffffffeeb000-1000000000000 rw-p 00000000 00:00 0                         [stack]\n\n(lldb) bt 20\n* thread #1, name = 'python3', stop reason = signal SIGSEGV: address not mapped to object (fault address=0xffffffeebfc0)\n  * frame #0: 0x0000aaaaaafeb430 python3.16`PyObject_RichCompare(v=0x0000fffff79d3d40, w=0x0000fffff7a35e00, op=2) at object.c:1100\n    frame #1: 0x0000aaaaaafeafac python3.16`PyObject_RichCompareBool(v=0x0000fffff79d3d40, w=0x0000fffff7a35e00, op=2) at object.c:1135:11\n    frame #2: 0x0000aaaaab05ac80 python3.16`tuple_richcompare(v=0x0000fffff79d3d90, w=0x0000fffff7a35e50, op=2) at tupleobject.c:755:17\n    frame #3: 0x0000aaaaaafeb678 python3.16`do_richcompare(tstate=0x0000aaaaab802580, v=0x0000fffff79d3d90, w=0x0000fffff7a35e50, op=2) at object.c:1064:15 [inlined]\n    frame #4: 0x0000aaaaaafeb594 python3.16`PyObject_RichCompare(v=0x0000fffff79d3d90, w=0x0000fffff7a35e50, op=2) at object.c:1113:21\n    frame #5: 0x0000aaaaaafeafac python3.16`PyObject_RichCompareBool(v=0x0000fffff79d3d90, w=0x0000fffff7a35e50, op=2) at object.c:1135:11\n    frame #6: 0x0000aaaaab05ac80 python3.16`tuple_richcompare(v=0x0000fffff79d3de0, w=0x0000fffff7a35ea0, op=2) at tupleobject.c:755:17\n    frame #7: 0x0000aaaaaafeb678 python3.16`do_richcompare(tstate=0x0000aaaaab802580, v=0x0000fffff79d3de0, w=0x0000fffff7a35ea0, op=2) at object.c:1064:15 [inlined]\n    frame #8: 0x0000aaaaaafeb594 python3.16`PyObject_RichCompare(v=0x0000fffff79d3de0, w=0x0000fffff7a35ea0, op=2) at object.c:1113:21\n    frame #9: 0x0000aaaaaafeafac python3.16`PyObject_RichCompareBool(v=0x0000fffff79d3de0, w=0x0000fffff7a35ea0, op=2) at object.c:1135:11\n    frame #10: 0x0000aaaaab05ac80 python3.16`tuple_richcompare(v=0x0000fffff79d3e30, w=0x0000fffff7a35ef0, op=2) at tupleobject.c:755:17\n    frame #11: 0x0000aaaaaafeb678 python3.16`do_richcompare(tstate=0x0000aaaaab802580, v=0x0000fffff79d3e30, w=0x0000fffff7a35ef0, op=2) at object.c:1064:15 [inlined]\n    frame #12: 0x0000aaaaaafeb594 python3.16`PyObject_RichCompare(v=0x0000fffff79d3e30, w=0x0000fffff7a35ef0, op=2) at object.c:1113:21\n    frame #13: 0x0000aaaaaafeafac python3.16`PyObject_RichCompareBool(v=0x0000fffff79d3e30, w=0x0000fffff7a35ef0, op=2) at object.c:1135:11\n    frame #14: 0x0000aaaaab05ac80 python3.16`tuple_richcompare(v=0x0000fffff79d3e80, w=0x0000fffff7a35f40, op=2) at tupleobject.c:755:17\n    frame #15: 0x0000aaaaaafeb678 python3.16`do_richcompare(tstate=0x0000aaaaab802580, v=0x0000fffff79d3e80, w=0x0000fffff7a35f40, op=2) at object.c:1064:15 [inlined]\n    frame #16: 0x0000aaaaaafeb594 python3.16`PyObject_RichCompare(v=0x0000fffff79d3e80, w=0x0000fffff7a35f40, op=2) at object.c:1113:21\n    frame #17: 0x0000aaaaaafeafac python3.16`PyObject_RichCompareBool(v=0x0000fffff79d3e80, w=0x0000fffff7a35f40, op=2) at object.c:1135:11\n    frame #18: 0x0000aaaaab05ac80 python3.16`tuple_richcompare(v=0x0000fffff79d3ed0, w=0x0000fffff7a35f90, op=2) at tupleobject.c:755:17\n    frame #19: 0x0000aaaaaafeb678 python3.16`do_richcompare(tstate=0x0000aaaaab802580, v=0x0000fffff79d3ed0, w=0x0000fffff7a35f90, op=2) at object.c:1064:15 [inlined]\n\n(lldb) p ((_PyThreadStateImpl*)tstate)-\u003ec_stack_top\n(uintptr_t) 281474976706560\n(lldb) p/x ((_PyThreadStateImpl*)tstate)-\u003ec_stack_top\n(uintptr_t) 0x0000fffffffff000\n(lldb) p/x ((_PyThreadStateImpl*)tstate)-\u003ec_stack_soft_limit\n(uintptr_t) 0x0000ffffff810000\n(lldb) p/x ((_PyThreadStateImpl*)tstate)-\u003ec_stack_hard_limit\n(uintptr_t) 0x0000ffffff808000\n\n(lldb) register read sp\n      sp = 0x0000ffffffeec220\n(lldb) p/d (uintptr_t)$sp - ((_PyThreadStateImpl*)tstate)-\u003ec_stack_hard_limit\n(uintptr_t) 7225888\n(lldb) p/x (uintptr_t)$sp - ((_PyThreadStateImpl*)tstate)-\u003ec_stack_hard_limit\n(uintptr_t) 0x00000000006e4220\n```\n\n\u003c/details\u003e\n\n\nAs I mentioned at the top, I don't even know how you'd defend against this, but we have a record now.\n\nNote: this also works at up to 50gb address space, if you alter the limits accordingly, so while 512mb is quite small, this does apply generally.\n\n**AI statement**: The crash was identified using AFL++ normal fuzzing, I used chatgpt and claude to streamline the debugging and reproducer development, but all findings were double-checked, reproduced, tested, and verified by me within my (somewhat limited) knowledge of python internals. If you have questions about this, feel free to ask.\n\n### CPython versions tested on:\n\nCPython main branch\n\n### Operating systems tested on:\n\nLinux\n\n### Output from running 'python -VV' on the command line:\n\nPython 3.16.0a0 (heads/main:540b3d0a7fa, Jun  1 2026, 10:44:26) [GCC 15.2.0]","author":{"url":"https://github.com/stestagg","@type":"Person","name":"stestagg"},"datePublished":"2026-06-01T15:29:38.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/150722/cpython/issues/150722"}