René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Segfault/UB from expat when re-entering the XML Parser","articleBody":"# Crash report\n\n### What happened?\n\nSemi-reliable(Interstingly, python in macos doesn't segfault for me, but docker/linux aarch64 does reliably) crash with this code:\n\n```python\nfrom xml.parsers import expat\n\np = expat.ParserCreate(encoding=\"utf-16\")\n\ndef start(name, attrs):\n    p.CharacterDataHandler = lambda data: p.Parse(data, 0)\n\np.StartElementHandler = start\n\ndata = b\"\\xff\\xfe\u003c\\x00a\\x00\u003e\\x00x\\x00\"\nfor i in range(len(data)):\n    try:\n        p.Parse(data[i:i+1], i == len(data) - 1)\n    except Exception:\n        pass\n```\n\nThis code /is/ doing some pretty naughty stuff, but the main problem seems to be that the handler is being set to re-enter the parser.  The expat docs do say:\n\n\u003e To state the obvious: the three parsing functions [XML_Parse](https://libexpat.github.io/doc/api/latest/#XML_Parse), [XML_ParseBuffer](https://libexpat.github.io/doc/api/latest/#XML_ParseBuffer) and [XML_GetBuffer](https://libexpat.github.io/doc/api/latest/#XML_GetBuffer) must not be called from within a handler unless they operate on a separate parser instance, that is, one that did not call the handler. For example, it is OK to call the parsing functions from within an XML_ExternalEntityRefHandler, if they apply to the parser created by [XML_ExternalEntityParserCreate](https://libexpat.github.io/doc/api/latest/#XML_ExternalEntityParserCreate).\n\nand I see that the python expat parser code tracks in_callback:\nhttps://github.com/python/cpython/blob/52c01864c4778a351e5aa3584e86ed6fd212a5a4/Modules/pyexpat.c#L91\n\nSo I wonder if we can avoid the segfault by preventing Parse calls when in_callback==true?\n\nThere's also a secondary issue in play here, that Parse() seems to call \n\nXML_SetEncoding\nhttps://github.com/python/cpython/blob/52c01864c4778a351e5aa3584e86ed6fd212a5a4/Modules/pyexpat.c#L872\n\nWithout the check outlined in the expat docs:\n\n\u003e Set the encoding to be used by the parser. It is equivalent to passing a non-NULL encoding argument to the parser creation functions. It must not be called after [XML_Parse](https://libexpat.github.io/doc/api/latest/#XML_Parse) or [XML_ParseBuffer](https://libexpat.github.io/doc/api/latest/#XML_ParseBuffer) have been called on the given parser. Returns XML_STATUS_OK on success or XML_STATUS_ERROR on error.\n\nThis is almost definitely not going to cause issues unless the encoding is actually changing, (not that common) at which point the UB will rear its head as the internal state of the parser becomes inconsistent.\n\n\u003cdetails\u003e\n\u003csummary\u003eDockerfile reproducer\u003c/summary\u003e\n\n```\nARG REPO=https://github.com/python/cpython.git\nARG BRANCH=main\n\nFROM ubuntu:24.04\n\nARG REPO\nARG BRANCH\nENV DEBIAN_FRONTEND=noninteractive\n\nRUN apt-get update \u0026\u0026 apt-get install -y \\\n    build-essential git pkg-config \\\n    libssl-dev libbz2-dev libreadline-dev libsqlite3-dev \\\n    liblzma-dev libffi-dev zlib1g-dev uuid-dev \\\n    \u0026\u0026 rm -rf /var/lib/apt/lists/*\n\nRUN git clone --branch ${BRANCH} --depth 1 \\\n    ${REPO} /cpython\n\nRUN cd /cpython \u0026\u0026 \\\n    ./configure --prefix=/python --without-ensurepip \u0026\u0026 \\\n    make -j$(nproc) \u0026\u0026 \\\n    make install\n\n# ── TEST SCRIPT ──────────────────────────────────────────────────\nRUN cat \u003e /test.py \u003c\u003c 'EOF'\nfrom xml.parsers import expat\n\np = expat.ParserCreate(encoding=\"utf-16\")\n\ndef start(name, attrs):\n    p.CharacterDataHandler = lambda data: p.Parse(data, 0)\n\np.StartElementHandler = start\n\ndata = b\"\\xff\\xfe\u003c\\x00a\\x00\u003e\\x00x\\x00\"\nfor i in range(len(data)):\n    try:\n        p.Parse(data[i:i+1], i == len(data) - 1)\n    except Exception:\n        pass\nEOF\n# ──────────────────────────────────────────────────────────────────────────────\n\nCMD [\"/bin/sh\", \"-c\", \"uname -m \u0026\u0026 /python/bin/python3 -VV \u0026\u0026 /python/bin/python3 /test.py\"]\n```\n\n\u003c/details\u003e\n\nGives on my pc:\n```\ndocker run --rm -it expattest\naarch64\nPython 3.15.0a7+ (heads/main:52c0186, Mar 19 2026, 13:06:19) [GCC 13.3.0]\n52c01864c4778a351e5aa3584e86ed6fd212a5a4\nSegmentation fault (core dumped)\n```\n\n### CPython versions tested on:\n\nCPython main branch\n\n### Operating systems tested on:\n\nmacOS\n\n### Output from running 'python -VV' on the command line:\n\nPython 3.15.0a7+ (heads/main:52c0186, Mar 19 2026, 13:06:19) [GCC 13.3.0]\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-146179\n* gh-146566\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/stestagg","@type":"Person","name":"stestagg"},"datePublished":"2026-03-19T13:28:29.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":6},"url":"https://github.com/146169/cpython/issues/146169"}