Title: Implement pyexpat character buffering and callback-safe flushes by youknowdot · Pull Request #9838 · RustPython/RustPython · GitHub
Open Graph Title: Implement pyexpat character buffering and callback-safe flushes by youknowdot · Pull Request #9838 · RustPython/RustPython
X Title: Implement pyexpat character buffering and callback-safe flushes by youknowdot · Pull Request #9838 · RustPython/RustPython
Description: Summary Implement pyexpat's Python-facing character buffering layer without changing the xml-rs dependency or manufacturing Expat-sized input chunks. Provide buffer_size and read-only buffer_used, with integer-only size validation, default capacity, same-size no-op, fallible allocation, and the documented enable/disable behavior. Accumulate character events, flushing before registered non-character callbacks and successful Parse/ParseFile completion. Flush to the old character handler before replacing it. Follow CPython 3.15.2's callback-safe ordering: detach pending data before invoking Python, release all Rust locks, and recheck handler/buffer changes after an overflow-triggered flush. Copy buffer configuration, but not pending data, to external parsers. Clear handlers and stop parsing after a callback raises. Preserve the distinct behavior of integer handler return-conversion errors, which do not perform that callback-error cleanup. Remove three expected-failure markers and add non-duplicated API/reentrancy/error regressions to the existing stdlib_pyexpat.py snippet. CPython reference CPython 3.15.2 Modules/pyexpat.c: flush_character_buffer, my_CharacterDataHandler, handler/property setters, get_parse_result, call_with_frame, and flag_error. In particular, 3.15.2 clears the used count before the callback. This matters when a callback changes the handler, buffer size, or buffering mode; the old data must not be delivered recursively. Limits intentionally retained This does not implement Expat's legacy encoding handling or streaming character-event boundaries. The iso8859/chunk-size tests and unbuffered newline-event test remain expected failures. Oversized xml-rs events are delivered whole, not arbitrarily split to make those tests pass. Existing missing error-code metadata properties also remain outside this patch. No changes to another PR's DTD implementation are included. Validation CPython 3.15.2: all newly added regression cases pass. Patched RustPython build passes. -m test test_pyexpat: 83 tests run; 16 skipped, 35 existing expected failures; no unexpected failures. Complete extra_tests/snippets/stdlib_pyexpat.py: passes. Relevant standard-library Clippy with threading and all targets: passed. Combined test_pyexpat, test_xml_etree, test_minidom, test_sax, test_pulldom, and test_plistlib: six modules successful, 714 tests run, 23 skips. Full pytest suite: 495 passed, 20 failed, 15 subtests passed (336.72s). The 20 failures match the baseline exactly: 14 host-CPython failures and 6 RustPython failures. No new failures observed. Full-workspace and C API validation were not run because of limited disk space. Implementation and investigation assisted by OpenAI Codex. CI host compatibility follow-up The first Ubuntu CI run had one failure: the host CPython 3.15.0 execution of the new reentrant-flush regression. The RustPython snippet passed. CPython 3.15.0's flush and character handler retain pending data during a callback; 3.15.2 clears it before invoking Python and rechecks buffer changes afterward. The follow-up guards only callback-mutation cases on older CPython hosts: resize, handler replacement, and overflow-triggered disable/removal. Executing those cases can recursively redeliver data or invalidate the old buffer, so guarding only the final assertions would not be safe. General exception propagation still runs everywhere; only its callback-time buffer_used == 0 assertion is version-gated. All RustPython regressions remain enabled. The final snippet was rerun successfully on actual CPython 3.15.0, CPython 3.15.2, and the previously validated RustPython binary. No Rust source changed. Upstream fix: CPython commit 0ab39af, “Handle pyexpat configuration changes during callbacks” (#160475). Summary by CodeRabbit New Features XML parsers can buffer character data with a configurable buffer size. External entity parsers inherit the parent parser’s buffering settings and handlers, but not pending text. Bug Fixes Buffer sizes now require a positive integer within the supported range; oversized text events are delivered whole. Pending character data is flushed before relevant callbacks and when buffering settings or the character-data handler change. Callback failures now end parsing and release registered handlers; subsequent parse attempts report that parsing has finished.
Open Graph Description: Summary Implement pyexpat's Python-facing character buffering layer without changing the xml-rs dependency or manufacturing Expat-sized input chunks. Provide buffer_size and read-only buffer_u...
X Description: Summary Implement pyexpat's Python-facing character buffering layer without changing the xml-rs dependency or manufacturing Expat-sized input chunks. Provide buffer_size and read-only buff...
Opengraph URL: https://github.com/RustPython/RustPython/pull/9838
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:56eedada-112c-34d7-463b-38608d8ef7ab |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | C00E:1148D3:32FBABD:4117C80:6B01DE7D |
| html-safe-nonce | 9d6366ec5246feb525643ff563443ffa4c317976b3a2f57b2e561d0d8d35fafa |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDMDBFOjExNDhEMzozMkZCQUJEOjQxMTdDODA6NkIwMURFN0QiLCJ2aXNpdG9yX2lkIjoiNTY3NzUxNTM4MDU5MjA3NDM2NSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 120c64308aff1db5b12055398c65c116efb4a5adf939a9ad1b1b0746b743d2c6 |
| hovercard-subject-tag | pull_request:5312700162 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/RustPython/RustPython/pull/9838/files |
| twitter:image | https://avatars.githubusercontent.com/u/336763669?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/336763669?s=400&v=4 |
| og:image:alt | Summary Implement pyexpat's Python-facing character buffering layer without changing the xml-rs dependency or manufacturing Expat-sized input chunks. Provide buffer_size and read-only buffer_u... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | cc5dfdd32f62ebd3223cab3af3aa2b268d3605218237326a48a99d1efd89495c |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/RustPython/RustPython git https://github.com/RustPython/RustPython.git |
| octolytics-dimension-user_id | 39710557 |
| octolytics-dimension-user_login | RustPython |
| octolytics-dimension-repository_id | 135201145 |
| octolytics-dimension-repository_nwo | RustPython/RustPython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 135201145 |
| octolytics-dimension-repository_network_root_nwo | RustPython/RustPython |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 8c77c9e8439031ad54c75a250694842892f693e1 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width