Title: `zlib.decompressobj`: `unconsumed_tail` keeps stale input when a `max_length`-bounded call reaches end of stream · Issue #8906 · RustPython/RustPython · GitHub
Open Graph Title: `zlib.decompressobj`: `unconsumed_tail` keeps stale input when a `max_length`-bounded call reaches end of stream · Issue #8906 · RustPython/RustPython
X Title: `zlib.decompressobj`: `unconsumed_tail` keeps stale input when a `max_length`-bounded call reaches end of stream · Issue #8906 · RustPython/RustPython
Description: Environment RustPython d31ccae04d039ca3ab294742f2493aa5f2b63ff7, host build (x86_64-unknown-linux-gnu, debug), zlib.ZLIB_VERSION = 1.3.0-zlib-rs-0.6.8. crates/common/src/compression/zlib.rs and crates/stdlib/src/zlib.rs on main (a7d75d25...
Open Graph Description: Environment RustPython d31ccae04d039ca3ab294742f2493aa5f2b63ff7, host build (x86_64-unknown-linux-gnu, debug), zlib.ZLIB_VERSION = 1.3.0-zlib-rs-0.6.8. crates/common/src/compression/zlib.rs and cra...
X Description: Environment RustPython d31ccae04d039ca3ab294742f2493aa5f2b63ff7, host build (x86_64-unknown-linux-gnu, debug), zlib.ZLIB_VERSION = 1.3.0-zlib-rs-0.6.8. crates/common/src/compression/zlib.rs and cra...
Opengraph URL: https://github.com/RustPython/RustPython/issues/8906
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"`zlib.decompressobj`: `unconsumed_tail` keeps stale input when a `max_length`-bounded call reaches end of stream","articleBody":"### Environment\n\n- RustPython `d31ccae04d039ca3ab294742f2493aa5f2b63ff7`, host build (x86_64-unknown-linux-gnu,\n debug), `zlib.ZLIB_VERSION` = `1.3.0-zlib-rs-0.6.8`.\n- `crates/common/src/compression/zlib.rs` and `crates/stdlib/src/zlib.rs` on `main`\n (`a7d75d25a7b2c291720411dc65445accd972e69d`, 2026-09-29) are byte-identical to `d31ccae`, so the\n line numbers below hold for both.\n- Reference: CPython 3.13.5.\n\n### Reproducer\n\n```python\nimport zlib\ndata = zlib.compress(b\"x\" * 20000) + b\"NEXT\"\nd = zlib.decompressobj()\nout = d.decompress(data, 100)\nprint(\"call 1:\", len(out), d.eof, len(d.unconsumed_tail), d.unused_data)\nout = d.decompress(d.unconsumed_tail, 100000)\nprint(\"call 2:\", len(out), d.eof, d.unconsumed_tail, d.unused_data)\nprint(\"flush:\", d.flush(), d.unconsumed_tail, d.unused_data)\n```\n\nCPython 3.13.5:\n\n```text\ncall 1: 100 False 28 b''\ncall 2: 19900 True b'NEXT' b'NEXT'\nflush: b'' b'NEXT' b'NEXTNEXT'\n```\n\nRustPython `d31ccae`:\n\n```text\ncall 1: 100 False 29 b''\ncall 2: 19900 True b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\xe8\\xc0\\x00eG\\xa1\\x1dNEXT' b'NEXT'\nflush: b'' b'' b'NEXT\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\xe8\\xc0\\x00eG\\xa1\\x1dNEXT'\n```\n\nAfter call 2 the stream is at its end. CPython reports only the bytes after the stream end in\n`unconsumed_tail`. RustPython reports the whole `unconsumed_tail` of call 1: compressed bytes that\ncall 2 already consumed, followed by `NEXT`. `flush()` then appends those stale bytes to\n`unused_data`. (The call 1 tail length, 28 vs 29, is a separate difference in how far inflate reads\nahead; this report does not depend on it.)\n\n### Source\n\n`crates/common/src/compression/zlib.rs`, `Decompressor::decompress_inner`, lines 674–683:\n\n```rust\nlet unconsumed = &data[consumed..];\nif !unconsumed.is_empty() {\n if stream_end {\n unused_data.extend_from_slice(unconsumed);\n } else {\n *unconsumed_tail = unconsumed.to_vec();\n }\n} else if !unconsumed_tail.is_empty() {\n unconsumed_tail.clear();\n}\n```\n\nWhen `stream_end` is true and input is left over, the branch updates `unused_data` and leaves\n`unconsumed_tail` unchanged, so it still holds the previous call's tail. `Decompressor::flush`\n(line 702) takes that stale `unconsumed_tail` as its input, and `decompress_inner` moves it to\n`unused_data`. CPython's observed result sets `unconsumed_tail` to the bytes after the stream end.\n\n### Impact\n\nurllib3 2.x (checked against 2.8.0) `GzipDecoder.decompress` continues a multi-member body with\n`self._obj.unconsumed_tail or self._obj.unused_data` and opens a new `decompressobj` when\n`self._obj.eof` is true. On RustPython the stale tail is fed to the new object as the next gzip\nmember. Measured with urllib3 2.8.0's `GzipDecoder` class copied into a script (no network),\nbody `gzip.compress(b\"x\" * 20000) * 12`, drained with `max_length=10240`:\n\n| | output bytes | final `_state` |\n|---|---|---|\n| CPython 3.13.5 | 240 000 | `OTHER_MEMBERS` |\n| RustPython `d31ccae` | 20 000 | `SWALLOW_DATA` |\n\nOn RustPython the second member raises `zlib.error` inside the decoder, the decoder switches to\n`SWALLOW_DATA`, and 11 of 12 members are dropped with no error to the caller.\n\nThe reproducers above were run locally on RustPython and CPython.\n\n---\n\n🤖 Written by Claude Code (2.1.283) using model claude-opus-5-5\n","author":{"url":"https://github.com/JMLX42","@type":"Person","name":"JMLX42"},"datePublished":"2026-09-29T15:51:36Z","url":"https://github.com/RustPython/RustPython/issues/8906"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:9a5fa4d1-1486-1af4-2211-41754364fe3d |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | D242:3A8E64:16CB88A:1E2D878:6B02069B |
| html-safe-nonce | 761abb16d12eed022a3073502f5f4f21e4ffbbbc71d277fb893518af5de9ede2 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJEMjQyOjNBOEU2NDoxNkNCODhBOjFFMkQ4Nzg6NkIwMjA2OUIiLCJ2aXNpdG9yX2lkIjoiNDQxMzk5Nzc2ODAzMzU2ODQxMSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 8d8737b0efb138a757ab0201a8d5d64bedd197fbca49286f13b098f2e7015dfa |
| hovercard-subject-tag | issue:5633265546 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/RustPython/RustPython/8906/issue_layout |
| twitter:image | https://opengraph.githubassets.com/145dfbbb435ab460794ad0b10a459ed5ac386d09fca66bd4a51ede6109ea8a91/RustPython/RustPython/issues/8906 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/145dfbbb435ab460794ad0b10a459ed5ac386d09fca66bd4a51ede6109ea8a91/RustPython/RustPython/issues/8906 |
| og:image:alt | Environment RustPython d31ccae04d039ca3ab294742f2493aa5f2b63ff7, host build (x86_64-unknown-linux-gnu, debug), zlib.ZLIB_VERSION = 1.3.0-zlib-rs-0.6.8. crates/common/src/compression/zlib.rs and cra... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | JMLX42 |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| go-import | github.com/RustPython/RustPython git https://github.com/RustPython/RustPython.git |
| octolytics-dimension-user_id | 39710557 |
| octolytics-dimension-user_login | RustPython |
| octolytics-dimension-repository_id | 135201145 |
| octolytics-dimension-repository_nwo | RustPython/RustPython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 135201145 |
| octolytics-dimension-repository_network_root_nwo | RustPython/RustPython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width