Title: Use a managed dict for the _io types · Issue #161184 · python/cpython · GitHub
Open Graph Title: Use a managed dict for the _io types · Issue #161184 · python/cpython
X Title: Use a managed dict for the _io types · Issue #161184 · python/cpython
Description: Feature or enhancement _io._IOBase and its C subclasses (BufferedReader/Writer/Random/RWPair, BytesIO, FileIO, StringIO, TextIOWrapper, _WindowsConsoleIO) each carry an explicit PyObject *dict field exposed through __dictoffset__. This h...
Open Graph Description: Feature or enhancement _io._IOBase and its C subclasses (BufferedReader/Writer/Random/RWPair, BytesIO, FileIO, StringIO, TextIOWrapper, _WindowsConsoleIO) each carry an explicit PyObject *dict fiel...
X Description: Feature or enhancement _io._IOBase and its C subclasses (BufferedReader/Writer/Random/RWPair, BytesIO, FileIO, StringIO, TextIOWrapper, _WindowsConsoleIO) each carry an explicit PyObject *dict fiel...
Opengraph URL: https://github.com/python/cpython/issues/161184
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Use a managed dict for the _io types","articleBody":"### Feature or enhancement\n\n`_io._IOBase` and its C subclasses (`BufferedReader`/`Writer`/`Random`/`RWPair`, `BytesIO`, `FileIO`, `StringIO`, `TextIOWrapper`, `_WindowsConsoleIO`) each carry an explicit `PyObject *dict` field exposed through `__dictoffset__`. This has two costs:\n\n- Every instance attribute write allocates a real dict. `open()` always does this: (e.g. `FileIO` gets `name`, `TextIOWrapper` gets `mode`, etc.)\n- Instance attribute access never specialises. `specialize_dict_access` bails with `SPEC_FAIL_ATTR_NOT_MANAGED_DICT` for types without `Py_TPFLAGS_MANAGED_DICT`, so `LOAD_ATTR`/`STORE_ATTR` stay generic. Because the legacy `tp_dictoffset` is inherited, this applies to every Python subclass of an io base as well.\n\nSwitching `_IOBase` to `Py_TPFLAGS_MANAGED_DICT` (as #160616 did for `ast.AST`) removes the eight `dict` fields and the `__dictoffset__` members; the flag is inherited by subclasses.\n\n## Benchmarks\n\n| Benchmark | main | branch |\n|---|---|---|\n| `http.client`: `HTTPResponse(sock).begin()` + `read()` of a 2-byte body | 50.0 us | 46.4 us: 1.08x faster |\n| `http.client`: `read()` of a chunked body (1000 chunks) | 1.63 ms | 1.34 ms: 1.22x faster |\n| `http.client`: `read(100)` x 1000 | 650 us | 411 us: 1.58x faster |\n| `http.client`: `for line in response` (1000 lines) | 724 us | 453 us: 1.60x faster |\n| `zipfile`: `zf.read(name)` for 200 small members | 2.55 ms | 1.99 ms: 1.28x faster |\n| `zipfile`: `for line in zf.open(name)` (1000 lines) | 583 us | 403 us: 1.44x faster |\n| `zipfile`: `ZipExtFile.read(100)` x 1000 | 545 us | 369 us: 1.48x faster |\n| `gzip`: `GzipFile.read(100)` x 1000 | 497 us | 382 us: 1.30x faster |\n| `gzip`: `GzipFile.write(100 bytes)` x 1000 | 646 us | 519 us: 1.24x faster |\n| `tempfile`: `SpooledTemporaryFile.write(100 bytes)` x 1000 | 340 us | 241 us: 1.41x faster |\n| `urllib3`: `HTTPResponse.read(100)` x 1000 | 3.94 ms | 3.12 ms: 1.26x faster |\n| `urllib3`: `for chunk in response.stream(100)` | 4.93 ms | 4.12 ms: 1.20x faster |\n| control (`list(range(20000))`) | 1.36 ms | 1.41 ms: 1.03x slower |\n| Geometric mean | | 1.30x faster |\n\nThe gain comes from `LOAD_ATTR`/`STORE_ATTR` on `self` specialising in the Python-level `read`/`readline`/`write` methods of these classes, and from `__init__` no longer allocating and growing a dict per instance. The C types themselves are not the target; for them the change is close to neutral.\n\n\u003cdetails>\u003csummary>Benchmark script\u003c/summary>\n\n```python\n\"\"\"Real-world code paths that run Python methods on io subclasses.\"\"\"\nimport gzip, io, sys, tempfile, zipfile, pyperf\nimport http.client\nfrom urllib3.response import HTTPResponse as U3Response\n\nCHUNK = 100\nN = 1000\nBODY = bytes(range(256)) * (CHUNK * N // 256) # ~100 KB\nLINES = b\"\".join(b\"%06d: some text on this line, to be read\\n\" % i for i in range(N))\n\nclass FakeSocket:\n def __init__(self, data):\n self.data = data\n def makefile(self, mode):\n return io.BytesIO(self.data)\n def close(self):\n pass\n\nHEADERS = (b\"HTTP/1.1 200 OK\\r\\nServer: bench\\r\\nDate: Mon, 02 Nov 2026 10:00:00 GMT\\r\\n\"\n b\"Content-Type: text/plain\\r\\nCache-Control: no-cache\\r\\n\"\n b\"Connection: keep-alive\\r\\nX-Request-Id: 0123456789abcdef\\r\\n\")\n\ndef chunked_body(data, size):\n out = []\n for i in range(0, len(data), size):\n c = data[i:i + size]\n out.append(b\"%x\\r\\n%s\\r\\n\" % (len(c), c))\n out.append(b\"0\\r\\n\\r\\n\")\n return b\"\".join(out)\n\nRESP_CL = HEADERS + b\"Content-Length: %d\\r\\n\\r\\n\" % len(BODY) + BODY\nRESP_LINES = HEADERS + b\"Content-Length: %d\\r\\n\\r\\n\" % len(LINES) + LINES\nRESP_CHUNKED = HEADERS + b\"Transfer-Encoding: chunked\\r\\n\\r\\n\" + chunked_body(BODY, CHUNK)\nRESP_SMALL = HEADERS + b\"Content-Length: 2\\r\\n\\r\\nok\"\n\ndef _response(raw):\n r = http.client.HTTPResponse(FakeSocket(raw), method=\"GET\")\n r.begin()\n return r\n\ndef httpclient_begin(loops):\n \"\"\"HTTPResponse(sock).begin(): one per HTTP request.\"\"\"\n r = range(loops); t0 = pyperf.perf_counter()\n for _ in r:\n _response(RESP_SMALL).read()\n return pyperf.perf_counter() - t0\n\ndef httpclient_read_chunked(loops):\n \"\"\"read() of a chunked body with 1000 100-byte chunks.\"\"\"\n r = range(loops); t0 = pyperf.perf_counter()\n for _ in r:\n _response(RESP_CHUNKED).read()\n return pyperf.perf_counter() - t0\n\ndef httpclient_read_small(loops):\n \"\"\"read(100) x 1000 on a Content-Length body.\"\"\"\n r = range(loops); t0 = pyperf.perf_counter()\n for _ in r:\n resp = _response(RESP_CL)\n read = resp.read\n while read(CHUNK):\n pass\n return pyperf.perf_counter() - t0\n\ndef httpclient_iter_lines(loops):\n \"\"\"for line in response: (1000 lines).\"\"\"\n r = range(loops); t0 = pyperf.perf_counter()\n for _ in r:\n for line in _response(RESP_LINES):\n pass\n return pyperf.perf_counter() - t0\n\n# --- zipfile ---------------------------------------------------------------\n_zbuf = io.BytesIO()\nwith zipfile.ZipFile(_zbuf, \"w\", zipfile.ZIP_DEFLATED) as zf:\n for i in range(200):\n zf.writestr(\"member%03d.txt\" % i, b\"x\" * 100)\n zf.writestr(\"lines.txt\", LINES)\n zf.writestr(\"body.bin\", BODY)\nZF = zipfile.ZipFile(_zbuf)\nMEMBERS = [n for n in ZF.namelist() if n.startswith(\"member\")]\n\ndef zipfile_read_members(loops):\n \"\"\"zf.read(name) for 200 small members (ZipExtFile created per member).\"\"\"\n r = range(loops); read = ZF.read; t0 = pyperf.perf_counter()\n for _ in r:\n for name in MEMBERS:\n read(name)\n return pyperf.perf_counter() - t0\n\ndef zipfile_iter_lines(loops):\n \"\"\"for line in zf.open(name): (1000 lines).\"\"\"\n r = range(loops); t0 = pyperf.perf_counter()\n for _ in r:\n with ZF.open(\"lines.txt\") as f:\n for line in f:\n pass\n return pyperf.perf_counter() - t0\n\ndef zipfile_read_small(loops):\n \"\"\"f.read(100) x 1000 from a deflated member.\"\"\"\n r = range(loops); t0 = pyperf.perf_counter()\n for _ in r:\n with ZF.open(\"body.bin\") as f:\n read = f.read\n while read(CHUNK):\n pass\n return pyperf.perf_counter() - t0\n\n# --- gzip -------------------------------------------------------------------\nGZ = gzip.compress(BODY)\n\ndef gzip_read_small(loops):\n \"\"\"GzipFile.read(100) x 1000.\"\"\"\n r = range(loops); t0 = pyperf.perf_counter()\n for _ in r:\n with gzip.GzipFile(fileobj=io.BytesIO(GZ)) as f:\n read = f.read\n while read(CHUNK):\n pass\n return pyperf.perf_counter() - t0\n\ndef gzip_write_small(loops):\n \"\"\"GzipFile.write(100 bytes) x 1000.\"\"\"\n r = range(loops); piece = BODY[:CHUNK]; t0 = pyperf.perf_counter()\n for _ in r:\n with gzip.GzipFile(fileobj=io.BytesIO(), mode=\"wb\") as f:\n write = f.write\n for _ in range(N):\n write(piece)\n return pyperf.perf_counter() - t0\n\n# --- tempfile ---------------------------------------------------------------\ndef spooled_write_small(loops):\n \"\"\"SpooledTemporaryFile.write(100 bytes) x 1000 (stays in memory).\"\"\"\n r = range(loops); piece = BODY[:CHUNK]; t0 = pyperf.perf_counter()\n for _ in r:\n with tempfile.SpooledTemporaryFile(max_size=10**7) as f:\n write = f.write\n for _ in range(N):\n write(piece)\n return pyperf.perf_counter() - t0\n\n# --- urllib3 ----------------------------------------------------------------\ndef _u3():\n return U3Response(body=io.BytesIO(BODY), headers={\"content-length\": str(len(BODY))},\n status=200, preload_content=False, decode_content=True)\n\ndef urllib3_read_small(loops):\n \"\"\"urllib3 HTTPResponse.read(100) x 1000.\"\"\"\n r = range(loops); t0 = pyperf.perf_counter()\n for _ in r:\n resp = _u3(); read = resp.read\n while read(CHUNK):\n pass\n return pyperf.perf_counter() - t0\n\ndef urllib3_stream(loops):\n \"\"\"for chunk in resp.stream(100): (1000 chunks).\"\"\"\n r = range(loops); t0 = pyperf.perf_counter()\n for _ in r:\n for chunk in _u3().stream(CHUNK):\n pass\n return pyperf.perf_counter() - t0\n\ndef control_list(loops):\n \"\"\"Unaffected code: noise floor of the two-binary comparison.\"\"\"\n r = range(loops); q = range(20000); t0 = pyperf.perf_counter()\n for _ in r:\n list(q); list(q); list(q); list(q); list(q)\n return pyperf.perf_counter() - t0\n\nBENCHES = (httpclient_begin, httpclient_read_chunked, httpclient_read_small, httpclient_iter_lines,\n zipfile_read_members, zipfile_iter_lines, zipfile_read_small,\n gzip_read_small, gzip_write_small, spooled_write_small,\n urllib3_read_small, urllib3_stream, control_list)\n\nif __name__ == \"__main__\":\n if \"--check\" in sys.argv:\n for f in BENCHES:\n print(f.__name__, \"%.3f ms\" % (f(1) * 1e3))\n sys.exit()\n runner = pyperf.Runner()\n for f in BENCHES:\n runner.bench_time_func(f.__name__, f)\n```\n\u003c/details>\n\nPR: https://github.com/eendebakpt/cpython/tree/io-managed-dict (branch, PR to follow)\n\n\n### Has this already been discussed elsewhere?\n\nNo response given\n\n### Links to previous discussion of this feature:\n\n_No response_\n\n\u003c!-- gh-linked-prs -->\n### Linked PRs\n* gh-161236\n\u003c!-- /gh-linked-prs -->\n","author":{"url":"https://github.com/eendebakpt","@type":"Person","name":"eendebakpt"},"datePublished":"2026-11-02T13:37:34Z","url":"https://github.com/python/cpython/issues/161184"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:39fb97d8-5eca-e3c0-3458-d3ae59994c5d |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | ECEC:399E4B:377481E:4991775:6B0202AD |
| html-safe-nonce | d987854ae79a9309d48a0594e08ecc753929398ebefb085b38d856172275e0a4 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJFQ0VDOjM5OUU0QjozNzc0ODFFOjQ5OTE3NzU6NkIwMjAyQUQiLCJ2aXNpdG9yX2lkIjoiNDMzNzM5MzUxMzAyNjE1OTI3NyIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | b46b96e183c4b0888bebe2df60a42a51e0ef1be5f098f5e5f6d07a4092a3a18b |
| hovercard-subject-tag | issue:6163656218 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/161184/issue_layout |
| twitter:image | https://opengraph.githubassets.com/19c90e4ec3f2653e59b0e9e10c32209c0049bf3d3c802e635b59d67a58e24040/python/cpython/issues/161184 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/19c90e4ec3f2653e59b0e9e10c32209c0049bf3d3c802e635b59d67a58e24040/python/cpython/issues/161184 |
| og:image:alt | Feature or enhancement _io._IOBase and its C subclasses (BufferedReader/Writer/Random/RWPair, BytesIO, FileIO, StringIO, TextIOWrapper, _WindowsConsoleIO) each carry an explicit PyObject *dict fiel... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | eendebakpt |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width