Title: Base the tokenizer API on source offsets · Issue #153569 · python/cpython · GitHub
Open Graph Title: Base the tokenizer API on source offsets · Issue #153569 · python/cpython
X Title: Base the tokenizer API on source offsets · Issue #153569 · python/cpython
Description: The tokenizer currently stores many source positions as pointers into buffers that can move. Reallocating the input means rebasing all these pointers, and missing one is very easy, especially with incremental input and f-strings. I think...
Open Graph Description: The tokenizer currently stores many source positions as pointers into buffers that can move. Reallocating the input means rebasing all these pointers, and missing one is very easy, especially with ...
X Description: The tokenizer currently stores many source positions as pointers into buffers that can move. Reallocating the input means rebasing all these pointers, and missing one is very easy, especially with ...
Opengraph URL: https://github.com/python/cpython/issues/153569
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Base the tokenizer API on source offsets","articleBody":"The tokenizer currently stores many source positions as pointers into buffers that can move. Reallocating the input means rebasing all these pointers, and missing one is very easy, especially with incremental input and f-strings.\n\nI think tokenizer positions should be offsets into the decoded source instead:\n\n```c\ntypedef Py_ssize_t TokenizerOffset;\n\ntypedef struct {\n TokenizerOffset start;\n TokenizerOffset end;\n} TokenizerSpan;\n```\n\nThe main ideas would be:\n\n- One source object owns the decoded text.\n- The cursor only stores its current offset and line boundaries.\n- Tokens, errors and f-string state use offset spans instead of pointers.\n- pegen and `_tokenize` ask the source for a view or copy of a span.\n- Sequential tokenization keeps the current line in the cursor, while uncommon line lookups can use a small sparse index.\n- Normal parsing can keep one contiguous buffer, while `tokenize(readline)` can eventually use reclaimable chunks.\n\nFor incremental tokenization, new decoded input would be appended only when the cursor needs more data. Existing offsets would remain valid even if the underlying storage moves. Once `tokenize` has returned copied token and line strings, old input could be discarded when no active token, cursor, f-string frame or error still refers to it.\n\nThis is very nice because it separates source storage from tokenizer state, removes pointer rebasing, and means consumers no longer need to access tokenizer internals.\n\n\u003c!-- gh-linked-prs -->\n### Linked PRs\n* gh-153585\n* gh-153587\n* gh-156472\n* gh-156482\n* gh-156484\n* gh-156654\n* gh-157055\n* gh-158124\n* gh-158849\n\u003c!-- /gh-linked-prs -->\n","author":{"url":"https://github.com/pablogsal","@type":"Person","name":"pablogsal"},"datePublished":"2026-07-11T13:57:27Z","url":"https://github.com/python/cpython/issues/153569"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:3a68825f-3ff7-e03e-d987-a100dc3311cd |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | B3B0:3A4875:32804AB:434B295:6B01F253 |
| html-safe-nonce | fbc8744bd84fbf4d975d3ec5214910d57a45ebb943388b1911a8411f7d24a298 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCM0IwOjNBNDg3NTozMjgwNEFCOjQzNEIyOTU6NkIwMUYyNTMiLCJ2aXNpdG9yX2lkIjoiODY2NjUwMzY1MTU0NTcwNzA5MSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | ca3b08d04c0a527d51fb9ad730ba26435449eff0e155b321dd949ff55067f46d |
| hovercard-subject-tag | issue:4862561196 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/153569/issue_layout |
| twitter:image | https://opengraph.githubassets.com/73faa20b1bef0b1e8c975ae4e090fb990f0dfd15728a2b0dc83b3b9454a642ef/python/cpython/issues/153569 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/73faa20b1bef0b1e8c975ae4e090fb990f0dfd15728a2b0dc83b3b9454a642ef/python/cpython/issues/153569 |
| og:image:alt | The tokenizer currently stores many source positions as pointers into buffers that can move. Reallocating the input means rebasing all these pointers, and missing one is very easy, especially with ... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | pablogsal |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width