Title: Consider not consuming all the buffer in one go in the tokenizer module · Issue #105069 · python/cpython · GitHub
Open Graph Title: Consider not consuming all the buffer in one go in the tokenizer module · Issue #105069 · python/cpython
X Title: Consider not consuming all the buffer in one go in the tokenizer module · Issue #105069 · python/cpython
Description: Seems that some tools were relying on the implementation detail that the readline-like callable that is provided to functions in the tokenize module is called as tokens are emitted and not consumed in one go. Although this was never part...
Open Graph Description: Seems that some tools were relying on the implementation detail that the readline-like callable that is provided to functions in the tokenize module is called as tokens are emitted and not consumed...
X Description: Seems that some tools were relying on the implementation detail that the readline-like callable that is provided to functions in the tokenize module is called as tokens are emitted and not consumed...
Opengraph URL: https://github.com/python/cpython/issues/105069
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Consider not consuming all the buffer in one go in the tokenizer module","articleBody":"Seems that some tools were relying on the implementation detail that the `readline`-like callable that is provided to functions in the `tokenize` module is called as tokens are emitted and not consumed in one go. Although this was never part of the contract and technically we don't need to change, it would make the implementation more efficient as we don't need to hold the entire input in memory at the same time and we won't break these tools.\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-105070\n* gh-105119\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/pablogsal","@type":"Person","name":"pablogsal"},"datePublished":"2023-05-29T19:13:02.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/105069/cpython/issues/105069"}| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:741052ef-77bb-b84d-73f8-98e0fec2b5b9 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | A726:30B952:548121:7303DB:6969DA39 |
| html-safe-nonce | 3488fe3435898f8046257a6ccee626f6819ee79c4cefd8f012d735baf2ff0476 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBNzI2OjMwQjk1Mjo1NDgxMjE6NzMwM0RCOjY5NjlEQTM5IiwidmlzaXRvcl9pZCI6IjUwMDcyODc1MjkzMjA4NjQzMTMiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | b358c90ccc8c29d9970cac86d7cae94c6340a687386d0dd8439c8f4a7ca0724f |
| hovercard-subject-tag | issue:1731150917 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/105069/issue_layout |
| twitter:image | https://opengraph.githubassets.com/689e4585b6d6df433aab536ae73cce27e980956fa8f903495439dcb34970921c/python/cpython/issues/105069 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/689e4585b6d6df433aab536ae73cce27e980956fa8f903495439dcb34970921c/python/cpython/issues/105069 |
| og:image:alt | Seems that some tools were relying on the implementation detail that the readline-like callable that is provided to functions in the tokenize module is called as tokens are emitted and not consumed... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | pablogsal |
| hostname | github.com |
| expected-hostname | github.com |
| None | 7b32f1c7c4549428ee399213e8345494fc55b5637195d3fc5f493657579235e8 |
| turbo-cache-control | no-preview |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | bdde15ad1b403e23b08bbd89b53fbe6bdf688cad |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width