Title: Include column offset information in the tokenizer · Issue #97997 · python/cpython · GitHub
Open Graph Title: Include column offset information in the tokenizer · Issue #97997 · python/cpython
X Title: Include column offset information in the tokenizer · Issue #97997 · python/cpython
Description: The tokenizer currently only holds information about the current line number (along with the starting line number for strings that span multiple lines). This makes computing column offset harder, since it has to be done with pointer arit...
Open Graph Description: The tokenizer currently only holds information about the current line number (along with the starting line number for strings that span multiple lines). This makes computing column offset harder, s...
X Description: The tokenizer currently only holds information about the current line number (along with the starting line number for strings that span multiple lines). This makes computing column offset harder, s...
Opengraph URL: https://github.com/python/cpython/issues/97997
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Include column offset information in the tokenizer","articleBody":"The tokenizer currently only holds information about the current line number (along with the starting line number for strings that span multiple lines). This makes computing column offset harder, since it has to be done with pointer arithmetic using pointers to the beginning and end of the token (even more complicated when line continuations or multiline tokens happen).\r\n\r\n### Feature or enhancement\r\n\r\nAll of this can be significantly simplified, if we keep a column offset counter in the tokenizer state. \r\n","author":{"url":"https://github.com/lysnikolaou","@type":"Person","name":"lysnikolaou"},"datePublished":"2022-10-06T22:30:56.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/97997/cpython/issues/97997"}| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:9eae5cf5-19e0-6491-1c4e-935daa5743c0 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | E1E4:DC558:9D7B3F:D0EEE9:69699128 |
| html-safe-nonce | 5726b8145dc57fcaa405d35a2dbd3894d3166dcd0ed83a83612d1b5a0aacc2a2 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJFMUU0OkRDNTU4OjlEN0IzRjpEMEVFRTk6Njk2OTkxMjgiLCJ2aXNpdG9yX2lkIjoiNTAxMjI4ODcxMzc5NDM1OTU5MiIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | f542ced5c71d98991dfaee514b8cdac83bb68baefa449d013097312d2903297d |
| hovercard-subject-tag | issue:1400370778 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/97997/issue_layout |
| twitter:image | https://opengraph.githubassets.com/afc607e4ccf2c04703f96c07850455fff45922dfd5381fde1a0ab4e602c7065e/python/cpython/issues/97997 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/afc607e4ccf2c04703f96c07850455fff45922dfd5381fde1a0ab4e602c7065e/python/cpython/issues/97997 |
| og:image:alt | The tokenizer currently only holds information about the current line number (along with the starting line number for strings that span multiple lines). This makes computing column offset harder, s... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | lysnikolaou |
| hostname | github.com |
| expected-hostname | github.com |
| None | 3542e147982176a7ebaa23dfb559c8af16f721c03ec560c68c56b64a0f35e751 |
| turbo-cache-control | no-preview |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | af80af7cc9e3de9c336f18b208a600950a3c187c |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width