René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Tokenizer returnings pointers make calculation of position info more complicated","articleBody":"Bacause the tokenizer only returns pointers to the beginning and the end of the token, calculating line numbers/column offsets is more complicated than needed. \r\n\r\n### Feature or enhancement\r\n\r\nInstead of the tokenizer returning a token type and settings pointers, we wanna return the token type (remains as is) and then set a `struct token` that has the following information:\r\n1. Pointers to beginning and end\r\n2. Location information (`lineno`, `col_offset`, etc.)\r\n3. Level (the level in the `parenstack`)\r\n\r\nThis way the parser will have a much easier job of setting line numbers \u0026 column offsets in the generated AST numbers and will make some of our work on the f-strings parsing easier.  \r\n","author":{"url":"https://github.com/lysnikolaou","@type":"Person","name":"lysnikolaou"},"datePublished":"2022-10-06T17:05:32.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/97973/cpython/issues/97973"}