| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/commits(/*name) |
| route-controller | commits |
| route-action | show |
| fetch-nonce | v2:58782770-99e5-1f91-8c08-036e69c45043 |
| current-catalog-service-hash | f3abb0cc802f3d7b95fc8762b94bdcb13bf39634c40c357301c4aa1d67a256fb |
| request-id | CE9A:215B92:395D536:4C5BE1C:6B021E17 |
| html-safe-nonce | 75c24ae032b4fe315dfaffc177f666637497d8da4f8f28ba4b50166364eb5aea |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDRTlBOjIxNUI5MjozOTVENTM2OjRDNUJFMUM6NkIwMjFFMTciLCJ2aXNpdG9yX2lkIjoiNjg4ODI0MjcxMjU5Njg0ODE1MSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 6296b34e484875c33deeba9bce7c79a7a4989dc61f33e0233d7efee90f6702d2 |
| hovercard-subject-tag | repository:1447790285 |
| github-keyboard-shortcuts | repository,commit-list,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | ///commits/show |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/sidlovescoding/Cpython/commits/sid-python/Lib/tokenize.py |
| twitter:image | https://opengraph.githubassets.com/ed1e8647ba3ae8c65931fa343b8bc8beb09e2048558df81b38a0b843ce1557b9/sidlovescoding/Cpython |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/ed1e8647ba3ae8c65931fa343b8bc8beb09e2048558df81b38a0b843ce1557b9/sidlovescoding/Cpython |
| og:image:alt | The Python programming language. Contribute to sidlovescoding/Cpython development by creating an account on GitHub. |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-cache |
| go-import | github.com/sidlovescoding/Cpython git https://github.com/sidlovescoding/Cpython.git |
| octolytics-dimension-user_id | 207958570 |
| octolytics-dimension-user_login | sidlovescoding |
| octolytics-dimension-repository_id | 1447790285 |
| octolytics-dimension-repository_nwo | sidlovescoding/Cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | true |
| octolytics-dimension-repository_parent_id | 81598961 |
| octolytics-dimension-repository_parent_nwo | python/cpython |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | commits |
| Skip to content | https://github.com/sidlovescoding/Cpython/commits/sid-python/Lib/tokenize.py#start-of-content |
| https://github.com/ |
| Sign in | https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsidlovescoding%2FCpython%2Fcommits%2Fsid-python%2FLib%2Ftokenize.py |
| GitHubRepository hosting, code review, and collaboration | https://github.com/platform |
| CopilotAI coding agent for tasks, projects, and automated work | https://github.com/features/copilot |
| ActionsCI/CD and automation | https://github.com/features/actions |
| Code QualityQuality analysis, fixes, and enforcement | https://github.com/features/code-quality |
| CodespacesCloud development environments | https://github.com/features/codespaces |
| Code SecurityVulnerability detection and remediation | https://github.com/security/advanced-security/code-security |
| Secret ProtectionSecret scanning and leak prevention | https://github.com/security/advanced-security/secret-protection |
| All Features | https://github.com/features |
| Documentation | https://docs.github.com/en |
| Training | https://learn.github.com/skills |
| Certifications | https://learn.github.com/credentials |
| Build with Copilot | https://github.com/features/ai |
| Customize Copilot | https://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview |
| Integrations | https://github.com/marketplace |
| Trending repos | https://github.com/trending |
| GitHub Sponsors | https://github.com/open-source/sponsors |
| Developer articles | https://github.com/resources/articles |
| ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place. | https://github.blog/changelog/ |
| Enterprises | https://github.com/enterprise |
| Team | https://github.com/team |
| Startups | https://github.com/enterprise/startups |
| Nonprofits | https://github.com/solutions/industry/nonprofits |
| Healthcare | https://github.com/solutions/industry/healthcare |
| Financial services | https://github.com/solutions/industry/financial-services |
| Manufacturing | https://github.com/solutions/industry/manufacturing |
| Government | https://github.com/solutions/industry/government |
| See all industries | https://github.com/solutions/industry |
| Why GitHub | https://github.com/why-github |
| Customer Stories | https://github.com/customer-stories |
| Business insights | https://github.com/solutions/executive-insights |
| Trust Center | https://github.com/trust-center |
| GitHub Advanced SecurityEnterprise-grade code security | https://github.com/security/advanced-security |
| Copilot for businessEnterprise-grade AI features | https://github.com/features/copilot/copilot-business |
| Premium SupportEnterprise-grade 24/7 support | https://github.com/enterprise/premium-support |
| Documentation | https://docs.github.com/en |
| Resource center | https://github.com/resources |
| Blog | https://github.blog/ |
| Events & webinars | https://github.com/resources/events |
| Community Forum | https://github.com/orgs/community/discussions |
| Customer Stories | https://github.com/customer-stories |
| Integrations | https://github.com/marketplace |
| Customer support | https://help.github.com/ |
| Training | https://learn.github.com/skills |
| GitHub Status | https://www.githubstatus.com/ |
| About | https://github.com/about |
| Partners | https://github.com/partners |
| Changelog | https://github.blog/changelog/ |
| GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub. | https://githubuniverse.com/ |
| Pricing | https://github.com/pricing |
| Sign in | https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsidlovescoding%2FCpython%2Fcommits%2Fsid-python%2FLib%2Ftokenize.py |
| Sign up | https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fcommits%2Fshow&source=header-repo&source_repo=sidlovescoding%2FCpython |
| Reload | https://github.com/sidlovescoding/Cpython/commits/sid-python/Lib/tokenize.py |
| Reload | https://github.com/sidlovescoding/Cpython/commits/sid-python/Lib/tokenize.py |
| Reload | https://github.com/sidlovescoding/Cpython/commits/sid-python/Lib/tokenize.py |
| sidlovescoding | https://github.com/sidlovescoding |
| Cpython | https://github.com/sidlovescoding/Cpython |
| Code | https://github.com/sidlovescoding/Cpython |
| Pull requests1 (1) | https://github.com/sidlovescoding/Cpython/pulls |
| Actions | https://github.com/sidlovescoding/Cpython/actions |
| Projects | https://github.com/sidlovescoding/Cpython/projects |
| Security and quality | https://github.com/sidlovescoding/Cpython/security |
| Insights | https://github.com/sidlovescoding/Cpython/pulse |
| https://github.com/sidlovescoding/Cpython |
| Cpython | https://github.com/sidlovescoding/Cpython/commits |
| Lib | https://github.com/sidlovescoding/Cpython/commits/sid-python/Lib |
| tokenize.py | https://github.com/sidlovescoding/Cpython/commits/sid-python/Lib/tokenize.py |
| sid-python | https://github.com/sidlovescoding/Cpython/tree/sid-python |
| [3.14] gh-63161: Fix tokenize.detect_encoding() (GH-139446) (GH-140378) | https://github.com/sidlovescoding/Cpython/commit/8e93f6e203bd6cd2f22ec238442f82f52b354289 |
| miss-islington | https://github.com/sidlovescoding/Cpython/commits?author=miss-islington |
| serhiy-storchaka | https://github.com/sidlovescoding/Cpython/commits?author=serhiy-storchaka |
| 8e93f6e | https://github.com/sidlovescoding/Cpython/commit/8e93f6e203bd6cd2f22ec238442f82f52b354289 |
| https://github.com/sidlovescoding/Cpython/blob/8e93f6e203bd6cd2f22ec238442f82f52b354289/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/8e93f6e203bd6cd2f22ec238442f82f52b354289 |
| [3.14] gh-134675: Add t-string prefixes to tokenizer module, lexical analysis doc, and add a test to make sure we catch this error in the future. (GH-134734) (#134739) | https://github.com/sidlovescoding/Cpython/commit/74f5667bd9293337bc7c1b2fb89d9a8cb0598bc8 |
| miss-islington | https://github.com/sidlovescoding/Cpython/commits?author=miss-islington |
| ericvsmith | https://github.com/sidlovescoding/Cpython/commits?author=ericvsmith |
| 74f5667 | https://github.com/sidlovescoding/Cpython/commit/74f5667bd9293337bc7c1b2fb89d9a8cb0598bc8 |
| https://github.com/sidlovescoding/Cpython/blob/74f5667bd9293337bc7c1b2fb89d9a8cb0598bc8/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/74f5667bd9293337bc7c1b2fb89d9a8cb0598bc8 |
| [3.14] gh-134582: Fix t-strings untokenize() roundtrip removing space between braces (GH-134603) (#134659) | https://github.com/sidlovescoding/Cpython/commit/c61fd7c0f2df757fb485f60a8e6a5f8164477566 |
| miss-islington | https://github.com/sidlovescoding/Cpython/commits?author=miss-islington |
| loic-simon | https://github.com/sidlovescoding/Cpython/commits?author=loic-simon |
| c61fd7c | https://github.com/sidlovescoding/Cpython/commit/c61fd7c0f2df757fb485f60a8e6a5f8164477566 |
| https://github.com/sidlovescoding/Cpython/blob/c61fd7c0f2df757fb485f60a8e6a5f8164477566/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/c61fd7c0f2df757fb485f60a8e6a5f8164477566 |
| gh-130645: Add color to stdlib argparse CLIs (gh-133380) | https://github.com/sidlovescoding/Cpython/commit/4ac916ae33b962cb6b4f8849556403594b22a7f2 |
| https://github.com/hugovk |
| hugovk | https://github.com/sidlovescoding/Cpython/commits?author=hugovk |
| 4ac916a | https://github.com/sidlovescoding/Cpython/commit/4ac916ae33b962cb6b4f8849556403594b22a7f2 |
| https://github.com/sidlovescoding/Cpython/blob/4ac916ae33b962cb6b4f8849556403594b22a7f2/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/4ac916ae33b962cb6b4f8849556403594b22a7f2 |
| gh-133306: Use \z instead of \Z in regular expressions in the stdlib (GH-133337) | https://github.com/sidlovescoding/Cpython/commit/84a08f8629dffae355407cc3239c981a2b7a930a |
| https://github.com/serhiy-storchaka |
| serhiy-storchaka | https://github.com/sidlovescoding/Cpython/commits?author=serhiy-storchaka |
| 84a08f8 | https://github.com/sidlovescoding/Cpython/commit/84a08f8629dffae355407cc3239c981a2b7a930a |
| https://github.com/sidlovescoding/Cpython/blob/84a08f8629dffae355407cc3239c981a2b7a930a/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/84a08f8629dffae355407cc3239c981a2b7a930a |
| gh-132661: Implement PEP 750 (#132662) | https://github.com/sidlovescoding/Cpython/commit/60202609a2c2d0971aadfa4729ba30b50e89c6ea |
| 6020260 | https://github.com/sidlovescoding/Cpython/commit/60202609a2c2d0971aadfa4729ba30b50e89c6ea |
| https://github.com/sidlovescoding/Cpython/blob/60202609a2c2d0971aadfa4729ba30b50e89c6ea/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/60202609a2c2d0971aadfa4729ba30b50e89c6ea |
| gh-131178: Add tests for `tokenize` command-line interface (#131274) | https://github.com/sidlovescoding/Cpython/commit/9b83670f0f183a69440972765cae5942a0bd9fd7 |
| https://github.com/donbarbos |
| donbarbos | https://github.com/sidlovescoding/Cpython/commits?author=donbarbos |
| 9b83670 | https://github.com/sidlovescoding/Cpython/commit/9b83670f0f183a69440972765cae5942a0bd9fd7 |
| https://github.com/sidlovescoding/Cpython/blob/9b83670f0f183a69440972765cae5942a0bd9fd7/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/9b83670f0f183a69440972765cae5942a0bd9fd7 |
| gh-125553: Fix backslash continuation in `untokenize` (#126010) | https://github.com/sidlovescoding/Cpython/commit/7ad793e5dbdf07e51a71b70d20f3e6e3ab60244d |
| https://github.com/tomasr8 |
| tomasr8 | https://github.com/sidlovescoding/Cpython/commits?author=tomasr8 |
| 7ad793e | https://github.com/sidlovescoding/Cpython/commit/7ad793e5dbdf07e51a71b70d20f3e6e3ab60244d |
| https://github.com/sidlovescoding/Cpython/blob/7ad793e5dbdf07e51a71b70d20f3e6e3ab60244d/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/7ad793e5dbdf07e51a71b70d20f3e6e3ab60244d |
| gh-128519: Align the docstring of untokenize() to match the docs (#128521) | https://github.com/sidlovescoding/Cpython/commit/aef52ca8b334ff90e8032da39f4d06e7b5130eb9 |
| https://github.com/tomasr8 |
| tomasr8 | https://github.com/sidlovescoding/Cpython/commits?author=tomasr8 |
| aef52ca | https://github.com/sidlovescoding/Cpython/commit/aef52ca8b334ff90e8032da39f4d06e7b5130eb9 |
| https://github.com/sidlovescoding/Cpython/blob/aef52ca8b334ff90e8032da39f4d06e7b5130eb9/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/aef52ca8b334ff90e8032da39f4d06e7b5130eb9 |
| gh-91818: Use default program name in the CLI of many modules (GH-124867) | https://github.com/sidlovescoding/Cpython/commit/7d2c39752fa6f685f15ad9c585d83a62553477c2 |
| https://github.com/serhiy-storchaka |
| serhiy-storchaka | https://github.com/sidlovescoding/Cpython/commits?author=serhiy-storchaka |
| 7d2c397 | https://github.com/sidlovescoding/Cpython/commit/7d2c39752fa6f685f15ad9c585d83a62553477c2 |
| https://github.com/sidlovescoding/Cpython/blob/7d2c39752fa6f685f15ad9c585d83a62553477c2/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/7d2c39752fa6f685f15ad9c585d83a62553477c2 |
| gh-125008: Fix `tokenize.untokenize` roundtrip for `\n{{` (#125013) | https://github.com/sidlovescoding/Cpython/commit/db23b8bb13863fcd88ff91bc22398f8e0312039e |
| https://github.com/tomasr8 |
| tomasr8 | https://github.com/sidlovescoding/Cpython/commits?author=tomasr8 |
| db23b8b | https://github.com/sidlovescoding/Cpython/commit/db23b8bb13863fcd88ff91bc22398f8e0312039e |
| https://github.com/sidlovescoding/Cpython/blob/db23b8bb13863fcd88ff91bc22398f8e0312039e/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/db23b8bb13863fcd88ff91bc22398f8e0312039e |
| gh-115154: Fix untokenize handling of unicode named literals (#115171) | https://github.com/sidlovescoding/Cpython/commit/ecf16ee50e42f979624e55fa343a8522942db2e7 |
| https://github.com/pablogsal |
| pablogsal | https://github.com/sidlovescoding/Cpython/commits?author=pablogsal |
| ecf16ee | https://github.com/sidlovescoding/Cpython/commit/ecf16ee50e42f979624e55fa343a8522942db2e7 |
| https://github.com/sidlovescoding/Cpython/blob/ecf16ee50e42f979624e55fa343a8522942db2e7/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/ecf16ee50e42f979624e55fa343a8522942db2e7 |
| gh-104169: Fix test_peg_generator after tokenizer refactoring (#110727) | https://github.com/sidlovescoding/Cpython/commit/17d65547df55eaefe077c45242a7f2d175961dfd |
| https://github.com/lysnikolaou |
| lysnikolaou | https://github.com/sidlovescoding/Cpython/commits?author=lysnikolaou |
| 17d6554 | https://github.com/sidlovescoding/Cpython/commit/17d65547df55eaefe077c45242a7f2d175961dfd |
| https://github.com/sidlovescoding/Cpython/blob/17d65547df55eaefe077c45242a7f2d175961dfd/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/17d65547df55eaefe077c45242a7f2d175961dfd |
| gh-71299: Fix __all__ in tokenize (#105907) | https://github.com/sidlovescoding/Cpython/commit/ab3823a97bdeefb0266b3c8d493f7f6223ce3686 |
| https://github.com/lysnikolaou |
| lysnikolaou | https://github.com/sidlovescoding/Cpython/commits?author=lysnikolaou |
| ab3823a | https://github.com/sidlovescoding/Cpython/commit/ab3823a97bdeefb0266b3c8d493f7f6223ce3686 |
| https://github.com/sidlovescoding/Cpython/blob/ab3823a97bdeefb0266b3c8d493f7f6223ce3686/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/ab3823a97bdeefb0266b3c8d493f7f6223ce3686 |
| gh-105390: Correctly raise TokenError instead of SyntaxError for tokenize errors (#105399) | https://github.com/sidlovescoding/Cpython/commit/ffd26545509b706437ca38bd6db6108c1d0e2326 |
| https://github.com/pablogsal |
| pablogsal | https://github.com/sidlovescoding/Cpython/commits?author=pablogsal |
| ffd2654 | https://github.com/sidlovescoding/Cpython/commit/ffd26545509b706437ca38bd6db6108c1d0e2326 |
| https://github.com/sidlovescoding/Cpython/blob/ffd26545509b706437ca38bd6db6108c1d0e2326/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/ffd26545509b706437ca38bd6db6108c1d0e2326 |
| gh-105324: Fix tokenize module main function for stdin (#105325) | https://github.com/sidlovescoding/Cpython/commit/f04c16875b649e2c2b420eb146308d0206c7e527 |
| https://github.com/pablogsal |
| pablogsal | https://github.com/sidlovescoding/Cpython/commits?author=pablogsal |
| f04c168 | https://github.com/sidlovescoding/Cpython/commit/f04c16875b649e2c2b420eb146308d0206c7e527 |
| https://github.com/sidlovescoding/Cpython/blob/f04c16875b649e2c2b420eb146308d0206c7e527/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/f04c16875b649e2c2b420eb146308d0206c7e527 |
| gh-105069: Add a readline-like callable to the tokenizer to consume input iteratively (#105070) | https://github.com/sidlovescoding/Cpython/commit/9216e69a87d16d871625721ed5a8aa302511f367 |
| https://github.com/pablogsal |
| pablogsal | https://github.com/sidlovescoding/Cpython/commits?author=pablogsal |
| 9216e69 | https://github.com/sidlovescoding/Cpython/commit/9216e69a87d16d871625721ed5a8aa302511f367 |
| https://github.com/sidlovescoding/Cpython/blob/9216e69a87d16d871625721ed5a8aa302511f367/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/9216e69a87d16d871625721ed5a8aa302511f367 |
| gh-104976: Ensure trailing dedent tokens are emitted as the previous tokenizer (#104980) | https://github.com/sidlovescoding/Cpython/commit/46b52e6e2bda51d0b89a64ee36ce2d305a7409f3 |
| https://github.com/pablogsal |
| pablogsal | https://github.com/sidlovescoding/Cpython/commits?author=pablogsal |
| 46b52e6 | https://github.com/sidlovescoding/Cpython/commit/46b52e6e2bda51d0b89a64ee36ce2d305a7409f3 |
| https://github.com/sidlovescoding/Cpython/blob/46b52e6e2bda51d0b89a64ee36ce2d305a7409f3/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/46b52e6e2bda51d0b89a64ee36ce2d305a7409f3 |
| gh-102856: Tokenize performance improvement (#104731) | https://github.com/sidlovescoding/Cpython/commit/8817886ae571f5b5ce4e2e6cfd2458622d0efac1 |
| https://github.com/mgmacias95 |
| mgmacias95 | https://github.com/sidlovescoding/Cpython/commits?author=mgmacias95 |
| 8817886 | https://github.com/sidlovescoding/Cpython/commit/8817886ae571f5b5ce4e2e6cfd2458622d0efac1 |
| https://github.com/sidlovescoding/Cpython/blob/8817886ae571f5b5ce4e2e6cfd2458622d0efac1/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/8817886ae571f5b5ce4e2e6cfd2458622d0efac1 |
| gh-104719: Restore Tokenize module constants (#104722) | https://github.com/sidlovescoding/Cpython/commit/ffe47cb623999db05959ec4b5168d1c87a1e40ef |
| https://github.com/mgmacias95 |
| mgmacias95 | https://github.com/sidlovescoding/Cpython/commits?author=mgmacias95 |
| ffe47cb | https://github.com/sidlovescoding/Cpython/commit/ffe47cb623999db05959ec4b5168d1c87a1e40ef |
| https://github.com/sidlovescoding/Cpython/blob/ffe47cb623999db05959ec4b5168d1c87a1e40ef/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/ffe47cb623999db05959ec4b5168d1c87a1e40ef |
| gh-102856: Python tokenizer implementation for PEP 701 (#104323) | https://github.com/sidlovescoding/Cpython/commit/6715f91edcf6f379f666e18f57b8a0dcb724bf79 |
| mgmacias95 | https://github.com/sidlovescoding/Cpython/commits?author=mgmacias95 |
| pablogsal | https://github.com/sidlovescoding/Cpython/commits?author=pablogsal |
| 6715f91 | https://github.com/sidlovescoding/Cpython/commit/6715f91edcf6f379f666e18f57b8a0dcb724bf79 |
| https://github.com/sidlovescoding/Cpython/blob/6715f91edcf6f379f666e18f57b8a0dcb724bf79/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/6715f91edcf6f379f666e18f57b8a0dcb724bf79 |
| bpo-46565: `del` loop vars that are leaking into module namespaces (GH-30993) | https://github.com/sidlovescoding/Cpython/commit/0cbdd2131195b0d313762968f604e80a3e65ca9f |
| https://github.com/sobolevn |
| sobolevn | https://github.com/sidlovescoding/Cpython/commits?author=sobolevn |
| 0cbdd21 | https://github.com/sidlovescoding/Cpython/commit/0cbdd2131195b0d313762968f604e80a3e65ca9f |
| https://github.com/sidlovescoding/Cpython/blob/0cbdd2131195b0d313762968f604e80a3e65ca9f/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/0cbdd2131195b0d313762968f604e80a3e65ca9f |
| Add tests for the C tokenizer and expose it as a private module (GH-27924) | https://github.com/sidlovescoding/Cpython/commit/a24676bedcd332dd7e6fa5521d0449206391d190 |
| https://github.com/pablogsal |
| pablogsal | https://github.com/sidlovescoding/Cpython/commits?author=pablogsal |
| a24676b | https://github.com/sidlovescoding/Cpython/commit/a24676bedcd332dd7e6fa5521d0449206391d190 |
| https://github.com/sidlovescoding/Cpython/blob/a24676bedcd332dd7e6fa5521d0449206391d190/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/a24676bedcd332dd7e6fa5521d0449206391d190 |
| bpo-44667: Treat correctly lines ending with comments and no newlines in the Python tokenizer (GH-27499) | https://github.com/sidlovescoding/Cpython/commit/b6bde9fc42aecad5be0457198d17cfe7b481ad79 |
| https://github.com/pablogsal |
| pablogsal | https://github.com/sidlovescoding/Cpython/commits?author=pablogsal |
| b6bde9f | https://github.com/sidlovescoding/Cpython/commit/b6bde9fc42aecad5be0457198d17cfe7b481ad79 |
| https://github.com/sidlovescoding/Cpython/blob/b6bde9fc42aecad5be0457198d17cfe7b481ad79/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/b6bde9fc42aecad5be0457198d17cfe7b481ad79 |
| bpo-43014: Improve performance of tokenize.tokenize by 20-30% | https://github.com/sidlovescoding/Cpython/commit/15bd9efd01e44087664e78bf766865a6d2e06626 |
| https://github.com/asottile |
| asottile | https://github.com/sidlovescoding/Cpython/commits?author=asottile |
| 15bd9ef | https://github.com/sidlovescoding/Cpython/commit/15bd9efd01e44087664e78bf766865a6d2e06626 |
| https://github.com/sidlovescoding/Cpython/blob/15bd9efd01e44087664e78bf766865a6d2e06626/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/15bd9efd01e44087664e78bf766865a6d2e06626 |
| bpo-5028: Fix up rest of documentation for tokenize documenting line (GH-13686) | https://github.com/sidlovescoding/Cpython/commit/2a58b0636d1f620f8a85a2e4c030cc10551936a5 |
| asottile | https://github.com/sidlovescoding/Cpython/commits?author=asottile |
| miss-islington | https://github.com/sidlovescoding/Cpython/commits?author=miss-islington |
| 2a58b06 | https://github.com/sidlovescoding/Cpython/commit/2a58b0636d1f620f8a85a2e4c030cc10551936a5 |
| https://github.com/sidlovescoding/Cpython/blob/2a58b0636d1f620f8a85a2e4c030cc10551936a5/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/2a58b0636d1f620f8a85a2e4c030cc10551936a5 |
| bpo-5028: fix doc bug for tokenize (GH-11683) | https://github.com/sidlovescoding/Cpython/commit/1e36f75d634383eb243aa1798c0f2405c9ceb5d4 |
| andrewnc | https://github.com/sidlovescoding/Cpython/commits?author=andrewnc |
| miss-islington | https://github.com/sidlovescoding/Cpython/commits?author=miss-islington |
| 1e36f75 | https://github.com/sidlovescoding/Cpython/commit/1e36f75d634383eb243aa1798c0f2405c9ceb5d4 |
| https://github.com/sidlovescoding/Cpython/blob/1e36f75d634383eb243aa1798c0f2405c9ceb5d4/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/1e36f75d634383eb243aa1798c0f2405c9ceb5d4 |
| bpo-36766: Typos in docs and code comments (GH-13116) | https://github.com/sidlovescoding/Cpython/commit/964663089547ca110199e23867b46b07ff4be88c |
| GiantRobato | https://github.com/sidlovescoding/Cpython/commits?author=GiantRobato |
| matrixise | https://github.com/sidlovescoding/Cpython/commits?author=matrixise |
| 9646630 | https://github.com/sidlovescoding/Cpython/commit/964663089547ca110199e23867b46b07ff4be88c |
| https://github.com/sidlovescoding/Cpython/blob/964663089547ca110199e23867b46b07ff4be88c/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/964663089547ca110199e23867b46b07ff4be88c |
| bpo-30455: Generate all token related code and docs from Grammar/Tokens. (GH-10370) | https://github.com/sidlovescoding/Cpython/commit/8ac658114dec4964479baecfbc439fceb40eaa79 |
| https://github.com/serhiy-storchaka |
| serhiy-storchaka | https://github.com/sidlovescoding/Cpython/commits?author=serhiy-storchaka |
| 8ac6581 | https://github.com/sidlovescoding/Cpython/commit/8ac658114dec4964479baecfbc439fceb40eaa79 |
| https://github.com/sidlovescoding/Cpython/blob/8ac658114dec4964479baecfbc439fceb40eaa79/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/8ac658114dec4964479baecfbc439fceb40eaa79 |
| bpo-33899: Make tokenize module mirror end-of-file is end-of-line behavior (GH-7891) | https://github.com/sidlovescoding/Cpython/commit/c4ef4896eac86a6759901c8546e26de4695a1389 |
| ammaraskar | https://github.com/sidlovescoding/Cpython/commits?author=ammaraskar |
| taleinat | https://github.com/sidlovescoding/Cpython/commits?author=taleinat |
| c4ef489 | https://github.com/sidlovescoding/Cpython/commit/c4ef4896eac86a6759901c8546e26de4695a1389 |
| https://github.com/sidlovescoding/Cpython/blob/c4ef4896eac86a6759901c8546e26de4695a1389/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/c4ef4896eac86a6759901c8546e26de4695a1389 |
| bpo-12486: Document tokenize.generate_tokens() as public API (#6957) | https://github.com/sidlovescoding/Cpython/commit/c56b17bd8c7a3fd03859822246633d2c9586f8bd |
| takluyver | https://github.com/sidlovescoding/Cpython/commits?author=takluyver |
| willingc | https://github.com/sidlovescoding/Cpython/commits?author=willingc |
| c56b17b | https://github.com/sidlovescoding/Cpython/commit/c56b17bd8c7a3fd03859822246633d2c9586f8bd |
| https://github.com/sidlovescoding/Cpython/blob/c56b17bd8c7a3fd03859822246633d2c9586f8bd/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/c56b17bd8c7a3fd03859822246633d2c9586f8bd |
| bpo-33338: [tokenize] Minor code cleanup (#6573) | https://github.com/sidlovescoding/Cpython/commit/c2d384dbd7c6ed9bdfaac45f05b463263c743ee7 |
| https://github.com/ambv |
| ambv | https://github.com/sidlovescoding/Cpython/commits?author=ambv |
| c2d384d | https://github.com/sidlovescoding/Cpython/commit/c2d384dbd7c6ed9bdfaac45f05b463263c743ee7 |
| https://github.com/sidlovescoding/Cpython/blob/c2d384dbd7c6ed9bdfaac45f05b463263c743ee7/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/c2d384dbd7c6ed9bdfaac45f05b463263c743ee7 |
| bpo-33260: Regenerate token.py after removing ASYNC and AWAIT. (GH-6447) | https://github.com/sidlovescoding/Cpython/commit/d08972fdb92cad6b813d22205752c97ea18df060 |
| https://github.com/serhiy-storchaka |
| serhiy-storchaka | https://github.com/sidlovescoding/Cpython/commits?author=serhiy-storchaka |
| d08972f | https://github.com/sidlovescoding/Cpython/commit/d08972fdb92cad6b813d22205752c97ea18df060 |
| https://github.com/sidlovescoding/Cpython/blob/d08972fdb92cad6b813d22205752c97ea18df060/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/d08972fdb92cad6b813d22205752c97ea18df060 |
| bpo-30406: Make async and await proper keywords (#1669) | https://github.com/sidlovescoding/Cpython/commit/ac317700ce7439e38a8b420218d9a5035bba92ed |
| JelleZijlstra | https://github.com/sidlovescoding/Cpython/commits?author=JelleZijlstra |
| ac31770 | https://github.com/sidlovescoding/Cpython/commit/ac317700ce7439e38a8b420218d9a5035bba92ed |
| https://github.com/sidlovescoding/Cpython/blob/ac317700ce7439e38a8b420218d9a5035bba92ed/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/ac317700ce7439e38a8b420218d9a5035bba92ed |
| bpo-25324: copy tok_name before changing it (#1608) | https://github.com/sidlovescoding/Cpython/commit/fc354f07855a9197e71f851ad930cbf5652f9160 |
| albertjan | https://github.com/sidlovescoding/Cpython/commits?author=albertjan |
| vstinner | https://github.com/sidlovescoding/Cpython/commits?author=vstinner |
| fc354f0 | https://github.com/sidlovescoding/Cpython/commit/fc354f07855a9197e71f851ad930cbf5652f9160 |
| https://github.com/sidlovescoding/Cpython/blob/fc354f07855a9197e71f851ad930cbf5652f9160/Lib/tokenize.py |
| https://github.com/sidlovescoding/Cpython/tree/fc354f07855a9197e71f851ad930cbf5652f9160 |
| Next | https://github.com/sidlovescoding/Cpython/commits/sid-python/Lib/tokenize.py?after=285abce9202534d8e8c7f3624b0cc776d636ff4f+34 |
|
| https://github.com |
| Terms | https://docs.github.com/site-policy/github-terms/github-terms-of-service |
| Privacy | https://docs.github.com/site-policy/privacy-policies/github-privacy-statement |
| Security | https://github.com/security |
| Status | https://www.githubstatus.com/ |
| Community | https://github.community/ |
| Docs | https://docs.github.com/ |
| Contact | https://support.github.com?tags=dotcom-footer |