Title: bpo-33338: [lib2to3] Synchronize token.py and tokenize.py with stdlib by ambv · Pull Request #6572 · python/cpython · GitHub
Open Graph Title: bpo-33338: [lib2to3] Synchronize token.py and tokenize.py with stdlib by ambv · Pull Request #6572 · python/cpython
X Title: bpo-33338: [lib2to3] Synchronize token.py and tokenize.py with stdlib by ambv · Pull Request #6572 · python/cpython
Description: (This is Step 1 in BPO-33337. See there for larger context.) lib2to3's token.py and tokenize.py were initially copies of the respective files from the standard library. They were copied to allow Python 3 to read Python 2's grammar. Since 2006, lib2to3 grew to be widely used as a Concrete Syntax Tree, also for parsing Python 3 code. Additions to support Python 3 grammar were added but sadly, the main token.py and tokenize.py diverged. This change brings them back together, minimizing the differences to the bare minimum that is in fact required by lib2to3. Before this change, almost every line in lib2to3/pgen2/tokenize.py was different from tokenize.py. After this change, the diff between the two files is only 200 lines long and is entirely filled with relevant Python 2 compatibility bits. Merging the implementations, there's numerous fixes to the lib2to3 tokenizer: docstrings made as similar as possible ported TokenInfo ported tokenize.tokenize() and tokenize.open() removed Python 2-only implementation cruft made Unicode identifier handling the same made string prefix handling the same added Ellipsis to the Special group Untokenizer backported bugfixes: 5e6db31 9dc3a36 5b8d2c3 e411b66 BPO-2495 detect_encoding tries to figure out a filename and find_cookie uses the filename in error messages, if available find_cookie bugfix: BPO-14990 BPO-16152: tokenizer doesn't crash on missing newline at the end of the stream (added \Z (end of string) to PseudoExtras) Improvements to token.py: taken from the current Lib/token.py tokens renumbered to match Lib/token.py __all__ properly defined ASYNC, AWAIT and BACKQUOTE exist under different numbers (100 + old number) ELLIPSIS added ENCODING added https://bugs.python.org/issue33338
Open Graph Description: (This is Step 1 in BPO-33337. See there for larger context.) lib2to3's token.py and tokenize.py were initially copies of the respective files from the standard library. They were copied to all...
X Description: (This is Step 1 in BPO-33337. See there for larger context.) lib2to3's token.py and tokenize.py were initially copies of the respective files from the standard library. They were copied to...
Opengraph URL: https://github.com/python/cpython/pull/6572
X: @github
Domain: github.com
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:8bae1a88-ea03-2b68-58bc-2f52a1c7fe00 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | EDFC:67485:C1C2E:10651B:6A5358B5 |
| html-safe-nonce | f719ce58bb0f5c81dda7aca5f7075fb2bf36683e1b8d8d56a801aff28a24cd43 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJFREZDOjY3NDg1OkMxQzJFOjEwNjUxQjo2QTUzNThCNSIsInZpc2l0b3JfaWQiOiI2NjYxMTA3OTc1NjM2MzQyOTY1IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | 9a18590b04eddf0c5a2b382a2ccd446c0a3039482bca8296980d5d57eab04e70 |
| hovercard-subject-tag | pull_request:183305702 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/python/cpython/pull/6572/files |
| twitter:image | https://avatars.githubusercontent.com/u/55281?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/55281?s=400&v=4 |
| og:image:alt | (This is Step 1 in BPO-33337. See there for larger context.) lib2to3's token.py and tokenize.py were initially copies of the respective files from the standard library. They were copied to all... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | b9a586c06a05a7a86fc7e3f4dbd03e42f6869085879aa184aa6369456dbd50fb |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 07a982c1d40157c619b364352b704c3ce66bb332 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width