Title: bpo-12486: Document tokenize.generate_tokens() as public API by takluyver · Pull Request #6957 · python/cpython · GitHub
Open Graph Title: bpo-12486: Document tokenize.generate_tokens() as public API by takluyver · Pull Request #6957 · python/cpython
X Title: bpo-12486: Document tokenize.generate_tokens() as public API by takluyver · Pull Request #6957 · python/cpython
Description: I think discussions of tokenizing unicode have often got caught up in trying to design nicer APIs for it. I am taking the much lazier approach of promoting the existing but undocumented generate_tokens API to a public API. Why should we do this? generate_tokens() has been there for several releases already (at least since 3.3.0), so we can start using it without having to wait until a new Python version is widespread. Someone (ahem 👼 ) might already be using it - IDE tab completion doesn't know that it's not public, and there's no other publicly documented way to tokenize code that's already unicode. Although the naming is not very clear, the interface is a close parallel to tokenize(), so it's easy to understand. We can do this without having to design, implement, review and debug any new interface! https://bugs.python.org/issue12486
Open Graph Description: I think discussions of tokenizing unicode have often got caught up in trying to design nicer APIs for it. I am taking the much lazier approach of promoting the existing but undocumented generate_to...
X Description: I think discussions of tokenizing unicode have often got caught up in trying to design nicer APIs for it. I am taking the much lazier approach of promoting the existing but undocumented generate_to...
Opengraph URL: https://github.com/python/cpython/pull/6957
X: @github
Domain: github.com
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:52a8a020-39bf-6ec4-6f85-34c0f76d0b9a |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | 98B4:18BA88:A89D441:ECAA653:6A50F1D5 |
| html-safe-nonce | 7aad065821c474b97a8ca9a23690b18d32d478fc7cab297e4dffe3ac2402064c |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5OEI0OjE4QkE4ODpBODlENDQxOkVDQUE2NTM6NkE1MEYxRDUiLCJ2aXNpdG9yX2lkIjoiNTQ0NDA2NzIyMjA5NDkzNDQ4NSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | a0e3b09fd3365bc0b840c35383ed64ed8fa763851dacdc8067fce2e4bddb87b0 |
| hovercard-subject-tag | pull_request:188855212 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/python/cpython/pull/6957/files |
| twitter:image | https://avatars.githubusercontent.com/u/327925?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/327925?s=400&v=4 |
| og:image:alt | I think discussions of tokenizing unicode have often got caught up in trying to design nicer APIs for it. I am taking the much lazier approach of promoting the existing but undocumented generate_to... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 5266e58c17a510c403505cc811606465e90a881d2007ee7df1c4800d5c659838 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 5ec60191a48933536a90c8a19f47142fcd0d4739 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width