Title: Base implementation of `_tokenize` module by ShaharNaveh · Pull Request #6240 · RustPython/RustPython · GitHub
Open Graph Title: Base implementation of `_tokenize` module by ShaharNaveh · Pull Request #6240 · RustPython/RustPython
X Title: Base implementation of `_tokenize` module by ShaharNaveh · Pull Request #6240 · RustPython/RustPython
Description: A Python Interpreter written in Rust. Contribute to RustPython/RustPython development by creating an account on GitHub.
Open Graph Description: Summary by CodeRabbit New Features New Python tokenizer module: Tokenizer iterator that yields token tuples (type, text, start, end, line), accepts a readline callable, optional encoding, and an...
X Description: Summary by CodeRabbit New Features New Python tokenizer module: Tokenizer iterator that yields token tuples (type, text, start, end, line), accepts a readline callable, optional encoding, and an...
Opengraph URL: https://github.com/RustPython/RustPython/pull/6240
X: @github
Domain: patch-diff.githubusercontent.com
| route-pattern | /:user_id/:repository/pull/:id/commits(.:format) |
| route-controller | pull_requests |
| route-action | commits |
| fetch-nonce | v2:c60f66ba-3064-1775-dd18-36a3fc6003bb |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | C6EA:2ACEE1:1182F66:17C5F63:69700457 |
| html-safe-nonce | 725e374a5b25ea60c38658c947e2491392fa6ca3deb0cbbf15ef745418c551ec |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDNkVBOjJBQ0VFMToxMTgyRjY2OjE3QzVGNjM6Njk3MDA0NTciLCJ2aXNpdG9yX2lkIjoiODUzMzM5NTk3NDk5Njg4NjYxNSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 73f2669dbcd59b87199eb06b1651a1155a26ceb5964515090c67f5259e49a6f7 |
| hovercard-subject-tag | pull_request:2991311673 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/RustPython/RustPython/pull/6240/commits |
| twitter:image | https://opengraph.githubassets.com/fe4a6308704ff03afdf7443a9f44e4c7e0d3e9f957b8144ccf6fb6e87568bfe5/RustPython/RustPython/pull/6240 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/fe4a6308704ff03afdf7443a9f44e4c7e0d3e9f957b8144ccf6fb6e87568bfe5/RustPython/RustPython/pull/6240 |
| og:image:alt | Summary by CodeRabbit New Features New Python tokenizer module: Tokenizer iterator that yields token tuples (type, text, start, end, line), accepts a readline callable, optional encoding, and an... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | ShaharNaveh |
| hostname | github.com |
| expected-hostname | github.com |
| None | 0366807b865cee6776368231232f84d6c8096e6bce43f701a4fb28ea795ec427 |
| turbo-cache-control | no-cache |
| go-import | github.com/RustPython/RustPython git https://github.com/RustPython/RustPython.git |
| octolytics-dimension-user_id | 39710557 |
| octolytics-dimension-user_login | RustPython |
| octolytics-dimension-repository_id | 135201145 |
| octolytics-dimension-repository_nwo | RustPython/RustPython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 135201145 |
| octolytics-dimension-repository_network_root_nwo | RustPython/RustPython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 33f356bb2fb58726ccb2f26395bf8ddc9a2d9eaa |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width