René's URL Explorer Experiment


Title: gh-131507: Add support for syntax highlighting in PyREPL by ambv · Pull Request #133247 · python/cpython · GitHub

Open Graph Title: gh-131507: Add support for syntax highlighting in PyREPL by ambv · Pull Request #133247 · python/cpython

X Title: gh-131507: Add support for syntax highlighting in PyREPL by ambv · Pull Request #133247 · python/cpython

Description: This is a much improved version of gh-131562. It uses the tokenizer for better speed and pattern matching for more robust handling of soft keywords. While it still won't hit all cases correctly, it's better than idlelib's regular expression-based colorizer and unlike our glorious PEG parser it supports incomplete input, which is crucial for an interactive shell. Relatedly, pasting support was tweaked to be way faster. Now the entire contents of Frankenstein can be pasted within 3 seconds both on Unix and Windows as long as bracketed pasting is supported by the terminal. This is a necessary tweak for syntax highlighting not to cripple performance of pastes above 5kB. There is experimental support for theming through _colorize.set_theme() that's mentioned in "What's New" but otherwise undocumented so far. Issue: gh-131507 📚 Documentation preview 📚: https://cpython-previews--133247.org.readthedocs.build/

Open Graph Description: This is a much improved version of gh-131562. It uses the tokenizer for better speed and pattern matching for more robust handling of soft keywords. While it still won't hit all cases correctly...

X Description: This is a much improved version of gh-131562. It uses the tokenizer for better speed and pattern matching for more robust handling of soft keywords. While it still won't hit all cases corre...

Opengraph URL: https://github.com/python/cpython/pull/133247

X: @github

direct link

Domain: github.com

route-pattern/:user_id/:repository/pull/:id/commits/:range(.:format)
route-controllerpull_requests
route-actioncommits
fetch-noncev2:eed26432-7b22-23ac-52bc-1263a24a4b78
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idD88A:B8D80:4A96739:666E136:6995EAF7
html-safe-nonce24daa14fcf3be075b8cad661d49978e0394aeec0430b6566026bbb32f43a4546
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJEODhBOkI4RDgwOjRBOTY3Mzk6NjY2RTEzNjo2OTk1RUFGNyIsInZpc2l0b3JfaWQiOiI5MjE2NzU4MTMyNTg3NzUyMTgzIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0=
visitor-hmac387de0e81526b66a4a55ad4882f80e3864697c01dc88d843ca0b23738358af2b
hovercard-subject-tagpull_request:2493591184
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/commits
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6
twitter:imagehttps://avatars.githubusercontent.com/u/55281?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/55281?s=400&v=4
og:image:altThis is a much improved version of gh-131562. It uses the tokenizer for better speed and pattern matching for more robust handling of soft keywords. While it still won't hit all cases correctly...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None1161b1e91d57e776171599ebe7c3b8c4690792d1e87e83b5c77a9b1aeb391094
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/python/cpython git https://github.com/python/cpython.git
octolytics-dimension-user_id1525981
octolytics-dimension-user_loginpython
octolytics-dimension-repository_id81598961
octolytics-dimension-repository_nwopython/cpython
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id81598961
octolytics-dimension-repository_network_root_nwopython/cpython
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
releasea8c6902042885fe21ac3491e553d741a50765a66
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6#start-of-content
https://github.com/
Sign in https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fpull%2F133247%2Fcommits%2Fb1f25575410086cd7fff0ae4881c5b3b360006a6
GitHub CopilotWrite better code with AIhttps://github.com/features/copilot
GitHub SparkBuild and deploy intelligent appshttps://github.com/features/spark
GitHub ModelsManage and compare promptshttps://github.com/features/models
MCP RegistryNewIntegrate external toolshttps://github.com/mcp
ActionsAutomate any workflowhttps://github.com/features/actions
CodespacesInstant dev environmentshttps://github.com/features/codespaces
IssuesPlan and track workhttps://github.com/features/issues
Code ReviewManage code changeshttps://github.com/features/code-review
GitHub Advanced SecurityFind and fix vulnerabilitieshttps://github.com/security/advanced-security
Code securitySecure your code as you buildhttps://github.com/security/advanced-security/code-security
Secret protectionStop leaks before they starthttps://github.com/security/advanced-security/secret-protection
Why GitHubhttps://github.com/why-github
Documentationhttps://docs.github.com
Bloghttps://github.blog
Changeloghttps://github.blog/changelog
Marketplacehttps://github.com/marketplace
View all featureshttps://github.com/features
Enterpriseshttps://github.com/enterprise
Small and medium teamshttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
App Modernizationhttps://github.com/solutions/use-case/app-modernization
DevSecOpshttps://github.com/solutions/use-case/devsecops
DevOpshttps://github.com/solutions/use-case/devops
CI/CDhttps://github.com/solutions/use-case/ci-cd
View all use caseshttps://github.com/solutions/use-case
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
View all industrieshttps://github.com/solutions/industry
View all solutionshttps://github.com/solutions
AIhttps://github.com/resources/articles?topic=ai
Software Developmenthttps://github.com/resources/articles?topic=software-development
DevOpshttps://github.com/resources/articles?topic=devops
Securityhttps://github.com/resources/articles?topic=security
View all topicshttps://github.com/resources/articles
Customer storieshttps://github.com/customer-stories
Events & webinarshttps://github.com/resources/events
Ebooks & reportshttps://github.com/resources/whitepapers
Business insightshttps://github.com/solutions/executive-insights
GitHub Skillshttps://skills.github.com
Documentationhttps://docs.github.com
Customer supporthttps://support.github.com
Community forumhttps://github.com/orgs/community/discussions
Trust centerhttps://github.com/trust-center
Partnershttps://github.com/partners
GitHub SponsorsFund open source developershttps://github.com/sponsors
Security Labhttps://securitylab.github.com
Maintainer Communityhttps://maintainers.github.com
Acceleratorhttps://github.com/accelerator
Archive Programhttps://archiveprogram.github.com
Topicshttps://github.com/topics
Trendinghttps://github.com/trending
Collectionshttps://github.com/collections
Enterprise platformAI-powered developer platformhttps://github.com/enterprise
GitHub Advanced SecurityEnterprise-grade security featureshttps://github.com/security/advanced-security
Copilot for BusinessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/premium-support
Pricinghttps://github.com/pricing
Search syntax tipshttps://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
documentationhttps://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
Sign in https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fpull%2F133247%2Fcommits%2Fb1f25575410086cd7fff0ae4881c5b3b360006a6
Sign up https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Fcommits&source=header-repo&source_repo=python%2Fcpython
Reloadhttps://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6
Reloadhttps://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6
Reloadhttps://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6
python https://github.com/python
cpythonhttps://github.com/python/cpython
Please reload this pagehttps://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6
Notifications https://github.com/login?return_to=%2Fpython%2Fcpython
Fork 34.1k https://github.com/login?return_to=%2Fpython%2Fcpython
Star 71.6k https://github.com/login?return_to=%2Fpython%2Fcpython
Code https://github.com/python/cpython
Issues 5k+ https://github.com/python/cpython/issues
Pull requests 2.1k https://github.com/python/cpython/pulls
Actions https://github.com/python/cpython/actions
Projects 31 https://github.com/python/cpython/projects
Security 0 https://github.com/python/cpython/security
Insights https://github.com/python/cpython/pulse
Code https://github.com/python/cpython
Issues https://github.com/python/cpython/issues
Pull requests https://github.com/python/cpython/pulls
Actions https://github.com/python/cpython/actions
Projects https://github.com/python/cpython/projects
Security https://github.com/python/cpython/security
Insights https://github.com/python/cpython/pulse
Sign up for GitHub https://github.com/signup?return_to=%2Fpython%2Fcpython%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fpython%2Fcpython%2Fissues%2Fnew%2Fchoose
ambvhttps://github.com/ambv
python:mainhttps://github.com/python/cpython/tree/main
ambv:pyrepl-syntax-highlighting-tokenshttps://github.com/ambv/cpython/tree/pyrepl-syntax-highlighting-tokens
Conversation 59 https://github.com/python/cpython/pull/133247
Commits 23 https://github.com/python/cpython/pull/133247/commits
Checks 46 https://github.com/python/cpython/pull/133247/checks
Files changed https://github.com/python/cpython/pull/133247/files
Please reload this pagehttps://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6
gh-131507: Add support for syntax highlighting in PyREPL https://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6#top
Show all changes 23 commits https://github.com/python/cpython/pull/133247/files
e921a80 gh-131507: Add support for syntax highlighting in PyREPL ambv Mar 18, 2025 https://github.com/python/cpython/pull/133247/commits/e921a809b2fcb802742eb58d704577ab55c92cba
fb95911 Add Blurb ambv Mar 21, 2025 https://github.com/python/cpython/pull/133247/commits/fb95911d044b4ed85ea244868f133f4a7d908263
b428513 Fix irrelevant Windows tests ambv Mar 21, 2025 https://github.com/python/cpython/pull/133247/commits/b428513fbab0b8513ec58f7ab1cf14633669f5dc
2bdcd06 Replace idlelib.colorizer with a faster solution ambv Apr 29, 2025 https://github.com/python/cpython/pull/133247/commits/2bdcd06c37c5364aa53b9fb334748fdda52a0983
8c70c45 Slurp the entire input buffer before refreshing display during bracke… ambv Apr 29, 2025 https://github.com/python/cpython/pull/133247/commits/8c70c4587287dec51aa1756b0a16d3652a96b428
4d7ae36 Implement getpending() on Windows ambv Apr 29, 2025 https://github.com/python/cpython/pull/133247/commits/4d7ae3658260d973c3da4936131acd854043217c
9585bd6 Adapt tests ambv Apr 29, 2025 https://github.com/python/cpython/pull/133247/commits/9585bd6bb271cfc5362ee162e43be70e1dc42f67
b1f2557 Support soft keywords (fight fire with fire) ambv Apr 30, 2025 https://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6
01e1129 Fix test ambv Apr 30, 2025 https://github.com/python/cpython/pull/133247/commits/01e1129886a1d9b55f9649abe990fea594d35d4b
20eff49 Remove unnecessary import ambv Apr 30, 2025 https://github.com/python/cpython/pull/133247/commits/20eff49682075da3862ae933c4a06e8923122219
8d3648a Add test for prev_next_window ambv Apr 30, 2025 https://github.com/python/cpython/pull/133247/commits/8d3648a080dde06083c0f4bedcf286e6df8c3042
656fea3 Windows: bracketed pasting of 448692 chars done in 2.38s ✊ ambv May 1, 2025 https://github.com/python/cpython/pull/133247/commits/656fea32bc4696f8d741aa818a182e1a07c7f7ab
dac8961 Remove colors from Windows low-level console tests ambv May 1, 2025 https://github.com/python/cpython/pull/133247/commits/dac896144c163a5308108af0faa4c62d498d76d5
7891fa7 Fix lint and stuff ambv May 1, 2025 https://github.com/python/cpython/pull/133247/commits/7891fa7937e62728458f6a06f49acaec94e51310
362a21b Add experimental theming support for syntax highlighting and the prompt ambv May 1, 2025 https://github.com/python/cpython/pull/133247/commits/362a21b324e6b90247e898aa00b83c9323aafbb3
ffebbbe Fix lint ambv May 1, 2025 https://github.com/python/cpython/pull/133247/commits/ffebbbe3d2df946daf1fd68fd0b7800cb9201a29
388e494 Merge branch 'main' into pyrepl-syntax-highlighting-tokens ambv May 1, 2025 https://github.com/python/cpython/pull/133247/commits/388e4945f292e8e75c4d87accdf5e9679f0c863c
9b60382 Add t-string support to syntax highlighting ambv May 1, 2025 https://github.com/python/cpython/pull/133247/commits/9b603826f6063b1deb534fc169f9170be5fe3f84
f835dba Apply suggestions from code review ambv May 2, 2025 https://github.com/python/cpython/pull/133247/commits/f835dba6e548d3e94205ee617c611efcc30c40de
9003d05 Add _colorize.ANSIColors.BOLD ambv May 2, 2025 https://github.com/python/cpython/pull/133247/commits/9003d05eb17ff28da2ffacba01c7479a8fe0fef7
ff1f92b Remove - and + from first sets matching for `match` ambv May 2, 2025 https://github.com/python/cpython/pull/133247/commits/ff1f92bf10c2341b9abfa16071bf42803aac16e3
bd84cd8 Address Pablo's review ambv May 2, 2025 https://github.com/python/cpython/pull/133247/commits/bd84cd89f64614804fb38f10e236fa9823a1fdd5
080f300 Exclude applying soft keywords inside any bracket pairs, not just par… ambv May 2, 2025 https://github.com/python/cpython/pull/133247/commits/080f3006cb01b06821a9df95f3eb0b6d7127e5c6
Clear filters https://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6
Please reload this pagehttps://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6
Please reload this pagehttps://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6
utils.py https://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6#diff-4f8422583c43023de2bd7243efa0263f3ba95c028938501ff8ffb51f67941686
windows_console.py https://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6#diff-77cdde1442e2d3a5fe5ee852304d1af6d0a2033e718dc588552d61596cb72100
Prev https://github.com/python/cpython/pull/133247/commits/9585bd6bb271cfc5362ee162e43be70e1dc42f67
Next https://github.com/python/cpython/pull/133247/commits/01e1129886a1d9b55f9649abe990fea594d35d4b
Please reload this pagehttps://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6
https://github.com/ambv
ambvhttps://github.com/python/cpython/commits?author=ambv
https://github.com/python/cpython/blob/main/.github/CODEOWNERS#L558
Lib/_pyrepl/utils.pyhttps://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6#diff-4f8422583c43023de2bd7243efa0263f3ba95c028938501ff8ffb51f67941686
View file https://github.com/ambv/cpython/blob/b1f25575410086cd7fff0ae4881c5b3b360006a6/Lib/_pyrepl/utils.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/133247/commits/{{ revealButtonHref }}
https://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6#diff-4f8422583c43023de2bd7243efa0263f3ba95c028938501ff8ffb51f67941686
https://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6#diff-4f8422583c43023de2bd7243efa0263f3ba95c028938501ff8ffb51f67941686
https://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6#diff-4f8422583c43023de2bd7243efa0263f3ba95c028938501ff8ffb51f67941686
Please reload this pagehttps://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6
https://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6#diff-4f8422583c43023de2bd7243efa0263f3ba95c028938501ff8ffb51f67941686
https://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6#diff-4f8422583c43023de2bd7243efa0263f3ba95c028938501ff8ffb51f67941686
https://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6#diff-4f8422583c43023de2bd7243efa0263f3ba95c028938501ff8ffb51f67941686
https://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6#diff-4f8422583c43023de2bd7243efa0263f3ba95c028938501ff8ffb51f67941686
https://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6#diff-4f8422583c43023de2bd7243efa0263f3ba95c028938501ff8ffb51f67941686
https://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6#diff-4f8422583c43023de2bd7243efa0263f3ba95c028938501ff8ffb51f67941686
https://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6#diff-4f8422583c43023de2bd7243efa0263f3ba95c028938501ff8ffb51f67941686
https://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6#diff-4f8422583c43023de2bd7243efa0263f3ba95c028938501ff8ffb51f67941686
Please reload this pagehttps://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6
https://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6#diff-4f8422583c43023de2bd7243efa0263f3ba95c028938501ff8ffb51f67941686
Please reload this pagehttps://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6
Please reload this pagehttps://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6
Please reload this pagehttps://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6
https://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6#diff-4f8422583c43023de2bd7243efa0263f3ba95c028938501ff8ffb51f67941686
https://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6#diff-4f8422583c43023de2bd7243efa0263f3ba95c028938501ff8ffb51f67941686
Please reload this pagehttps://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6
https://github.com/python/cpython/blob/main/.github/CODEOWNERS#L558
Lib/_pyrepl/windows_console.pyhttps://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6#diff-77cdde1442e2d3a5fe5ee852304d1af6d0a2033e718dc588552d61596cb72100
View file https://github.com/ambv/cpython/blob/b1f25575410086cd7fff0ae4881c5b3b360006a6/Lib/_pyrepl/windows_console.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/133247/commits/{{ revealButtonHref }}
https://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6#diff-77cdde1442e2d3a5fe5ee852304d1af6d0a2033e718dc588552d61596cb72100
https://github.com/python/cpython/pull/133247/commits/b1f25575410086cd7fff0ae4881c5b3b360006a6#diff-77cdde1442e2d3a5fe5ee852304d1af6d0a2033e718dc588552d61596cb72100
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width

URLs of crawlers that visited me.