René's URL Explorer Experiment


Title: Lexer: extract matches with PREG_PATTERN_ORDER (~40% faster lexing) by JanTvrdik · Pull Request #313 · phpstan/phpdoc-parser · GitHub

Open Graph Title: Lexer: extract matches with PREG_PATTERN_ORDER (~40% faster lexing) by JanTvrdik · Pull Request #313 · phpstan/phpdoc-parser

X Title: Lexer: extract matches with PREG_PATTERN_ORDER (~40% faster lexing) by JanTvrdik · Pull Request #313 · phpstan/phpdoc-parser

Description: Lexer::tokenize() spends most of its time building match arrays that it throws away again. Two changes, which only pay off together, take ~40% off lexing and ~19% off end-to-end PHPDoc parsing. No API change: tokenize() still returns list. 1. The numeric patterns hold 12 capturing groups nobody reads TOKEN_FLOAT and TOKEN_INTEGER use (_[0-9]++)* for the _ digit separators — 12 capturing groups between them. Nothing reads them, but preg_match_all() collects every group of the pattern for every token it finds, so each token carries 12 unused entries. Made them non-capturing. 2. PREG_SET_ORDER allocates one array per token With the groups gone, PREG_SET_ORDER still allocates a two-entry array per match — the whole match and the MARK — which tokenize() reads out and drops immediately. PREG_PATTERN_ORDER instead collects the whole PHPDoc into two arrays: $matches[0] for the values, $matches['MARK'] for the token types. Two array headers per PHPDoc rather than one per token. The two are multiplicative rather than additive, which is why they are one PR: lexing baseline — non-capturing groups only −7% PREG_PATTERN_ORDER only (groups left capturing) −7% both −41% PREG_PATTERN_ORDER with capturing groups present is no better than the status quo, because it then has to build one array per group of one entry per token. That makes a capturing group in a token pattern a silent 35-percentage-point regression, so LexerTest::testTokenPatternsHaveNoCapturingGroups() pins it. Numbers Corpus: 37 150 unique docblocks, 8.38 MiB, 2 355 859 tokens, collected by tokenizing 10 001 PHP files and keeping every unique T_DOC_COMMENT — 20 OSS libraries, one full vendor tree (nikic/php-parser, nette, symfony, react, phpunit, jetbrains/phpstorm-stubs, doctrine, twig), phpstan-src/src, and this library. Pinned to one core, php -n, minimum of 15 rounds with the variants interleaved. lexing parse (lex + PhpDocParser) PHP 8.1 −42% PHP 8.4 −40% PHP 8.5 −40% −19% PHP 8.5, lines+indexes+comments −19% PHP 8.5, opcache + tracing JIT −45% −23% Lexing is ~41% of total parse time in the baseline, which is why a lexer-only change moves the end-to-end number this much. Behaviour Tokens are byte-identical to before, and so are the ASTs printed from them, over all 37 150 docblocks (xxh128 over both). make check passes on PHP 8.5: 1750 tests, phpcs, PHPStan. The only behavioural edge case is a subject that produces zero matches — an empty string — where $matches['MARK'] is absent; that returns just the TOKEN_END token, as before. Not in this PR While benchmarking I also tried changing the token representation, in case it is of interest for a future major: lexing parse bytes/token Token objects instead of tuples +21% +8% 164 (−39%) struct-of-arrays TokenList −56% −31% 124 (−54%) Token objects trade 21% lexing time for 39% less memory (new Token(...) is a userland call where the array literal is one opcode; the parser does get ~2% faster from reading properties instead of indexing arrays, but not enough). Three parallel packed arrays win on both axes, because $matches[0] becomes the values array with no per-token allocation at all — but that one changes the return type of tokenize(), so PHPStan would have to follow. Happy to open an issue with the details if either is worth discussing.

Open Graph Description: Lexer::tokenize() spends most of its time building match arrays that it throws away again. Two changes, which only pay off together, take ~40% off lexing and ~19% off end-to-end PHPDoc parsing. No ...

X Description: Lexer::tokenize() spends most of its time building match arrays that it throws away again. Two changes, which only pay off together, take ~40% off lexing and ~19% off end-to-end PHPDoc parsing. No ...

Opengraph URL: https://github.com/phpstan/phpdoc-parser/pull/313

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:fb2137f9-286f-fe5f-cd5a-b8d66b0769dd
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idD02A:1D4C16:3EEA77:5622D5:6B006E41
html-safe-nonce2bf5204b757205a9504dd696ab308a1807d76fdd94abd22c8cac84d0f90a381e
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJEMDJBOjFENEMxNjozRUVBNzc6NTYyMkQ1OjZCMDA2RTQxIiwidmlzaXRvcl9pZCI6IjQ3NTg3Njk0NDMwMTY4MzA1MzAiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmacc24972d3afcaf13339fd56070e1f2e949a6d8a26f865ccf76b0d0e97fd5fca13
hovercard-subject-tagpull_request:4403378390
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/phpstan/phpdoc-parser/pull/313/files
twitter:imagehttps://avatars.githubusercontent.com/u/175109?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/175109?s=400&v=4
og:image:altLexer::tokenize() spends most of its time building match arrays that it throws away again. Two changes, which only pay off together, take ~40% off lexing and ~19% off end-to-end PHPDoc parsing. No ...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None6527e91c60c0b29d16606ff32ac5df8ecdc37a2a62205ea057a14d7762c870fa
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/phpstan/phpdoc-parser git https://github.com/phpstan/phpdoc-parser.git
octolytics-dimension-user_id16435710
octolytics-dimension-user_loginphpstan
octolytics-dimension-repository_id111290081
octolytics-dimension-repository_nwophpstan/phpdoc-parser
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id111290081
octolytics-dimension-repository_network_root_nwophpstan/phpdoc-parser
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release46f1cd2ae3cd8ae1a81019b5d619742e76f13bbd
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/phpstan/phpdoc-parser/pull/313/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fphpstan%2Fphpdoc-parser%2Fpull%2F313%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fphpstan%2Fphpdoc-parser%2Fpull%2F313%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=phpstan%2Fphpdoc-parser
Reloadhttps://github.com/phpstan/phpdoc-parser/pull/313/files
Reloadhttps://github.com/phpstan/phpdoc-parser/pull/313/files
Reloadhttps://github.com/phpstan/phpdoc-parser/pull/313/files
phpstanhttps://github.com/phpstan
phpdoc-parserhttps://github.com/phpstan/phpdoc-parser
Codehttps://github.com/phpstan/phpdoc-parser
Issues7 (7)https://github.com/phpstan/phpdoc-parser/issues
Pull requests14 (14)https://github.com/phpstan/phpdoc-parser/pulls
Actionshttps://github.com/phpstan/phpdoc-parser/actions
Projectshttps://github.com/phpstan/phpdoc-parser/projects
Security and qualityhttps://github.com/phpstan/phpdoc-parser/security
Insightshttps://github.com/phpstan/phpdoc-parser/insights
Sign up for GitHub https://github.com/signup?return_to=%2Fphpstan%2Fphpdoc-parser%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fphpstan%2Fphpdoc-parser%2Fissues%2Fnew%2Fchoose
ondrejmirteshttps://github.com/ondrejmirtes
2.3.xhttps://github.com/phpstan/phpdoc-parser/tree/2.3.x
lexer-preg-pattern-orderhttps://github.com/phpstan/phpdoc-parser/tree/lexer-preg-pattern-order
Conversation 4 https://github.com/phpstan/phpdoc-parser/pull/313
Commits 2 https://github.com/phpstan/phpdoc-parser/pull/313/commits
Checks 61 https://github.com/phpstan/phpdoc-parser/pull/313/checks
Files changed https://github.com/phpstan/phpdoc-parser/pull/313/files
Please reload this pagehttps://github.com/phpstan/phpdoc-parser/pull/313/files
Lexer: extract matches with PREG_PATTERN_ORDER (~40% faster lexing) https://github.com/phpstan/phpdoc-parser/pull/313/files#top
Show all changes 2 commits https://github.com/phpstan/phpdoc-parser/pull/313/files
8fdbe8d Lexer: make the numeric-separator groups non-capturing JanTvrdik Aug 31, 2026 https://github.com/phpstan/phpdoc-parser/pull/313/commits/8fdbe8d1c278fe561129ba5c170657df16c19519
735aee4 Lexer: extract the matches with PREG_PATTERN_ORDER JanTvrdik Aug 31, 2026 https://github.com/phpstan/phpdoc-parser/pull/313/commits/735aee44cd17e0a13efdb271850abce679cc1d8d
Clear filters https://github.com/phpstan/phpdoc-parser/pull/313/files
Please reload this pagehttps://github.com/phpstan/phpdoc-parser/pull/313/files
Please reload this pagehttps://github.com/phpstan/phpdoc-parser/pull/313/files
Lexer.php https://github.com/phpstan/phpdoc-parser/pull/313/files#diff-5b2a6ad326ef6352f776b19087e035f332c2c60673c6ca09c0c06fb4080e9097
LexerTest.php https://github.com/phpstan/phpdoc-parser/pull/313/files#diff-585a1994d68e3d879132da8c0f7e91ae0801ff5e56a2baa42d204f34392bc0a8
src/Lexer/Lexer.phphttps://github.com/phpstan/phpdoc-parser/pull/313/files#diff-5b2a6ad326ef6352f776b19087e035f332c2c60673c6ca09c0c06fb4080e9097
View file https://github.com/phpstan/phpdoc-parser/blob/735aee44cd17e0a13efdb271850abce679cc1d8d/src/Lexer/Lexer.php
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/phpstan/phpdoc-parser/pull/313/{{ revealButtonHref }}
https://github.com/phpstan/phpdoc-parser/pull/313/files#diff-5b2a6ad326ef6352f776b19087e035f332c2c60673c6ca09c0c06fb4080e9097
https://github.com/phpstan/phpdoc-parser/pull/313/files#diff-5b2a6ad326ef6352f776b19087e035f332c2c60673c6ca09c0c06fb4080e9097
https://github.com/phpstan/phpdoc-parser/pull/313/files#diff-5b2a6ad326ef6352f776b19087e035f332c2c60673c6ca09c0c06fb4080e9097
https://github.com/phpstan/phpdoc-parser/pull/313/files#diff-5b2a6ad326ef6352f776b19087e035f332c2c60673c6ca09c0c06fb4080e9097
https://github.com/phpstan/phpdoc-parser/pull/313/files#diff-5b2a6ad326ef6352f776b19087e035f332c2c60673c6ca09c0c06fb4080e9097
https://github.com/phpstan/phpdoc-parser/pull/313/files#diff-5b2a6ad326ef6352f776b19087e035f332c2c60673c6ca09c0c06fb4080e9097
https://github.com/phpstan/phpdoc-parser/pull/313/files#diff-5b2a6ad326ef6352f776b19087e035f332c2c60673c6ca09c0c06fb4080e9097
tests/PHPStan/Lexer/LexerTest.phphttps://github.com/phpstan/phpdoc-parser/pull/313/files#diff-585a1994d68e3d879132da8c0f7e91ae0801ff5e56a2baa42d204f34392bc0a8
View file https://github.com/phpstan/phpdoc-parser/blob/735aee44cd17e0a13efdb271850abce679cc1d8d/tests/PHPStan/Lexer/LexerTest.php
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/phpstan/phpdoc-parser/pull/313/{{ revealButtonHref }}
Please reload this pagehttps://github.com/phpstan/phpdoc-parser/pull/313/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.