René's URL Explorer Experiment


Title: token stream expander by danmar · Pull Request #758 · cppcheck-opensource/simplecpp · GitHub

Open Graph Title: token stream expander by danmar · Pull Request #758 · cppcheck-opensource/simplecpp

X Title: token stream expander by danmar · Pull Request #758 · cppcheck-opensource/simplecpp

Description: It is based on TokenStreamExpander and on the f(f)(1) result I checked against gcc earlier. I did not trace the longer DEFER/EXPAND example, so the description only states its result. The algorithm: TokenStreamExpander This follows the rescan rules of the standard (C11 6.10.3.4, C++ [cpp.rescan]) directly: the result of a macro is scanned again together with the rest of the source, and a macro name found while its own expansion is being scanned is not expanded again. The old code tracked this with an expandedmacros set copied down the call stack, "expanded from" markers on tokens, and the rule "an expansion that ends with a function-like macro name picks up arguments from the source" implemented in three places. Here there is one input stream and one rule. Data Name Meaning Input::pending Tokens that are scanned again. The next token to read is at the front. Input::raw The next token of the source. It is only read when a macro call needs its arguments. End-of-expansion marker A special token ("\x01", it cannot occur in source) that is placed after the result of a macro. Token::macro is the name of the macro that ends. disabled std::map: the number of active expansions of each macro. A macro is disabled when the count is not 0. Token::painted The name has been found while its macro was disabled. It is never expanded, also not later. Expanding one macro call expand(output, rawtok) starts with the macro name as the onlandInput`: while there are pending tokens: tok = read() # a name that is read whpainted if tok is not a macro, or is painted: output tok; continue if the macro is function-like and the next token is not '(': output tok; continue # not a call if the macro is built-in (__FILE__, __LINE__, __COUNTER__): output its value; continue read the call (tokens from pending first, then the source) if the call is not terminated: put the tokens after the name back; output tok; continue result = substitute(macro, call) push result in front of pending push a marker for the macro after the result, and disable Reading a marker enables its macro again (count - 1). Markers in front of the next token are read by hasPending(), peek() and read(), so the macro is enabled again at exactly the point where its replacement has b Substitution Arguments are collected as written, and then each argumand on its own** (an Input with only that argument's tokens).Both versions are kept. The replacement list is walked, with the rule from the stan operand of # → stringified from the argument as written operand of ## → the argument as written anywhere else → the expanded argument ## pastes the last token of the result with the first tokperand is a placemarker. The checks for invalid pasting (invalid token, universal character name, etc.) are the ones that Macro already had. __VA_OPT__(...) is replaced by its content, or by nothingexpand to nothing (C23 6.10.5.1). #__VA_OPT__ and ##__VA_OPT__ are handled as # and ## of that result. defined X and defined(X) in a replacement list are repl X is a parameter or A ## B. Every token of the result gets Token::macro set and the location of the macro call, so the usual output (-l, Token::macro) is the same as before. Why this is correct for recursion The macro is disabled from the moment its result is pushed back until the marker after it has been read. Anything that is read in between is part of the rescan of that expansion, so: Direct and mutual recursion (#define f f, A → B → A)while the macro is disabled, so it is painted and stays. An argument that is the name of the macro (f(f)(1) with #define f(x) x): the argument f is not disabled while arguments are expanded, but it is read again after substitution, while f is disabled. It is pa is not a call. The result is f ( 1 ), as with gcc. A name at the end of an expansion that gets its arguments from the source (#define g f / g(1)): f is read when g is already enabled again (the marker was read before (), so f(1) is expanded. This No special code is needed; it is just the next token in thestream. Deferred expansion (the DEFER / EXPAND / R_I idiom more scan of the tokens, so one more level is expanded. Apainted name stays painted, so it is never expanded too early. Unterminated calls A function-like macro name followed by ( whose ) is never found is not a call. startLog() records the markers that are read while the arguments are collected, and pushBackLogged() puts the tokens back with s (and disables the macros again). The name is output as itis. This is what makes R at the end of the file work. Differences from Prosser's algorithm (#742) Prosser's algorithm gives each token a hide set and computes intersections for function-like macros. This expander gets the same results with a counter per macro and one marker per expansion, which is how e.g. GCC'g's DisableMacro work. There is nothing to copy per token, and Token does not need to know about Macro. Cost Each argument is expanded once (not once per use, as before), and each token is pushed back once per macro that produces it. Macro usage is reported once per call. Two statements in it come from my reading of the code, not fro GCC and Clang mechanisms: the sentence in the "Differences from Prosser" section is from memory of how they work. Cost: the "reported once per call" claim was not measured. I can cut them. Fixed issues Each reproducer was run with master and with this branch, and compared with gcc -E. Issue Input master this PR (same as gcc) #742 #define f(x) x / f(f)(1) 1 f ( 1 ) #742 #define R() / R at end of file syntax error R #226 EVAL ( HELLO EMPTY() () ) HELLO ( ) "Hello World" #259 ADVANCED(int, , foo, ) (## with empty args) Invalid ## usage int foo ; #378 a ## b ## c, a ## ## c x# "y" xy #394 MACRO_COMPOSE(FOO, BAR(1)) FOOBAR ( 1 ) 1 #458 # and ## with __VA_OPT__ #, "x", "Hello" world "\"x\"", "Helloworld" #469 N4917 __VA_OPT__(G R X) example Wrong number of parameters int x = 42 ; #491 HASH define foo bar # and define… on separate lines # define foo bar #656 PAR-style PREFIX_ ## kind indirection PREFIX_SCALAR ( int , x ) int x Closes #226, #259, #378, #394, #458, #469, #491, #656. Implements the goal of #742 with a different algorithm, see above. gcc/clang testsuite macro_paste_hard.c and macro_rescan_varargs.c now match gcc/clang and are removed from the todo list in run-tests.py. All 176 cases were compared with master. No regressions. pr58844-1.c and pr58844-2.c are added to skip. For x######x, gcc gives xx and clang gives x##x. Master happened to give x##x, this PR gives xx. On macOS gcc is clang, so only the clang result is accepted there. Not fixed #653, #218, #300, #75: defined(a ## b) inside a macro used #572: a preprocessor directive inside macro arguments still gives an error. #373: __attribute()##8; still pastes )8. Not confirmed #662, #414: the fuzzing inputs do not crash on master, with ken` code they pointed to is removed, but I cannot claim a fix. #159, #96: not tested, they need the Boost headers or the full input from the issue. Behaviour changes A # that comes from a macro at the start of a line gets a space before it, so the output is not read as a directive (as gcc and clang do). See #491. A function-like macro name at the end of the file, or withouo it is no longer an error. The tokens of an expansion get the location of the macro call.

Open Graph Description: It is based on TokenStreamExpander and on the f(f)(1) result I checked against gcc earlier. I did not trace the longer DEFER/EXPAND example, so the description only states its result. The algorithm...

X Description: It is based on TokenStreamExpander and on the f(f)(1) result I checked against gcc earlier. I did not trace the longer DEFER/EXPAND example, so the description only states its result. The algorithm...

Opengraph URL: https://github.com/cppcheck-opensource/simplecpp/pull/758

X: @github

direct link

Domain: redirect.github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:534f857c-a6cd-113b-779f-2a66ad8e046f
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idA4E8:2EE40A:6A656:953F5:6AFC5920
html-safe-nonce3dcf659e56865544ec1aa91c8d718ec95395217fc3ff25b77a58a51c041bcb70
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBNEU4OjJFRTQwQTo2QTY1Njo5NTNGNTo2QUZDNTkyMCIsInZpc2l0b3JfaWQiOiI4MDYyMDkxODEwNDI1Mjk3MTg0IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0=
visitor-hmac7b583ffdb2520fb94ee79f4c357a83d24556f44103a13a293a910b2bb984246d
hovercard-subject-tagpull_request:5267935767
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/cppcheck-opensource/simplecpp/pull/758/files
twitter:imagehttps://avatars.githubusercontent.com/u/60195?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/60195?s=400&v=4
og:image:altIt is based on TokenStreamExpander and on the f(f)(1) result I checked against gcc earlier. I did not trace the longer DEFER/EXPAND example, so the description only states its result. The algorithm...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
Nonedca5d7b31d726645ba76b526bf3a137e74fee0f5f75412c83dafde3dc9b64ccb
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/cppcheck-opensource/simplecpp git https://github.com/cppcheck-opensource/simplecpp.git
octolytics-dimension-user_id278097703
octolytics-dimension-user_logincppcheck-opensource
octolytics-dimension-repository_id33434464
octolytics-dimension-repository_nwocppcheck-opensource/simplecpp
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id33434464
octolytics-dimension-repository_network_root_nwocppcheck-opensource/simplecpp
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
releasebedf3a505b023592c321f5456010d0d5a61a4e05
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://redirect.github.com/cppcheck-opensource/simplecpp/pull/758/files#start-of-content
https://redirect.github.com/
Sign inhttps://redirect.github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fcppcheck-opensource%2Fsimplecpp%2Fpull%2F758%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://redirect.github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fcppcheck-opensource%2Fsimplecpp%2Fpull%2F758%2Ffiles
Sign uphttps://redirect.github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=cppcheck-opensource%2Fsimplecpp
Reloadhttps://redirect.github.com/cppcheck-opensource/simplecpp/pull/758/files
Reloadhttps://redirect.github.com/cppcheck-opensource/simplecpp/pull/758/files
Reloadhttps://redirect.github.com/cppcheck-opensource/simplecpp/pull/758/files
cppcheck-opensourcehttps://redirect.github.com/cppcheck-opensource
simplecpphttps://redirect.github.com/cppcheck-opensource/simplecpp
Codehttps://redirect.github.com/cppcheck-opensource/simplecpp
Issues86 (86)https://redirect.github.com/cppcheck-opensource/simplecpp/issues
Pull requests36 (36)https://redirect.github.com/cppcheck-opensource/simplecpp/pulls
Actionshttps://redirect.github.com/cppcheck-opensource/simplecpp/actions
Wikihttps://redirect.github.com/cppcheck-opensource/simplecpp/wiki
Security and qualityhttps://redirect.github.com/cppcheck-opensource/simplecpp/security
Insightshttps://redirect.github.com/cppcheck-opensource/simplecpp/pulse
Sign up for GitHub https://redirect.github.com/signup?return_to=%2Fcppcheck-opensource%2Fsimplecpp%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://redirect.github.com/login?return_to=%2Fcppcheck-opensource%2Fsimplecpp%2Fissues%2Fnew%2Fchoose
danmarhttps://redirect.github.com/danmar
masterhttps://redirect.github.com/cppcheck-opensource/simplecpp/tree/master
pr-token-stream-expanderhttps://redirect.github.com/cppcheck-opensource/simplecpp/tree/pr-token-stream-expander
Conversation 3 https://redirect.github.com/cppcheck-opensource/simplecpp/pull/758
Commits 4 https://redirect.github.com/cppcheck-opensource/simplecpp/pull/758/commits
Checks 47 https://redirect.github.com/cppcheck-opensource/simplecpp/pull/758/checks
Files changed https://redirect.github.com/cppcheck-opensource/simplecpp/pull/758/files
Please reload this pagehttps://redirect.github.com/cppcheck-opensource/simplecpp/pull/758/files
token stream expander https://redirect.github.com/cppcheck-opensource/simplecpp/pull/758/files#top
Show all changes 4 commits https://redirect.github.com/cppcheck-opensource/simplecpp/pull/758/files
03edd42 Add tests for macro expansion dmcppcheck Nov 11, 2026 https://redirect.github.com/cppcheck-opensource/simplecpp/pull/758/commits/03edd42c7e77fe6e563e2af97c84e8cce5a8a882
7a7889b Add TokenStreamExpander, a macro expansion with a token stream dmcppcheck Nov 11, 2026 https://redirect.github.com/cppcheck-opensource/simplecpp/pull/758/commits/7a7889b8a2d9b4ae527b51e531412d8f04cd8770
3d82c3e Use TokenStreamExpander for macro expansion dmcppcheck Nov 11, 2026 https://redirect.github.com/cppcheck-opensource/simplecpp/pull/758/commits/3d82c3e7bbf4b744054611a3d82f8500bcbcf8f2
e9fd2ad Remove the old macro expansion dmcppcheck Nov 11, 2026 https://redirect.github.com/cppcheck-opensource/simplecpp/pull/758/commits/e9fd2ad747011d0277ee1e1625a851df64aec135
Clear filters https://redirect.github.com/cppcheck-opensource/simplecpp/pull/758/files
Please reload this pagehttps://redirect.github.com/cppcheck-opensource/simplecpp/pull/758/files
Please reload this pagehttps://redirect.github.com/cppcheck-opensource/simplecpp/pull/758/files
run-tests.py https://redirect.github.com/cppcheck-opensource/simplecpp/pull/758/files#diff-55af8247ec95043557b9810df657aade6cf363e20e83a29ed961b4f1f00ccb7a
simplecpp.cpp https://redirect.github.com/cppcheck-opensource/simplecpp/pull/758/files#diff-c847846a0c414ac05f52e4f9686648f22c76c9bb152579a253b600223c202832
simplecpp.h https://redirect.github.com/cppcheck-opensource/simplecpp/pull/758/files#diff-8feef6892144dbe750ffc83a1525c796968b56cd4d6500fe8bf22bac61cf9174
test.cpp https://redirect.github.com/cppcheck-opensource/simplecpp/pull/758/files#diff-042cc0321bd0d3c8cb8262c89142b3e6aba34345ae77b4294f5e07dd2687ffcf
run-tests.pyhttps://redirect.github.com/cppcheck-opensource/simplecpp/pull/758/files#diff-55af8247ec95043557b9810df657aade6cf363e20e83a29ed961b4f1f00ccb7a
View file https://redirect.github.com/cppcheck-opensource/simplecpp/blob/e9fd2ad747011d0277ee1e1625a851df64aec135/run-tests.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://redirect.github.com/cppcheck-opensource/simplecpp/pull/758/{{ revealButtonHref }}
https://redirect.github.com/cppcheck-opensource/simplecpp/pull/758/files#diff-55af8247ec95043557b9810df657aade6cf363e20e83a29ed961b4f1f00ccb7a
https://redirect.github.com/cppcheck-opensource/simplecpp/pull/758/files#diff-55af8247ec95043557b9810df657aade6cf363e20e83a29ed961b4f1f00ccb7a
https://redirect.github.com/cppcheck-opensource/simplecpp/pull/758/files#diff-55af8247ec95043557b9810df657aade6cf363e20e83a29ed961b4f1f00ccb7a
Please reload this pagehttps://redirect.github.com/cppcheck-opensource/simplecpp/pull/758/files
Please reload this pagehttps://redirect.github.com/cppcheck-opensource/simplecpp/pull/758/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.