René's URL Explorer Experiment


Title: Replace the lexical foreign splitter with a real Lua 5.1 parser producing the Lua AST · Issue #173 · purescript-lua/purescript-lua · GitHub

Open Graph Title: Replace the lexical foreign splitter with a real Lua 5.1 parser producing the Lua AST · Issue #173 · purescript-lua/purescript-lua

X Title: Replace the lexical foreign splitter with a real Lua 5.1 parser producing the Lua AST · Issue #173 · purescript-lua/purescript-lua

Description: Problem Lua.Linker.Foreign does not parse foreign files — it lexically splits them into a header text and (Key, Text) export pairs, extracting each value as a raw text blob by counting balanced parentheses (hence the "every value is wrap...

Open Graph Description: Problem Lua.Linker.Foreign does not parse foreign files — it lexically splits them into a header text and (Key, Text) export pairs, extracting each value as a raw text blob by counting balanced par...

X Description: Problem Lua.Linker.Foreign does not parse foreign files — it lexically splits them into a header text and (Key, Text) export pairs, extracting each value as a raw text blob by counting balanced par...

Opengraph URL: https://github.com/purescript-lua/purescript-lua/issues/173

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Replace the lexical foreign splitter with a real Lua 5.1 parser producing the Lua AST","articleBody":"## Problem\n\n`Lua.Linker.Foreign` does not parse foreign files — it lexically splits them into a header text and `(Key, Text)` export pairs, extracting each value as a raw text blob by counting balanced parentheses (hence the \"every value is wrapped in parens\" contract in Note [Foreign module source format]). The blobs are embedded into the Lua AST as opaque verbatim nodes, so every optimization stops at a foreign boundary.\n\nThe measurable cost: when the inliner pastes a foreign lambda into an application site, the result is an applied function literal the optimizer cannot see into. `Golden.Fibonacci` compiles every addition to `(function(x) return function(y) return x + y end end)(a)(b)` — two closure allocations and two calls per arithmetic op. fib(30) runs 8.3x slower than handwritten Lua under PUC 5.1; under LuaJIT the per-call `FNEW` aborts every trace recording and gets the loop blacklisted, so the JIT contributes nothing.\n\n## Approach\n\nImplement our own Lua 5.1 parser (megaparsec is already a dependency) targeting the existing `Lua.Types` AST:\n\n1. Full Lua 5.1 expression/statement grammar (it is small). Each foreign export value parses into an AST expression, the header into statements.\n2. Comment preservation: comments attach to AST nodes via the existing annotation slots, so printed output keeps the load-bearing FFI comments and structural goldens stay close. Formatting/blank-line fidelity is a non-goal.\n3. The balanced-paren scanner and the parens-around-values contract become unnecessary; the overall file format contract (optional header, then `return { ... }`) stays.\n4. Foreign files get compile-time syntax validation — today a syntax error inside a value blob is caught only by luacheck or at runtime.\n\nDecision recorded: own parser, not hackage `language-lua` — the values must land in our `Lua.Types` with comments attached, and converting from a third-party comment-less AST buys nothing. This parser is a standing decision for the series: later issues build on foreign values being real AST.\n\n## Prerequisites / Relations\n\nInfrastructure with no hard prerequisite (megaparsec is already a dependency). What it unlocks:\n\n- Immediately, a Lua-level beta-reduction rule for applied function literals: `FunctionCall (Function ps body) args` substitutes trivial arguments (variable/literal) and `local`-binds non-trivial ones — the same work-duplication discipline #167 asks of the IR-level betaReduce. This dissolves the foreign IIFE pattern above and generalizes the special-cased foreign scope-IIFE folds (#158, #159).\n- Later, lifting the pure return-tree subset of foreign values into the IR as primitive operations derived from the actual fork sources (#178); this parser is that work's prerequisite. #175's principled \"is a table constructor\" check also wants this parser.\n\n## Verification / Measurement\n\nThe fib(30) macrobench and `FNEW` counters from #172 show the win directly: the foreign IIFE (`(function(x) return function(y) return x + y end end)(a)(b)`, two closure allocations + two calls per add) collapses, and the per-call `FNEW` that aborts LuaJIT trace recording disappears. Separately, a foreign file with a syntax error inside a value now fails at compile time rather than only under luacheck or at runtime.\n","author":{"url":"https://github.com/Unisay","@type":"Person","name":"Unisay"},"datePublished":"2026-07-06T09:26:12Z","url":"https://github.com/purescript-lua/purescript-lua/issues/173"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:df5ecb0f-0c62-0caa-7dfe-9904d0859b9e
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-idB3F0:A99C2:AA38BD:DC9AE0:6B00E457
html-safe-nonce2a425c8590de033afdb788b97ad7e5323ec97c98da343cf49cd4d5b2e9fb500f
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCM0YwOkE5OUMyOkFBMzhCRDpEQzlBRTA6NkIwMEU0NTciLCJ2aXNpdG9yX2lkIjoiODIyMTU4MjUzNDc0NDUzMjA1NSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmac654285b0e0c33cf49d55b57e2757748a499599bcf8361109fd6c5618718fa039
hovercard-subject-tagissue:4817767511
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/purescript-lua/purescript-lua/173/issue_layout
twitter:imagehttps://opengraph.githubassets.com/645711e84d06aeb72f75e488f08b3381500001d14145adf0709918259bd1168e/purescript-lua/purescript-lua/issues/173
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/645711e84d06aeb72f75e488f08b3381500001d14145adf0709918259bd1168e/purescript-lua/purescript-lua/issues/173
og:image:altProblem Lua.Linker.Foreign does not parse foreign files — it lexically splits them into a header text and (Key, Text) export pairs, extracting each value as a raw text blob by counting balanced par...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernameUnisay
hostnamegithub.com
expected-hostnamegithub.com
None27088b6977484d0d3321d9a11c907c6d6637b315cfa359ef7952dbf6292f9b14
turbo-cache-controlno-preview
go-importgithub.com/purescript-lua/purescript-lua git https://github.com/purescript-lua/purescript-lua.git
octolytics-dimension-user_id131581790
octolytics-dimension-user_loginpurescript-lua
octolytics-dimension-repository_id587200487
octolytics-dimension-repository_nwopurescript-lua/purescript-lua
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id587200487
octolytics-dimension-repository_network_root_nwopurescript-lua/purescript-lua
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release624179fb8b88d26ba5bba08894eb60a1c5a1d93d
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/purescript-lua/purescript-lua/issues/173#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpurescript-lua%2Fpurescript-lua%2Fissues%2F173
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpurescript-lua%2Fpurescript-lua%2Fissues%2F173
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=purescript-lua%2Fpurescript-lua
Reloadhttps://github.com/purescript-lua/purescript-lua/issues/173
Reloadhttps://github.com/purescript-lua/purescript-lua/issues/173
Reloadhttps://github.com/purescript-lua/purescript-lua/issues/173
purescript-luahttps://github.com/purescript-lua
purescript-luahttps://github.com/purescript-lua/purescript-lua
Codehttps://github.com/purescript-lua/purescript-lua
Issues48 (48)https://github.com/purescript-lua/purescript-lua/issues
Pull requestshttps://github.com/purescript-lua/purescript-lua/pulls
Discussionshttps://github.com/purescript-lua/purescript-lua/discussions
Actionshttps://github.com/purescript-lua/purescript-lua/actions
Projectshttps://github.com/purescript-lua/purescript-lua/projects
Security and qualityhttps://github.com/purescript-lua/purescript-lua/security
Insightshttps://github.com/purescript-lua/purescript-lua/insights
https://github.com/purescript-lua/purescript-lua
New issuehttps://github.com/login?return_to=https://github.com/purescript-lua/purescript-lua/issues/173
New issuehttps://github.com/login?return_to=https://github.com/purescript-lua/purescript-lua/issues/173
#197https://github.com/purescript-lua/purescript-lua/pull/197
Replace the lexical foreign splitter with a real Lua 5.1 parser producing the Lua ASThttps://github.com/purescript-lua/purescript-lua/issues/173#top
#197https://github.com/purescript-lua/purescript-lua/pull/197
https://github.com/Unisay
OptimisationA Compiler Optimisationhttps://github.com/purescript-lua/purescript-lua/issues?q=state%3Aopen%20label%3A%22Optimisation%22
area: codegenLua code generation / printinghttps://github.com/purescript-lua/purescript-lua/issues?q=state%3Aopen%20label%3A%22area%3A%20codegen%22
https://github.com/Unisay
Unisayhttps://github.com/Unisay
on Jul 6, 2026https://github.com/purescript-lua/purescript-lua/issues/173#issue-4817767511
betaReduce substitutes a non-trivial argument into every occurrence of the parameter, multiplying work #167https://github.com/purescript-lua/purescript-lua/issues/167
IR.Abs lowering wraps a chunk body in a scope IIFE that a separate rule then strips #158https://github.com/purescript-lua/purescript-lua/issues/158
Fold a field projection through a foreign-header scope IIFE #159https://github.com/purescript-lua/purescript-lua/issues/159
Lift the pure subset of foreign values into the IR: primop nodes and an allowlist-driven lifter #178https://github.com/purescript-lua/purescript-lua/issues/178
Foreign table-constructor exports are duplicated per use site (fresh identity and allocation per copy) #175https://github.com/purescript-lua/purescript-lua/issues/175
#172https://github.com/purescript-lua/purescript-lua/issues/172
Unisayhttps://github.com/Unisay
area: codegenLua code generation / printinghttps://github.com/purescript-lua/purescript-lua/issues?q=state%3Aopen%20label%3A%22area%3A%20codegen%22
OptimisationA Compiler Optimisationhttps://github.com/purescript-lua/purescript-lua/issues?q=state%3Aopen%20label%3A%22Optimisation%22
on Jul 6, 2026https://github.com/purescript-lua/purescript-lua/issues/173#event-27599349588
Unisayhttps://github.com/Unisay
on Jul 6, 2026https://github.com/purescript-lua/purescript-lua/issues/173#event-27606251591
Closing audit: prove the cumulative win of the optimisation series #187https://github.com/purescript-lua/purescript-lua/issues/187
Unisayhttps://github.com/Unisay
on Jul 7, 2026https://github.com/purescript-lua/purescript-lua/issues/173#event-27653976823
Unisayhttps://github.com/Unisay
on Jul 7, 2026https://github.com/purescript-lua/purescript-lua/issues/173#event-27664511640
fix: reject chunk-scope '...' in foreign files (#173)https://github.com/purescript-lua/purescript-lua/commit/3a677f02e7507726c7033b4f4b9fcd973fecf25d
#173https://github.com/purescript-lua/purescript-lua/issues/173
3a677f0https://github.com/purescript-lua/purescript-lua/commit/3a677f02e7507726c7033b4f4b9fcd973fecf25d
chore: refresh bench counter goldens after #173 line shiftshttps://github.com/purescript-lua/purescript-lua/commit/4e450028715ac116fcae751c3ccd07f80ea01f5d
#173https://github.com/purescript-lua/purescript-lua/issues/173
4e45002https://github.com/purescript-lua/purescript-lua/commit/4e450028715ac116fcae751c3ccd07f80ea01f5d
Unisayhttps://github.com/Unisay
completedhttps://github.com/purescript-lua/purescript-lua/issues?q=is%3Aissue%20state%3Aclosed%20archived%3Afalse%20reason%3Acompleted
#197https://github.com/purescript-lua/purescript-lua/pull/197
on Jul 7, 2026https://github.com/purescript-lua/purescript-lua/issues/173#event-27665073343
Unisayhttps://github.com/Unisay
on Jul 12, 2026https://github.com/purescript-lua/purescript-lua/issues/173#event-27875400234
docs: sync FFI foreign-module shape in copilot-instructions.md with Q…https://github.com/purescript-lua/purescript-lua/commit/4d43b25555e8f68cc481919353d17ec66931b9e8
4d43b25https://github.com/purescript-lua/purescript-lua/commit/4d43b25555e8f68cc481919353d17ec66931b9e8
Sign up for freehttps://github.com/signup?return_to=https://github.com/purescript-lua/purescript-lua/issues/173
Sign in to commenthttps://github.com/login?return_to=https://github.com/purescript-lua/purescript-lua/issues/173
Unisayhttps://github.com/Unisay
OptimisationA Compiler Optimisationhttps://github.com/purescript-lua/purescript-lua/issues?q=state%3Aopen%20label%3A%22Optimisation%22
area: codegenLua code generation / printinghttps://github.com/purescript-lua/purescript-lua/issues?q=state%3Aopen%20label%3A%22area%3A%20codegen%22
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.