Title: Replace the lexical foreign splitter with a real Lua 5.1 parser producing the Lua AST · Issue #173 · purescript-lua/purescript-lua · GitHub
Open Graph Title: Replace the lexical foreign splitter with a real Lua 5.1 parser producing the Lua AST · Issue #173 · purescript-lua/purescript-lua
X Title: Replace the lexical foreign splitter with a real Lua 5.1 parser producing the Lua AST · Issue #173 · purescript-lua/purescript-lua
Description: Problem Lua.Linker.Foreign does not parse foreign files — it lexically splits them into a header text and (Key, Text) export pairs, extracting each value as a raw text blob by counting balanced parentheses (hence the "every value is wrap...
Open Graph Description: Problem Lua.Linker.Foreign does not parse foreign files — it lexically splits them into a header text and (Key, Text) export pairs, extracting each value as a raw text blob by counting balanced par...
X Description: Problem Lua.Linker.Foreign does not parse foreign files — it lexically splits them into a header text and (Key, Text) export pairs, extracting each value as a raw text blob by counting balanced par...
Opengraph URL: https://github.com/purescript-lua/purescript-lua/issues/173
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Replace the lexical foreign splitter with a real Lua 5.1 parser producing the Lua AST","articleBody":"## Problem\n\n`Lua.Linker.Foreign` does not parse foreign files — it lexically splits them into a header text and `(Key, Text)` export pairs, extracting each value as a raw text blob by counting balanced parentheses (hence the \"every value is wrapped in parens\" contract in Note [Foreign module source format]). The blobs are embedded into the Lua AST as opaque verbatim nodes, so every optimization stops at a foreign boundary.\n\nThe measurable cost: when the inliner pastes a foreign lambda into an application site, the result is an applied function literal the optimizer cannot see into. `Golden.Fibonacci` compiles every addition to `(function(x) return function(y) return x + y end end)(a)(b)` — two closure allocations and two calls per arithmetic op. fib(30) runs 8.3x slower than handwritten Lua under PUC 5.1; under LuaJIT the per-call `FNEW` aborts every trace recording and gets the loop blacklisted, so the JIT contributes nothing.\n\n## Approach\n\nImplement our own Lua 5.1 parser (megaparsec is already a dependency) targeting the existing `Lua.Types` AST:\n\n1. Full Lua 5.1 expression/statement grammar (it is small). Each foreign export value parses into an AST expression, the header into statements.\n2. Comment preservation: comments attach to AST nodes via the existing annotation slots, so printed output keeps the load-bearing FFI comments and structural goldens stay close. Formatting/blank-line fidelity is a non-goal.\n3. The balanced-paren scanner and the parens-around-values contract become unnecessary; the overall file format contract (optional header, then `return { ... }`) stays.\n4. Foreign files get compile-time syntax validation — today a syntax error inside a value blob is caught only by luacheck or at runtime.\n\nDecision recorded: own parser, not hackage `language-lua` — the values must land in our `Lua.Types` with comments attached, and converting from a third-party comment-less AST buys nothing. This parser is a standing decision for the series: later issues build on foreign values being real AST.\n\n## Prerequisites / Relations\n\nInfrastructure with no hard prerequisite (megaparsec is already a dependency). What it unlocks:\n\n- Immediately, a Lua-level beta-reduction rule for applied function literals: `FunctionCall (Function ps body) args` substitutes trivial arguments (variable/literal) and `local`-binds non-trivial ones — the same work-duplication discipline #167 asks of the IR-level betaReduce. This dissolves the foreign IIFE pattern above and generalizes the special-cased foreign scope-IIFE folds (#158, #159).\n- Later, lifting the pure return-tree subset of foreign values into the IR as primitive operations derived from the actual fork sources (#178); this parser is that work's prerequisite. #175's principled \"is a table constructor\" check also wants this parser.\n\n## Verification / Measurement\n\nThe fib(30) macrobench and `FNEW` counters from #172 show the win directly: the foreign IIFE (`(function(x) return function(y) return x + y end end)(a)(b)`, two closure allocations + two calls per add) collapses, and the per-call `FNEW` that aborts LuaJIT trace recording disappears. Separately, a foreign file with a syntax error inside a value now fails at compile time rather than only under luacheck or at runtime.\n","author":{"url":"https://github.com/Unisay","@type":"Person","name":"Unisay"},"datePublished":"2026-07-06T09:26:12Z","url":"https://github.com/purescript-lua/purescript-lua/issues/173"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:df5ecb0f-0c62-0caa-7dfe-9904d0859b9e |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | B3F0:A99C2:AA38BD:DC9AE0:6B00E457 |
| html-safe-nonce | 2a425c8590de033afdb788b97ad7e5323ec97c98da343cf49cd4d5b2e9fb500f |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCM0YwOkE5OUMyOkFBMzhCRDpEQzlBRTA6NkIwMEU0NTciLCJ2aXNpdG9yX2lkIjoiODIyMTU4MjUzNDc0NDUzMjA1NSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 654285b0e0c33cf49d55b57e2757748a499599bcf8361109fd6c5618718fa039 |
| hovercard-subject-tag | issue:4817767511 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/purescript-lua/purescript-lua/173/issue_layout |
| twitter:image | https://opengraph.githubassets.com/645711e84d06aeb72f75e488f08b3381500001d14145adf0709918259bd1168e/purescript-lua/purescript-lua/issues/173 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/645711e84d06aeb72f75e488f08b3381500001d14145adf0709918259bd1168e/purescript-lua/purescript-lua/issues/173 |
| og:image:alt | Problem Lua.Linker.Foreign does not parse foreign files — it lexically splits them into a header text and (Key, Text) export pairs, extracting each value as a raw text blob by counting balanced par... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | Unisay |
| hostname | github.com |
| expected-hostname | github.com |
| None | 27088b6977484d0d3321d9a11c907c6d6637b315cfa359ef7952dbf6292f9b14 |
| turbo-cache-control | no-preview |
| go-import | github.com/purescript-lua/purescript-lua git https://github.com/purescript-lua/purescript-lua.git |
| octolytics-dimension-user_id | 131581790 |
| octolytics-dimension-user_login | purescript-lua |
| octolytics-dimension-repository_id | 587200487 |
| octolytics-dimension-repository_nwo | purescript-lua/purescript-lua |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 587200487 |
| octolytics-dimension-repository_network_root_nwo | purescript-lua/purescript-lua |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 624179fb8b88d26ba5bba08894eb60a1c5a1d93d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width