René's URL Explorer Experiment


Title: Bench trace oracle: a marginal spot can redden CI on any line-shifting codegen change · Issue #346 · purescript-lua/purescript-lua · GitHub

Open Graph Title: Bench trace oracle: a marginal spot can redden CI on any line-shifting codegen change · Issue #346 · purescript-lua/purescript-lua

X Title: Bench trace oracle: a marginal spot can redden CI on any line-shifting codegen change · Issue #346 · purescript-lua/purescript-lua

Description: Problem ./bench/ci pins deterministic LuaJIT counters against committed oracles. Two of the three are pure functions of the linked artifact and are byte-stable by construction. The third, the per-spec trace report, is not: it records whi...

Open Graph Description: Problem ./bench/ci pins deterministic LuaJIT counters against committed oracles. Two of the three are pure functions of the linked artifact and are byte-stable by construction. The third, the per-s...

X Description: Problem ./bench/ci pins deterministic LuaJIT counters against committed oracles. Two of the three are pure functions of the linked artifact and are byte-stable by construction. The third, the per-s...

Opengraph URL: https://github.com/purescript-lua/purescript-lua/issues/346

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Bench trace oracle: a marginal spot can redden CI on any line-shifting codegen change","articleBody":"## Problem\n\n`./bench/ci` pins deterministic LuaJIT counters against committed oracles. Two of the three are pure functions of the linked artifact and are byte-stable by construction. The third, the per-spec trace report, is not: it records which source locations reached a compiled trace (`J*` opcodes) or were blacklisted (`I*`), and trace formation is nondeterministic per process. `trace_report.lua` absorbs that with a majority vote over nine fresh LuaJIT processes, and the vote works — for spots whose per-process formation probability `p` sits near 0 or 1.\n\nIt does not fix a spot whose `p` sits in between, and such a spot can appear from nothing more than a line shift. LuaJIT's hot counters live in a small hashed table keyed by bytecode address, so removing a single `local` from an artifact reshuffles which counters collide. The tool's own header already records how sensitive this is: the same artifact measures `p ~ 0.05` merely when loaded through a shorter relative path.\n\n`Bench.RefLoop.lua:32 JFUNCF` — the entry of the ST thunk wrapping the hot loop — became such a spot when the single-use foreign-import fold (#251) removed one hoisted table from the artifact. Measured over raw single-process trials it forms with `p` roughly 0.6 to 0.85, drifting between measurement batches rather than sitting at a fixed value.\n\n## Why the existing knobs do not reach it\n\nRaising the workload `reps` through 4, 8, 16 and 32 leaves `p` unchanged, which rules out the documented \"not enough slack above the hot-call threshold\" cause and confirms counter aliasing.\n\nRaising the trial count does not help either, and the reason is worth stating: more trials sharpen the estimate of `p`, which only helps when `p` is a fixed value away from one half. Here `p` itself drifts, so more trials converge on a moving target, at linear cost in fresh processes.\n\n## What it costs\n\nWith nine trials, the probability that the vote lands on the wrong side is `P(Binom(9, p) ≤ 4)`:\n\n```\np = 0.8  ->  ~2% of runs flip\np = 0.7  ->  ~10%\np = 0.6  ->  ~27%\n```\n\nSo this one golden line carries somewhere between a 2% and a 27% chance of reddening CI per run. Observed so far: of two CI runs that reached the bench step, one went red on it and one green. Ten consecutive local `./bench/ci` runs matched, which is why \"present\" is the side currently pinned — it is what one machine produces every time and what CI reaches too, whereas pinning \"absent\" would fail locally on every run.\n\nThe general risk is the point: any codegen change that shifts lines in a bench artifact can push some borderline entry across, so the trace goldens carry a standing intermittent-red risk that is not attributable to the change under review.\n\n## Approach\n\n**Retry the bench step once on mismatch.** Cheapest, and narrower than it looks: the only failures a retry absorbs are the coin-flip ones, because a genuine codegen change moves the counters deterministically and fails both attempts. Costs a doubled ~10-minute step on failure, and in principle masks a regression that itself lands near one half.\n\n**Quarantine named spots.** Let a golden mark one line unpinnable and have the diff skip it. Keeps every other signal and makes the instability explicit and reviewed rather than silently absorbed. Costs a maintained exception list, and risks a real regression later hiding behind a quarantined line.\n\n**Classify marginal spots as a third state.** Print a distinct token when the vote share falls in a middle band, so both machines emit the same thing for a borderline spot. This looks like a fix but is not: every band has edges, and a spot can sit on one. A 1/3–2/3 band leaves this spot at `p ~ 0.6-0.85` straddling the upper edge; widening it to 0.1–0.9 catches this spot but puts the `Bench.BindChain` run wrapper the tool header records at `p ~ 0.9` on the new edge.\n\n**Drop the signal class.** Exclude artifact function-entry `J*FUNCF` states, keeping loop states and `I*FUNCF` blacklists. This does remove the failure mode — the marginal spots are entry traces racing their own inner loop's counter, which is exactly why the tool already excludes the spec harness's own wrappers. It also gives up real information: a compiled function entry is genuine codegen signal elsewhere in the corpus.\n\nSuggested: the retry as the immediate mitigation, quarantine as the principled follow-up if it recurs on other specs. Avoid the band approach specifically because it resembles a fix without being one.\n\n## Prerequisites / Relations\n\nIndependent. Surfaced by #251 (PR #344), whose line shift pushed `Bench.RefLoop.lua:32` onto the boundary, but the fragility is the oracle's own — any codegen change that shifts artifact lines can do the same to a different spot. Confined to `bench/`; no compiler code involved.\n\n## Acceptance criteria\n\n- A codegen change that only shifts lines in a bench artifact does not redden CI through a marginal trace spot.\n- A real change to which bytecodes trace or blacklist still fails the oracle.\n- Whatever is chosen is recorded in `bench/README.md` next to the existing majority-vote explanation, since that text currently implies the vote is sufficient.\n","author":{"url":"https://github.com/Unisay","@type":"Person","name":"Unisay"},"datePublished":"2026-07-29T08:25:34Z","url":"https://github.com/purescript-lua/purescript-lua/issues/346"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:b0c03fb1-0b05-6277-b134-7e395ab2db80
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-idD42E:24FC21:87FBFE:B14247:6B00DB66
html-safe-nonce0e14d021e049be1544b1796d57533397298ffc6b83aaab1cce41dfe1d25e25af
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJENDJFOjI0RkMyMTo4N0ZCRkU6QjE0MjQ3OjZCMDBEQjY2IiwidmlzaXRvcl9pZCI6IjQ3MzMwNjQ3Mjk0NzAyMjkzNTEiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmac1da72f89b8ea488feda4737a8b08daa40f07fc2cd2329662e2badb7e28d4a300
hovercard-subject-tagissue:5007100886
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/purescript-lua/purescript-lua/346/issue_layout
twitter:imagehttps://opengraph.githubassets.com/91400372d821d2e73b4f01641fbbb44c45b765bf3aa4a12ab4df8fd6e2f1bdde/purescript-lua/purescript-lua/issues/346
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/91400372d821d2e73b4f01641fbbb44c45b765bf3aa4a12ab4df8fd6e2f1bdde/purescript-lua/purescript-lua/issues/346
og:image:altProblem ./bench/ci pins deterministic LuaJIT counters against committed oracles. Two of the three are pure functions of the linked artifact and are byte-stable by construction. The third, the per-s...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernameUnisay
hostnamegithub.com
expected-hostnamegithub.com
None043c3f30fbcbe2c6c8e729e10cbbbc90c406546f9fbbb67c648d38f6b7d16daf
turbo-cache-controlno-preview
go-importgithub.com/purescript-lua/purescript-lua git https://github.com/purescript-lua/purescript-lua.git
octolytics-dimension-user_id131581790
octolytics-dimension-user_loginpurescript-lua
octolytics-dimension-repository_id587200487
octolytics-dimension-repository_nwopurescript-lua/purescript-lua
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id587200487
octolytics-dimension-repository_network_root_nwopurescript-lua/purescript-lua
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release5425dc24e8bd8d6fca9641a12afc984825a43875
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/purescript-lua/purescript-lua/issues/346#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpurescript-lua%2Fpurescript-lua%2Fissues%2F346
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpurescript-lua%2Fpurescript-lua%2Fissues%2F346
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=purescript-lua%2Fpurescript-lua
Reloadhttps://github.com/purescript-lua/purescript-lua/issues/346
Reloadhttps://github.com/purescript-lua/purescript-lua/issues/346
Reloadhttps://github.com/purescript-lua/purescript-lua/issues/346
purescript-luahttps://github.com/purescript-lua
purescript-luahttps://github.com/purescript-lua/purescript-lua
Codehttps://github.com/purescript-lua/purescript-lua
Issues48 (48)https://github.com/purescript-lua/purescript-lua/issues
Pull requestshttps://github.com/purescript-lua/purescript-lua/pulls
Discussionshttps://github.com/purescript-lua/purescript-lua/discussions
Actionshttps://github.com/purescript-lua/purescript-lua/actions
Projectshttps://github.com/purescript-lua/purescript-lua/projects
Security and qualityhttps://github.com/purescript-lua/purescript-lua/security
Insightshttps://github.com/purescript-lua/purescript-lua/insights
https://github.com/purescript-lua/purescript-lua
New issuehttps://github.com/login?return_to=https://github.com/purescript-lua/purescript-lua/issues/346
New issuehttps://github.com/login?return_to=https://github.com/purescript-lua/purescript-lua/issues/346
#355https://github.com/purescript-lua/purescript-lua/pull/355
Bench trace oracle: a marginal spot can redden CI on any line-shifting codegen changehttps://github.com/purescript-lua/purescript-lua/issues/346#top
#355https://github.com/purescript-lua/purescript-lua/pull/355
area: toolchainnix / spago / purs / buildhttps://github.com/purescript-lua/purescript-lua/issues?q=state%3Aopen%20label%3A%22area%3A%20toolchain%22
choreMaintenance / infrastructurehttps://github.com/purescript-lua/purescript-lua/issues?q=state%3Aopen%20label%3A%22chore%22
https://github.com/Unisay
Unisayhttps://github.com/Unisay
on Jul 29, 2026https://github.com/purescript-lua/purescript-lua/issues/346#issue-5007100886
#251https://github.com/purescript-lua/purescript-lua/issues/251
#251https://github.com/purescript-lua/purescript-lua/issues/251
#344https://github.com/purescript-lua/purescript-lua/pull/344
Unisayhttps://github.com/Unisay
area: toolchainnix / spago / purs / buildhttps://github.com/purescript-lua/purescript-lua/issues?q=state%3Aopen%20label%3A%22area%3A%20toolchain%22
choreMaintenance / infrastructurehttps://github.com/purescript-lua/purescript-lua/issues?q=state%3Aopen%20label%3A%22chore%22
on Jul 29, 2026https://github.com/purescript-lua/purescript-lua/issues/346#event-28646922145
Unisayhttps://github.com/Unisay
on Jul 29, 2026https://github.com/purescript-lua/purescript-lua/issues/346#event-28665945189
test(bench): pin that a failed re-measurement aborts rather than spen…https://github.com/purescript-lua/purescript-lua/commit/1865e7ea7a785d285affabcd9c9af9a1f0af7504
1865e7ehttps://github.com/purescript-lua/purescript-lua/commit/1865e7ea7a785d285affabcd9c9af9a1f0af7504
fix(bench): re-measure a trace report that differs only in the measur…https://github.com/purescript-lua/purescript-lua/commit/22a7431b428dd630183c9a6e8e91b46728ded3d7
22a7431https://github.com/purescript-lua/purescript-lua/commit/22a7431b428dd630183c9a6e8e91b46728ded3d7
Unisayhttps://github.com/Unisay
completedhttps://github.com/purescript-lua/purescript-lua/issues?q=is%3Aissue%20state%3Aclosed%20archived%3Afalse%20reason%3Acompleted
#355https://github.com/purescript-lua/purescript-lua/pull/355
on Jul 29, 2026https://github.com/purescript-lua/purescript-lua/issues/346#event-28675518894
Sign up for freehttps://github.com/signup?return_to=https://github.com/purescript-lua/purescript-lua/issues/346
Sign in to commenthttps://github.com/login?return_to=https://github.com/purescript-lua/purescript-lua/issues/346
area: toolchainnix / spago / purs / buildhttps://github.com/purescript-lua/purescript-lua/issues?q=state%3Aopen%20label%3A%22area%3A%20toolchain%22
choreMaintenance / infrastructurehttps://github.com/purescript-lua/purescript-lua/issues?q=state%3Aopen%20label%3A%22chore%22
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.