René's URL Explorer Experiment


Title: fix(redis-worker): stop fair queue leaking concurrency slots by matt-aitken · Pull Request #4540 · triggerdotdev/trigger.dev · GitHub

Open Graph Title: fix(redis-worker): stop fair queue leaking concurrency slots by matt-aitken · Pull Request #4540 · triggerdotdev/trigger.dev

X Title: fix(redis-worker): stop fair queue leaking concurrency slots by matt-aitken · Pull Request #4540 · triggerdotdev/trigger.dev

Description: Summary Fair queue consumers could leak the per-tenant concurrency slots that gate admission. Slots were freed on some paths and skipped on others, and once enough leaked slots accumulated for a tenant, every queue that tenant owned stopped being served until someone cleared the set by hand. This PR frees slots on every path and, more importantly, makes the remaining failure modes self-healing. Design The fix applies one rule uniformly: releasing a concurrency slot is best-effort cleanup and must never block the message's primary state transition. Blocking completion re-delivers the message, which duplicates customer work; blocking a retry loses the attempt increment, so the message can circle forever; blocking a reclaim strands the message in flight. A leaked slot is the better failure in every one of those trades because it is the only one that is recoverable. A failed release is therefore logged and the transition proceeds. Leaked slots then heal through two mechanisms: reserve re-admits a message that is already a member of its own concurrency set, since re-admitting it does not increase concurrency. A message whose earlier release failed can no longer be blocked by its own leftover slot. A reconcile loop periodically removes any set member with no in-flight record (interval configurable via reconcileIntervalMs, default 60s). The check-and-remove is atomic, and it is sound because a message is always registered in flight before its slot is reserved, so a member with no in-flight record can only be a leak. This also covers leaks this PR cannot prevent directly, such as a release that resolves the wrong concurrency group from queue metadata. Ordering hardening from earlier revisions stays: slots are released before the in-flight record needed to describe them is discarded, the release Lua scripts write the message back to the queue before removing it from in-flight (Lua does not roll back on error), and dangling in-flight entries with no payload are dropped instead of being rescanned forever. Every guard test was verified to fail without its specific fix, including the duplicate-execution case: completing a message while its slot release fails used to re-deliver and re-execute it.

Open Graph Description: Summary Fair queue consumers could leak the per-tenant concurrency slots that gate admission. Slots were freed on some paths and skipped on others, and once enough leaked slots accumulated for a te...

X Description: Summary Fair queue consumers could leak the per-tenant concurrency slots that gate admission. Slots were freed on some paths and skipped on others, and once enough leaked slots accumulated for a te...

Opengraph URL: https://github.com/triggerdotdev/trigger.dev/pull/4540

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:795cd422-9508-ed70-dab7-c023c354abdd
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idBD6A:1A28DE:30B2C7F:4558617:6AFAF174
html-safe-nonce08e5f014b74faf025fefc3c3a08a79986d7cbdddd2074ea736dae93a12c0db75
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCRDZBOjFBMjhERTozMEIyQzdGOjQ1NTg2MTc6NkFGQUYxNzQiLCJ2aXNpdG9yX2lkIjoiNTYyMTI2NDYzMjAwMjU3MjY2MCIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmacc2b2a7bee0fc6de22a6884836103abf487759ea92a6830f3b6af421af5cc7f59
hovercard-subject-tagpull_request:4235961735
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/triggerdotdev/trigger.dev/pull/4540/files
twitter:imagehttps://avatars.githubusercontent.com/u/10635986?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/10635986?s=400&v=4
og:image:altSummary Fair queue consumers could leak the per-tenant concurrency slots that gate admission. Slots were freed on some paths and skipped on others, and once enough leaked slots accumulated for a te...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None0178036d8230524a18422872393ab7e708443ae4c2047f758560a2db870c1913
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/triggerdotdev/trigger.dev git https://github.com/triggerdotdev/trigger.dev.git
octolytics-dimension-user_id95297378
octolytics-dimension-user_logintriggerdotdev
octolytics-dimension-repository_id572570113
octolytics-dimension-repository_nwotriggerdotdev/trigger.dev
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id572570113
octolytics-dimension-repository_network_root_nwotriggerdotdev/trigger.dev
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
releasea85e96a83bed7871993e6b1ffda24105928eee17
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/triggerdotdev/trigger.dev/pull/4540/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ftriggerdotdev%2Ftrigger.dev%2Fpull%2F4540%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ftriggerdotdev%2Ftrigger.dev%2Fpull%2F4540%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=triggerdotdev%2Ftrigger.dev
Reloadhttps://github.com/triggerdotdev/trigger.dev/pull/4540/files
Reloadhttps://github.com/triggerdotdev/trigger.dev/pull/4540/files
Reloadhttps://github.com/triggerdotdev/trigger.dev/pull/4540/files
triggerdotdevhttps://github.com/triggerdotdev
trigger.devhttps://github.com/triggerdotdev/trigger.dev
Codehttps://github.com/triggerdotdev/trigger.dev
Issues203 (203)https://github.com/triggerdotdev/trigger.dev/issues
Pull requests67 (67)https://github.com/triggerdotdev/trigger.dev/pulls
Discussionshttps://github.com/triggerdotdev/trigger.dev/discussions
Actionshttps://github.com/triggerdotdev/trigger.dev/actions
Projectshttps://github.com/triggerdotdev/trigger.dev/projects
Security and quality39 (39)https://github.com/triggerdotdev/trigger.dev/security
Insightshttps://github.com/triggerdotdev/trigger.dev/pulse
Sign up for GitHub https://github.com/signup?return_to=%2Ftriggerdotdev%2Ftrigger.dev%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Ftriggerdotdev%2Ftrigger.dev%2Fissues%2Fnew%2Fchoose
matt-aitkenhttps://github.com/matt-aitken
mainhttps://github.com/triggerdotdev/trigger.dev/tree/main
fix/fair-queue-concurrency-slot-leakhttps://github.com/triggerdotdev/trigger.dev/tree/fix/fair-queue-concurrency-slot-leak
Conversation 72 https://github.com/triggerdotdev/trigger.dev/pull/4540
Commits 13 https://github.com/triggerdotdev/trigger.dev/pull/4540/commits
Checks 50 https://github.com/triggerdotdev/trigger.dev/pull/4540/checks
Files changed https://github.com/triggerdotdev/trigger.dev/pull/4540/files
Please reload this pagehttps://github.com/triggerdotdev/trigger.dev/pull/4540/files
fix(redis-worker): stop fair queue leaking concurrency slots https://github.com/triggerdotdev/trigger.dev/pull/4540/files#top
Show all changes 13 commits https://github.com/triggerdotdev/trigger.dev/pull/4540/files
4df6cda fix(redis-worker): stop fair queue leaking concurrency slots matt-aitken Aug 8, 2026 https://github.com/triggerdotdev/trigger.dev/pull/4540/commits/4df6cda09dbe42868bd2f38bfb21b6993319a917
35568a3 fix(redis-worker): keep metadata-derived concurrency groups on the re… matt-aitken Aug 8, 2026 https://github.com/triggerdotdev/trigger.dev/pull/4540/commits/35568a32e7ad4762ba9c44eab12d370476c15eae
b2ebf18 fix(redis-worker): close the remaining concurrency slot leaks in fair… matt-aitken Aug 11, 2026 https://github.com/triggerdotdev/trigger.dev/pull/4540/commits/b2ebf18abc314a633e515594c8ca5ec26dbf38e3
c6524ca fix(redis-worker): stop losing dead-lettered messages when the DLQ wr… matt-aitken Aug 11, 2026 https://github.com/triggerdotdev/trigger.dev/pull/4540/commits/c6524ca51592d7ef50ac11d95a46fefdf1c22110
3f8ed8b fix(redis-worker,run-engine): close the remaining ways a batch item i… matt-aitken Aug 11, 2026 https://github.com/triggerdotdev/trigger.dev/pull/4540/commits/3f8ed8b05d8eae6f0814e49310879a6dc95255d8
a2f12f2 fix(redis-worker,run-engine): give the item heartbeat real margin and… matt-aitken Aug 11, 2026 https://github.com/triggerdotdev/trigger.dev/pull/4540/commits/a2f12f2b5d983ead4dbc29a345fa58cf83dae229
ff348c4 fix(run-engine,redis-worker): drop a batch item result once its lease… matt-aitken Aug 11, 2026 https://github.com/triggerdotdev/trigger.dev/pull/4540/commits/ff348c4822eb6d4489d571a2f6ed474c8bf1bfd9
1b24915 docs(run-engine): state what the lease signal actually guarantees matt-aitken Aug 11, 2026 https://github.com/triggerdotdev/trigger.dev/pull/4540/commits/1b2491520ed4433ab10a0efbd3285f01b2a0d2da
01d0666 chore(run-engine): move batch item heartbeating to its own branch matt-aitken Aug 11, 2026 https://github.com/triggerdotdev/trigger.dev/pull/4540/commits/01d06660054ae64c6b3f6bf99ddfaa9dce5e98da
72947fe test(redis-worker): assert reclaim recovery on a stable signal matt-aitken Aug 11, 2026 https://github.com/triggerdotdev/trigger.dev/pull/4540/commits/72947fe22689272cb1411b3700ca0d3eab518b00
e7c3c86 fix(redis-worker): heal leaked concurrency slots instead of blocking … matt-aitken Aug 12, 2026 https://github.com/triggerdotdev/trigger.dev/pull/4540/commits/e7c3c86307dac25eb3ecfb1329312205a9faefee
21eb0d3 fix(redis-worker): bound the reconcile sweep cost and write the relea… matt-aitken Aug 12, 2026 https://github.com/triggerdotdev/trigger.dev/pull/4540/commits/21eb0d330f0ca1139aae9d860911c4e0ad836079
ba57004 fix(redis-worker): make the orphan sweep fail safe and page set members matt-aitken Aug 12, 2026 https://github.com/triggerdotdev/trigger.dev/pull/4540/commits/ba57004b972abdbf51eb800c86daa67b9763a142
Clear filters https://github.com/triggerdotdev/trigger.dev/pull/4540/files
Please reload this pagehttps://github.com/triggerdotdev/trigger.dev/pull/4540/files
Please reload this pagehttps://github.com/triggerdotdev/trigger.dev/pull/4540/files
fair-queue-concurrency-slot-leak.md https://github.com/triggerdotdev/trigger.dev/pull/4540/files#diff-1e4ea19c951228c20b6f41b4b49c6f297606eab9b6e51640723e8b3001c85944
concurrency.ts https://github.com/triggerdotdev/trigger.dev/pull/4540/files#diff-e1d7cfd40c4975d8cdf0eaff2f04b3a602f6cdfe01db5dbfbbb93498fbe3abee
index.ts https://github.com/triggerdotdev/trigger.dev/pull/4540/files#diff-71f1c15fc5900a4ef63735b95c96f6a4023481b45ef011a47310d1d21f678223
concurrency.test.ts https://github.com/triggerdotdev/trigger.dev/pull/4540/files#diff-420dcf0f7351ae5c56b415a04c60fefd1f429e89e3c32176bc7796f7571a427f
fairQueue.test.ts https://github.com/triggerdotdev/trigger.dev/pull/4540/files#diff-007b421c97a964906ea4708b3bad5ee95a3ef2c50b26d080f16f5ab40a350c03
visibility.test.ts https://github.com/triggerdotdev/trigger.dev/pull/4540/files#diff-350d6e943698a810aeb203430ad70c237854b978707143eb699a1745a8b31da3
types.ts https://github.com/triggerdotdev/trigger.dev/pull/4540/files#diff-59cdfd5be4629d05371b9f5ce798a5188f0b40af733b2359360915968f79d4f5
visibility.ts https://github.com/triggerdotdev/trigger.dev/pull/4540/files#diff-b11e57d16be2c4d56d20a6df685359b3fafb312bec7475ded648fc8723e3309d
.changeset/fair-queue-concurrency-slot-leak.mdhttps://github.com/triggerdotdev/trigger.dev/pull/4540/files#diff-1e4ea19c951228c20b6f41b4b49c6f297606eab9b6e51640723e8b3001c85944
View file https://github.com/triggerdotdev/trigger.dev/blob/ba57004b972abdbf51eb800c86daa67b9763a142/.changeset/fair-queue-concurrency-slot-leak.md
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/triggerdotdev/trigger.dev/pull/4540/{{ revealButtonHref }}
packages/redis-worker/src/fair-queue/concurrency.tshttps://github.com/triggerdotdev/trigger.dev/pull/4540/files#diff-e1d7cfd40c4975d8cdf0eaff2f04b3a602f6cdfe01db5dbfbbb93498fbe3abee
View file https://github.com/triggerdotdev/trigger.dev/blob/ba57004b972abdbf51eb800c86daa67b9763a142/packages/redis-worker/src/fair-queue/concurrency.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/triggerdotdev/trigger.dev/pull/4540/{{ revealButtonHref }}
https://github.com/triggerdotdev/trigger.dev/pull/4540/files#diff-e1d7cfd40c4975d8cdf0eaff2f04b3a602f6cdfe01db5dbfbbb93498fbe3abee
https://github.com/triggerdotdev/trigger.dev/pull/4540/files#diff-e1d7cfd40c4975d8cdf0eaff2f04b3a602f6cdfe01db5dbfbbb93498fbe3abee
https://github.com/triggerdotdev/trigger.dev/pull/4540/files#diff-e1d7cfd40c4975d8cdf0eaff2f04b3a602f6cdfe01db5dbfbbb93498fbe3abee
https://github.com/triggerdotdev/trigger.dev/pull/4540/files#diff-e1d7cfd40c4975d8cdf0eaff2f04b3a602f6cdfe01db5dbfbbb93498fbe3abee
Please reload this pagehttps://github.com/triggerdotdev/trigger.dev/pull/4540/files
https://github.com/triggerdotdev/trigger.dev/pull/4540/files#diff-e1d7cfd40c4975d8cdf0eaff2f04b3a602f6cdfe01db5dbfbbb93498fbe3abee
Please reload this pagehttps://github.com/triggerdotdev/trigger.dev/pull/4540/files
Please reload this pagehttps://github.com/triggerdotdev/trigger.dev/pull/4540/files
Please reload this pagehttps://github.com/triggerdotdev/trigger.dev/pull/4540/files
https://github.com/triggerdotdev/trigger.dev/pull/4540/files#diff-e1d7cfd40c4975d8cdf0eaff2f04b3a602f6cdfe01db5dbfbbb93498fbe3abee
https://github.com/triggerdotdev/trigger.dev/pull/4540/files#diff-e1d7cfd40c4975d8cdf0eaff2f04b3a602f6cdfe01db5dbfbbb93498fbe3abee
https://github.com/triggerdotdev/trigger.dev/pull/4540/files#diff-e1d7cfd40c4975d8cdf0eaff2f04b3a602f6cdfe01db5dbfbbb93498fbe3abee
https://github.com/triggerdotdev/trigger.dev/pull/4540/files#diff-e1d7cfd40c4975d8cdf0eaff2f04b3a602f6cdfe01db5dbfbbb93498fbe3abee
Please reload this pagehttps://github.com/triggerdotdev/trigger.dev/pull/4540/files
Please reload this pagehttps://github.com/triggerdotdev/trigger.dev/pull/4540/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.