Title: fix(run-engine,webapp): return unclaimed runs to the queue when pausing by matt-aitken · Pull Request #4398 · triggerdotdev/trigger.dev · GitHub
Open Graph Title: fix(run-engine,webapp): return unclaimed runs to the queue when pausing by matt-aitken · Pull Request #4398 · triggerdotdev/trigger.dev
X Title: fix(run-engine,webapp): return unclaimed runs to the queue when pausing by matt-aitken · Pull Request #4398 · triggerdotdev/trigger.dev
Description: The gap Queuing happens in two phases. Phase 1 checks concurrency and admits runs; phase 2 is a plain FIFO list in Redis per worker region that supervisors drain. Pausing a queue or an environment only sets its concurrency limit to 0. That correctly stops phase 1 admitting anything new — both the master-queue dequeue and the trigger-time fast path check the limit — but it does nothing about runs phase 1 had already admitted. Those are sitting in the worker queue, and the pop path is a bare BLPOP with no pause or concurrency check, so they went on to execute during the pause. The exposure is bigger than it first looks. A worker-queue entry already holds a concurrency slot, so the ceiling is the queue's whole concurrency limit — and for an idle environment with headroom the trigger-time fast path pushes straight to the worker queue, so a burst can fill it before you can hit pause. The fix RunQueue.returnUnclaimedMessagesToQueue({ env, queue? }) moves runs that have been admitted but not yet claimed by a worker back into the pending queue. Finding them doesn't need a scan of the region-wide list. A run in the worker queue is in currentConcurrency but not yet in currentDequeued (that's only added at pop), so the difference of those two sets is exactly the candidate set. The worker-queue element itself only carries org + run id, so filtering the list directly wasn't an option anyway. Each move is a single Lua script, and the LREM is the claim: if it removes nothing, a worker has already popped that entry and owns the run, so every other key is left untouched. BLPOP and the follow-up payload read are two separate round trips, so there's a window where a run is in neither the list nor currentDequeued and will show up as a candidate — keying off the LREM return value is what makes that race safe instead of a double dispatch. Three things the script has to get right beyond the claim: Score. Runs come back at their original message.timestamp. nackMessage re-scores to now + backoff, which would reorder, so this deliberately doesn't reuse it. TTL. Admission removes the run's member from the TTL sorted set, so returning it without re-adding leaves a run that can never expire — the next admission takes the already-expired branch, which hands cleanup to a TTL consumer that no longer has an entry to find. The per-run expireRun job isn't a backstop; it's only armed for development environments. Both worker-queue keys. The trigger-time fast path pushes to the raw workerQueue on the message while the master-queue consumer pushes to the resolver-mapped one, so with a region override active the two producers disagree. The claim LREMs both. The ordering constraint Returning a run costs it its place in the worker queue FIFO, and it can only re-enter at the back. So if the queue can still admit work, a newer run can overtake one on its way back, no matter how correct the re-scoring is. This is only safe once the concurrency limit is 0, and the doc comments say so. All three callers set the limit first. Failure handling The sweep runs after the pause is committed, never inside the rollback window. Earlier it sat inside the try that compensates a failed pause by clearing the DB flag — but the Redis limit was already 0 by then, so a sweep failure left the environment with no concurrency and no paused flag: dark, and unrecoverable without a manual pause/resume cycle. The pause is durable and the sweep is idempotent, so a failure is now logged rather than propagated. The sweep makes up to two passes. Admission isn't atomic with the push onto the worker queue — the dequeue script claims the slot and a separate round trip does the RPUSH — so a run caught in that gap fails its claim on the first pass and is claimable on the second. A pass that errored retries too: rejected script calls never increment the skipped count, so without that a transient Redis blip would end the sweep with those runs still admitted. skippedLastPass is deliberately named: it's the final pass's residual, and it can't distinguish a run a worker claimed first (escaped the pause, executing now) from a leaked concurrency slot with no worker-queue entry behind it. Callers manual queue pause manual environment pause billing-limit environment pause (behind the same injection seam as its concurrency update, so it stays off the module-load path) Tests returnUnclaimedMessagesToQueue.test.ts (17 cases): runs returned with concurrency released; original score and dequeue order preserved; a claimed run left alone; the LREM claim exercised directly by emptying the worker queue without marking the run dequeued; TTL round trip; concurrency-key queues including the length and running counters; fast-path enqueues; the dual worker-queue-key claim under a region override; retry-pass behaviour and the non-finite maxPasses guard; an acknowledged run never resurrected; error isolation and the retry it triggers; env-wide vs single-queue scope; cross-queue and cross-environment isolation; no-op and idempotency. dequeueMessageFromWorkerQueue.test.ts gains a case pinning the currentConcurrency / currentDequeued invariant the sweep relies on — it wasn't asserted anywhere. Webapp: pauseSweepWiring.test.ts (sweep fires after the limit is zeroed, not on resume), sweepUnclaimedRuns.test.ts (a failing sweep can't fail a pause), and three more cases on the billing-converge path. Full run-queue suite is 128 green; engine dequeuing/ttl/priority suites pass. Key behaviours were mutation-tested rather than assumed: removing the dual-key LREM, the retry loop, the CK lengthCounter INCR, the non-finite pass guard, the per-message error isolation, the retry-on-error condition, and the sweep call site each fail a specific test. Two earlier tests that passed vacuously were found that way and rewritten. Not in this PR A timeout-based return for runs stuck in a worker queue that nothing is draining. The same ordering constraint applies but without a pause to freeze phase 1, so it needs its own design. The trigger-time fast path ignores RUN_ENGINE_WORKER_QUEUE_OVERRIDES while the master-queue consumer honours it, so fast-pathed runs land on the un-rerouted worker queue during a region migration. Pre-existing routing bug; the sweep compensates for itself but the bug wants its own fix. PauseQueueService has no rollback if its concurrency-limit write fails, unlike PauseEnvironmentService. Pre-existing. The Lua guard that refuses to write back a run whose payload has been acknowledged is not covered. It is only reachable when an acknowledgement lands between the candidate read and the script, which isn't reproducible from the public API. nackMessage doesn't LREM the worker queue the way acknowledgeMessage does. Currently unreachable because nack is only called post-pop, but it would be a duplicate-dispatch bug if that changed. Note Touches run-queue/index.ts, which #4367 is also editing. The changes are additive and in a different part of the file (two new Lua commands + a new method), but worth a heads-up on merge order.
Open Graph Description: The gap Queuing happens in two phases. Phase 1 checks concurrency and admits runs; phase 2 is a plain FIFO list in Redis per worker region that supervisors drain. Pausing a queue or an environment ...
X Description: The gap Queuing happens in two phases. Phase 1 checks concurrency and admits runs; phase 2 is a plain FIFO list in Redis per worker region that supervisors drain. Pausing a queue or an environment ...
Opengraph URL: https://github.com/triggerdotdev/trigger.dev/pull/4398
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:388a19ce-8418-de88-4ceb-6f1debd3ff63 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | 83EA:3E915:3241992:475434A:6AFAF346 |
| html-safe-nonce | cbaf4294a68e63a8d45c35fca97771af4d5c8fdc3b503a623cb52e2ab7eac31a |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI4M0VBOjNFOTE1OjMyNDE5OTI6NDc1NDM0QTo2QUZBRjM0NiIsInZpc2l0b3JfaWQiOiI3MzIwNDE4NDcwOTk2OTMxMzk4IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | cc8a76ade0e67bbb42a44a3e0df76b724724263d6a6a7df07e6ac513b0b9ffb6 |
| hovercard-subject-tag | pull_request:4144374298 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/triggerdotdev/trigger.dev/pull/4398/files |
| twitter:image | https://avatars.githubusercontent.com/u/10635986?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/10635986?s=400&v=4 |
| og:image:alt | The gap Queuing happens in two phases. Phase 1 checks concurrency and admits runs; phase 2 is a plain FIFO list in Redis per worker region that supervisors drain. Pausing a queue or an environment ... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 0178036d8230524a18422872393ab7e708443ae4c2047f758560a2db870c1913 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/triggerdotdev/trigger.dev git https://github.com/triggerdotdev/trigger.dev.git |
| octolytics-dimension-user_id | 95297378 |
| octolytics-dimension-user_login | triggerdotdev |
| octolytics-dimension-repository_id | 572570113 |
| octolytics-dimension-repository_nwo | triggerdotdev/trigger.dev |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 572570113 |
| octolytics-dimension-repository_network_root_nwo | triggerdotdev/trigger.dev |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | a85e96a83bed7871993e6b1ffda24105928eee17 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width