Title: feat(webapp,run-store,database): env-configurable transaction resilience (maxWait + tx-start retry) by ericallam · Pull Request #4623 · triggerdotdev/trigger.dev · GitHub
Open Graph Title: feat(webapp,run-store,database): env-configurable transaction resilience (maxWait + tx-start retry) by ericallam · Pull Request #4623 · triggerdotdev/trigger.dev
X Title: feat(webapp,run-store,database): env-configurable transaction resilience (maxWait + tx-start retry) by ericallam · Pull Request #4623 · triggerdotdev/trigger.dev
Description: What Makes two transaction-resilience behaviors real and env-var configurable, defaults set to the good values, so we can tune during and after the Aug 15 database patch window without a redeploy: maxWait 2s → 10s (TRI-12982): how long Prisma waits to borrow a connection before it can BEGIN. A restart freeze holds the pool full, and the only thing that errored was transaction starts giving up at 2s. Retry transaction-start P2028-at-acquisition (TRI-12984): when Prisma can't borrow a connection within maxWait it raises P2028 (Unable to start a transaction in the given time) and no SQL ran, so retrying is safe. Scoped narrowly: only that error (never P2024 pool-exhaustion), 2 attempts, jittered backoff, and a token-bucket budget so a mass freeze can't amplify into a retry storm. Env vars (DATABASE_* convention) Generic defaults: var default DATABASE_TRANSACTION_MAX_WAIT_MS 10000 DATABASE_TRANSACTION_START_RETRY_ENABLED true (kill switch) DATABASE_TRANSACTION_START_RETRY_MAX_ATTEMPTS 2 DATABASE_TRANSACTION_START_RETRY_BACKOFF_MIN_MS 50 DATABASE_TRANSACTION_START_RETRY_BACKOFF_MAX_MS 250 DATABASE_TRANSACTION_START_RETRY_BUDGET_PER_SEC 50 DATABASE_TRANSACTION_START_RETRY_BUDGET_BURST 100 Per-writer-pool overrides, each falling back to the generic when unset (same pattern as the per-client pool/connect-timeout work): RUN_OPS_DATABASE_TRANSACTION_* and RUN_OPS_LEGACY_DATABASE_TRANSACTION_* (all 7 knobs each). Transactions only open on writer pools, so those are the only pools with their own knobs. Each pool gets its own token bucket, so a storm on one pool can't drain another's retry budget. Design The retry primitives live in internal-packages/database and never read process.env (IoC): a P2028-at-acquisition classifier, a TokenBucketRetryBudget, and withTransactionStartRetry, folded into the $transaction helper via a new startRetry option. Config is resolved at the app boundary and threaded in. The $transaction helper is the chokepoint (wraps the whole transaction), not the per-statement $allOperations extension. The run engine's writes go through PostgresRunStore's own .$transaction(...), not the webapp helper, so both the helper and the two PostgresRunStore sites apply maxWait + retry (sharing the per-pool config). Builds on the options?: { timeout, maxWait } seam added in #4514. Webapp $transaction call sites get the default maxWait + retry injected at one merge point, so no call site needed editing. Evidence Unit red/green in internal-packages/database: reverting the helper wiring turned the acquisition-retry test red (Unable to start a transaction in the given time), re-applying it green. Full package suite 25/25. Covers: classifier (P2028-acq yes, P2024 no, in-tx P2028 no), retry (retry-then-succeed, no-retry P2024, stop at maxAttempts, disabled, budget-exhausted, jitter bounds), token bucket, and $transaction wiring. Typecheck clean: webapp, run-store, run-engine. Full-stack run: bounded queue-ay pass (15 projects, real dev runs through the run-engine PostgresRunStore transaction path). 13 pass; the 2 failures are one documented known-failure and one stale-worker-state flake that passes 2/2 with this change active on a fresh app. Boots cleanly with per-pool overrides set. Configuration & rollout Ship inert first (zero behavior change), then flip to the good values live via env — no redeploy needed for either. Inert — behaves exactly as today DATABASE_TRANSACTION_MAX_WAIT_MS=2000 # Prisma's built-in default (change defaults to 10000) DATABASE_TRANSACTION_START_RETRY_ENABLED=false # disable the new retry entirely maxWait=2000 is what every path used before (Prisma's default; the run-store sites and the helper passed no maxWait). retry=false short-circuits withTransactionStartRetry to a single run and makes the serialization-retry exclusion a no-op. Verified on the pooler-freeze rig: identical fail-fast P2028 at ~2003ms with zero retries — byte-for-byte current behavior, across all pools. Production ("good") — the baked defaults Rely on defaults (nothing to set) or set explicitly: DATABASE_TRANSACTION_MAX_WAIT_MS=10000 DATABASE_TRANSACTION_START_RETRY_ENABLED=true DATABASE_TRANSACTION_START_RETRY_MAX_ATTEMPTS=3 # 3 attempts (2 retries); ~30s acquisition tolerance covers a ~20-25s freeze DATABASE_TRANSACTION_START_RETRY_BACKOFF_MIN_MS=50 DATABASE_TRANSACTION_START_RETRY_BACKOFF_MAX_MS=250 DATABASE_TRANSACTION_START_RETRY_BUDGET_PER_SEC=50 DATABASE_TRANSACTION_START_RETRY_BUDGET_BURST=100 Per-pool overrides RUN_OPS_DATABASE_TRANSACTION_* and RUN_OPS_LEGACY_DATABASE_TRANSACTION_* (all seven knobs each) are optional and fall back to the generic set — not needed for v1; the generic set covers the control-plane, run-ops, and run-ops-legacy writer pools. Readers open no transactions and take nothing. Guardrail: the retry only engages when a pool's pool_timeout > maxWait. Prod is fine (DATABASE_POOL_TIMEOUT=60 >> 10). Do not set any writer pool's pool_timeout at or under maxWait, or saturation failures flip from retryable P2028 to non-retryable P2024 and the retry silently stops helping. Rollback Env flip (set inert) or revert. Retry only fires where no SQL ran, and the per-pool token bucket caps a storm. No migration. refs TRI-13295, TRI-12982, TRI-12984
Open Graph Description: What Makes two transaction-resilience behaviors real and env-var configurable, defaults set to the good values, so we can tune during and after the Aug 15 database patch window without a redeploy: ...
X Description: What Makes two transaction-resilience behaviors real and env-var configurable, defaults set to the good values, so we can tune during and after the Aug 15 database patch window without a redeploy: ...
Opengraph URL: https://github.com/triggerdotdev/trigger.dev/pull/4623
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:fe43cd95-6dad-7730-757c-b0eeb69f89af |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | D310:3D538F:6C3B27:A08329:6AFAF46B |
| html-safe-nonce | 0bd42a201c22101223507aac74c24ca85d6b2647b8dc42746ac5e96606c19c39 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJEMzEwOjNENTM4Rjo2QzNCMjc6QTA4MzI5OjZBRkFGNDZCIiwidmlzaXRvcl9pZCI6IjQ4NjE0NTY5OTE0NjI4MTQ4MjciLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | b04779a608e483b64d2351d003b6c5a03023287c87eae00f1ca691fa9958e8ec |
| hovercard-subject-tag | pull_request:4281179044 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/triggerdotdev/trigger.dev/pull/4623/files |
| twitter:image | https://avatars.githubusercontent.com/u/534?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/534?s=400&v=4 |
| og:image:alt | What Makes two transaction-resilience behaviors real and env-var configurable, defaults set to the good values, so we can tune during and after the Aug 15 database patch window without a redeploy: ... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 0178036d8230524a18422872393ab7e708443ae4c2047f758560a2db870c1913 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/triggerdotdev/trigger.dev git https://github.com/triggerdotdev/trigger.dev.git |
| octolytics-dimension-user_id | 95297378 |
| octolytics-dimension-user_login | triggerdotdev |
| octolytics-dimension-repository_id | 572570113 |
| octolytics-dimension-repository_nwo | triggerdotdev/trigger.dev |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 572570113 |
| octolytics-dimension-repository_network_root_nwo | triggerdotdev/trigger.dev |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | a85e96a83bed7871993e6b1ffda24105928eee17 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width