Title: fix(redis): size cold-connection waits to survive a dead handshake by waleedlatif1 · Pull Request #7764 · simstudioai/sim · GitHub
Open Graph Title: fix(redis): size cold-connection waits to survive a dead handshake by waleedlatif1 · Pull Request #7764 · simstudioai/sim
X Title: fix(redis): size cold-connection waits to survive a dead handshake by waleedlatif1 · Pull Request #7764 · simstudioai/sim
Description: Summary Every readiness wait for a cold Redis connection was sized to a single deadline, so it expired while the first attempt was still being diagnosed — a configured retry could never be what rescued it Adds coldConnectionBudgetMs beside the pinned client (lib/core/config/redis-budget.ts): a dead attempt is charged at the larger of what governs it — connectTimeout for a connect that never completes (destroyed outright), or the handshake's command deadlines plus disconnectTimeout, because after the ready check times out ioredis half-closes the socket and a wedged peer never answers with a FIN. Then the client's first reconnect delay, then a healthy attempt. Authenticated URLs cost one command deadline (a timed-out AUTH is fatal), unauthenticated two (CLIENT SETNAME/SETINFO settle before INFO starts); the budget takes the larger so it holds for either without parsing the URL Execution-signal subscriber: readiness budget was the same constant as its commandTimeout (5s/5s). The command deadline stays at 5s — it also bounds the live SUBSCRIBE, and an initial subscribe that rejects fails the run — and the budget is derived in the constructor from the exact options the client is built with: 13.5s Each waiter gets its own deadline. Waiters race the shared readiness signal against a per-waiter timer; the last to leave detaches the listeners, and a signal that settles clears itself. Before this branch the memoized promise shared one timer, so a late joiner inherited only the remainder Shared client warm-up: budget was 10s against a 5s deadline — exactly the moment ioredis first tears a stalled socket down. Now 14.3s, derived from the client's own connect and disconnect deadlines and a pure sharedReconnectDelayMs that retryStrategy uses too The signal subscriber is connected on intent. Constructing the hub is what connects, so executeWorkflowCore — the one path every execution shares — constructs it as its first statement, synchronously and never throwing, and the handshake overlaps the reads and preprocessing ahead of the subscribe. Only tasks that execute a workflow ever subscribe, and they are a minority of runs (over a week, 71% never do; one document-processing task is 61% alone), so neither the Trigger.dev init hook nor the Next server opens it eagerly Verified against the installed ioredis With a server that stalls only the first handshake and never sends a FIN, exact production options: unauthenticated, the socket closes at 12.0s (10s of command deadlines + 2s half-close) and a healthy connection #2 is ready at 12.5s; authenticated, at 7.0s / 7.5s. Today's subscriber budget (5000) fails at 5.0s on connection #1, matching the production failure; the derived 13.5s budget recovers both shapes. What this is and is not Not a strict improvement, stated precisely. In the failure shape observed in production — shared client healthy, subscriber handshake dead — a run now fails at ~13.5s instead of ~5s, and the same wait applies when Redis is simply unreachable. During a total outage the awaited warm-up costs 14.3s before the run starts and then either the shared client's 5s admission deadline or the 13.5s subscribe fails it: roughly 19–28s of worker hold versus ~15s before this PR. That is the whole regression surface, in exchange for recovering from every transient stall. The guarantee is one dead attempt from a fresh or previously-ready client; a wait that begins while the client is already deep in a reconnect loop faces larger delays this does not model. A widened wait also widens the window in which a short execution timeout pre-empts it and reports itself instead of the Redis failure. On the healthy path nothing waits longer: the subscriber's ~40ms handshake now overlaps preprocessing instead of running inside run() before the engine starts, and the 71% of runs that never subscribe never open that connection. Pub/Sub is at-most-once, so the run-fails-if-it-cannot-subscribe stance is kept deliberately — this change makes that subscription recoverable, it does not make it optional. Type of Change Bug fix Testing New tests: the budget helper (command deadlines plus half-close, the connect deadline when larger, the healthy-attempt allowance); the shared client's pure reconnect delay; the warm-up outlasts one dead attempt and still gives up past its budget; the subscriber waits exactly the budget derived from its own options; a subscribe joining another in-flight wait keeps its own full budget; waiter accounting stays exact when one waiter times out before the signal settles; connecting ahead of a subscription never throws on a misconfigured URL; executeWorkflowCore connects synchronously before its first await. Each verified to fail under a targeted sabotage on its intended assertion (dropping the half-close term, a shared timer, never detaching, the connect moved after the first await). Shared test infra changed (redis-config.mock.ts), so the full suite was run on the rebased tree: 50,151 tests pass in apps/sim, 381 in apps/realtime, 222 in packages/utils, 50 in packages/testing. bun run type-check clean in apps/sim, packages/utils, and packages/testing. bun run lint, docs-manifest:check, block-registry check, and all 46 audits in check:audits pass. Checklist Code follows project style guidelines Self-reviewed my changes Tests added/updated and passing No new warnings introduced I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)
Open Graph Description: Summary Every readiness wait for a cold Redis connection was sized to a single deadline, so it expired while the first attempt was still being diagnosed — a configured retry could never be what re...
X Description: Summary Every readiness wait for a cold Redis connection was sized to a single deadline, so it expired while the first attempt was still being diagnosed — a configured retry could never be what re...
Opengraph URL: https://github.com/simstudioai/sim/pull/7764
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:b02387f0-187d-4e2d-d8c8-27e8b4e42f5d |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | D2C8:2E7595:4FA03D:721C1B:6AFB3C0F |
| html-safe-nonce | 27737d082476352a770fd45891914b26d68cd636f34f741352cc689ea723bd1b |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJEMkM4OjJFNzU5NTo0RkEwM0Q6NzIxQzFCOjZBRkIzQzBGIiwidmlzaXRvcl9pZCI6IjE3NTE0NDgzNjE4NTc5MjQxMTEiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | cb35806f12879ff3b0ad68bcbb13639fae15b8f0a074da7dcbab38222fc12f6e |
| hovercard-subject-tag | pull_request:4503027757 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/7764/files |
| twitter:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| og:image:alt | Summary Every readiness wait for a cold Redis connection was sized to a single deadline, so it expired while the first attempt was still being diagnosed — a configured retry could never be what re... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 2169b0dfbef67ed6b128074a5ed45bb7d5d1859783b10178b09a7765ad5f3693 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/simstudioai/sim git https://github.com/simstudioai/sim.git |
| octolytics-dimension-user_id | 199344406 |
| octolytics-dimension-user_login | simstudioai |
| octolytics-dimension-repository_id | 912559512 |
| octolytics-dimension-repository_nwo | simstudioai/sim |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 912559512 |
| octolytics-dimension-repository_network_root_nwo | simstudioai/sim |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 3b4fc8df5ef1c6757e733a23e6b81b6725021823 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width