Title: fix(redis): reclaim a distributed lock that a timed-out acquire may have taken by icecrasher321 · Pull Request #6864 · simstudioai/sim · GitHub
Open Graph Title: fix(redis): reclaim a distributed lock that a timed-out acquire may have taken by icecrasher321 · Pull Request #6864 · simstudioai/sim
X Title: fix(redis): reclaim a distributed lock that a timed-out acquire may have taken by icecrasher321 · Pull Request #6864 · simstudioai/sim
Description: The bug acquireLock awaited SET NX and let a rejection propagate: const result = await redis.set(lockKey, value, 'EX', expirySeconds, 'NX') return result === 'OK' A rejected SET does not mean the server declined it. The client is configured with commandTimeout: 5000 (redis.ts:164), and ioredis rejects locally when that elapses (ioredis/built/Command.js:195, message Command timed out) while the command can still reach Redis and take the lock. When that happens the lock is held by a caller that never learned it won, so it never releases. Every caller treats a throw as "did not acquire", so nothing else releases it either. Every contender skips until the TTL expires. How staging hit it The sim-staging-us-east-1-cron-lambda-errors alarm at 21:06 UTC: acquireLock threw Command timed out in app/api/webhooks/poll/[provider]/route.ts:43 The outer catch returned 500 outlook polling failed: Command timed out after ~10.8s The next scheduled poll and the Lambda retry both got 202 Polling already in progress - skipped, against a lock whose holder had never started polling The 180s TTL cleared it Worth being precise about one thing, because the incident's own remediation note pointed at it: the route's finally/releaseLock is not missing. It lives inside the runDetached task, which never started, because the throw happened at lock acquisition. The poll never ran, so there was nothing to terminate — the lock simply had no owner that could release it. The fix, and why it is opt-in On failure, best-effort compare-and-delete through the existing releaseLock — but only when the caller opts in, because reclaiming unconditionally is unsafe for two caller classes that exist today. Both were caught in review; the first revision of this PR had it on for everyone and was wrong. Callers that fall open. withLeaderLock (leader-lock.ts:36-43) and the MCP OAuth refresh mutex (mcp/oauth/storage.ts:329-336) catch the throw and run their work uncoordinated. If the SET landed, the lock they hold is what keeps everyone else out while they run. Freeing it under them admits a second concurrent runner — for OAuth refresh, two rotations of the same refresh token and an invalid_grant. Callers whose lock value is not unique. The copilot chat lock keys on streamId, which is the client-supplied userMessageId (copilot/chat/post.ts:266 declares it z.string().max(128).optional(); post.ts:592 passes it as streamId). Two sends can carry the same value, so a compare-and-delete from a contender that timed out can match — and delete — the lock the active stream holds. So the option carries its preconditions in its own TSDoc: the value must be unique to the holder, and a throw must mean the caller does no work. Default behavior is byte-for-byte what it was before this PR. Opted in The four cron/poll callers that satisfy both preconditions — each mints its value with generateShortId() and returns 5xx rather than proceeding when acquisition throws: Caller Value On throw app/api/webhooks/poll/[provider] generateShortId() 500 app/api/resume/poll generateShortId() 5xx via route handler app/api/workspace-events/poll generateShortId() 500 app/api/cron/renew-subscriptions generateShortId() 5xx via route handler For those, every branch is better than today or identical to it: Case Before After SET landed, client timed out held for full TTL reclaimed immediately SET never landed, another holder won untouched untouched (value mismatch, no-op) SET never landed, nobody holds it no-op no-op Redis still unreachable TTL backstop TTL backstop (cleanup fails, swallowed) Not opted in, and unchanged: lib/concurrency/leader-lock, lib/mcp/oauth/storage, lib/copilot/request/session/abort, lib/table/cascade-lock, app/api/tools/file/manage. Control flow is unchanged The error still propagates in every case, so callers see exactly what they saw before — the only difference is a best-effort side effect for the four that opt in. Deliberately not converted to return false: that would make the poll route answer Polling already in progress - skipped when polling was not in progress, hiding a real Redis outage from the alarm that correctly fired here. It also makes the existing recovery work as designed — the Lambda retry that got already in progress will now find the lock free and actually poll. Tests Six cases in redis.test.ts: SET NX taking the lock (no cleanup issued), contention returning false (no cleanup issued), a timed-out SET issuing the compare-and-delete and rethrowing when reclaim is on, a timed-out SET issuing no cleanup by default so a fall-open caller keeps holding its lock, a failed cleanup still surfacing the original error, and the no-Redis no-op path. The webhook polling route test now asserts it passes { reclaimOnFailure: true }, pinning the incident's actual fix at the call site rather than only in the primitive. I verified the reclaim test fails against the original implementation (1 failed / 20 passed) and passes with the fix, so it isn't a test that would pass either way. Verification bun run test across redis.test.ts and all affected callers: 64 passed bun run type-check (apps/sim): clean biome check: clean 🤖 Generated with Claude Code
Open Graph Description: The bug acquireLock awaited SET NX and let a rejection propagate: const result = await redis.set(lockKey, value, 'EX', expirySeconds, 'NX') return result === 'OK' A rejected...
X Description: The bug acquireLock awaited SET NX and let a rejection propagate: const result = await redis.set(lockKey, value, 'EX', expirySeconds, 'NX') return result === ...
Opengraph URL: https://github.com/simstudioai/sim/pull/6864
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:4368305f-29e2-f024-b6f8-619f54442505 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | 8C3A:160F17:AEA585:F59D42:6AFB798B |
| html-safe-nonce | ebf29ff26dd8882b771c52e857656364d4ccf59473e6bfc06f1d3d9b5e8012e6 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI4QzNBOjE2MEYxNzpBRUE1ODU6RjU5RDQyOjZBRkI3OThCIiwidmlzaXRvcl9pZCI6IjkxNDQ0MzgwNzMxMjE3OTg1MzkiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | 0173f7eb093ab27861a6852e78a946adc7ba1bc0ea8ca6c4469cc94359668562 |
| hovercard-subject-tag | pull_request:4317645183 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/6864/files |
| twitter:image | https://avatars.githubusercontent.com/u/17950982?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/17950982?s=400&v=4 |
| og:image:alt | The bug acquireLock awaited SET NX and let a rejection propagate: const result = await redis.set(lockKey, value, 'EX', expirySeconds, 'NX') return result === 'OK' A rejected... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 0e29e7974fdda2feeaf5c891efd4b23b0734b23480232b62c7bdef13f40b5b48 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/simstudioai/sim git https://github.com/simstudioai/sim.git |
| octolytics-dimension-user_id | 199344406 |
| octolytics-dimension-user_login | simstudioai |
| octolytics-dimension-repository_id | 912559512 |
| octolytics-dimension-repository_nwo | simstudioai/sim |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 912559512 |
| octolytics-dimension-repository_network_root_nwo | simstudioai/sim |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 2e8872b770cc4d5d0cc1f2026afab747a268e2ce |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width