Title: fix(webhooks): run a delivery whose dedup claim reply was lost or late by abhishekray07 · Pull Request #8999 · simstudioai/sim · GitHub
Open Graph Title: fix(webhooks): run a delivery whose dedup claim reply was lost or late by abhishekray07 · Pull Request #8999 · simstudioai/sim
X Title: fix(webhooks): run a delivery whose dedup claim reply was lost or late by abhishekray07 · Pull Request #8999 · simstudioai/sim
Description: Hey 👋 This bug was found by Opslane. A webhook delivery whose Redis dedup claim applied, but whose reply was lost or late, is answered 200 and never runs. Its redeliveries are skipped too, for up to 2 hours. The job reads completed (duplicate: true) or failed: Command timed out, and no workflow run exists. It takes one Redis hiccup while a delivery is being claimed: a connection reset after the claim script ran (failover, maintenance, a proxy or load balancer closing idle connections, a network blip); a reply slower than SHARED_COMMAND_TIMEOUT_MS (5 s), e.g. behind a slow command. It needs an idempotency key, which Generic Webhook has through idempotencyField and most providers have through a delivery-id header. Why: Lost reply. ioredis replays a command whose reply a dropped connection lost (maxRetriesPerRequest: 5). The replayed CLAIM_REDIS_LEASE_SCRIPT finds the value the first run wrote and returns it as the current owner. atomicallyClaimRedis treats any existing value as another holder's, so executeOrSkipInProgress reports in-progress, and webhook-execution logs "Skipping duplicate webhook delivery already in progress". The delivery skipped itself. Late reply. commandTimeout rejects the EVAL, but the script already ran. The claim sits outside the try/catch that releases it, and the error isn't a RetryableSetupError, so the #7038 requeue path doesn't apply. The key stays in-progress until the lease ends: the execution deadline + 5 min, or WEBHOOK_IN_PROGRESS_LEASE_SECONDS (2 h) with no async timeout. Every redelivery in that window is skipped. Fix apps/sim/lib/core/idempotency/service.ts: CLAIM_REDIS_LEASE_SCRIPT returns "claimed" when the stored value is exactly ARGV[1]. Each claim value carries a fresh claimToken, so only a replay of the same call can match. Another holder's claim still comes back as {0, current}. atomicallyClaimRedis, when the claim round-trip throws, releases this call's own claim by token (the existing DELETE_REDIS_RESULT_IF_OWNER_SCRIPT). It then throws a RetryableSetupError, and webhook-execution's existing setup-failure path requeues the delivery under the same executionId. If the release fails too, the claim may still be held, so it rethrows the original error unchanged and behaves as before. Only the Redis round-trip is wrapped. A missing Redis client still throws as before. Other IdempotencyService users (polling, Stripe webhooks, chat send) get the same two fixes. For a polling or Stripe caller, a lost reply used to mean waiting on its own claim until the lease ended. Verified end to end I ran Sim locally from each branch (next dev) against Postgres (pgvector/pg17) and Redis 7, with protocol-aware fault proxies in front of both. The workflow was a Generic Webhook (idempotencyField: event_id, no auth) followed by a 5 s Wait. Each run sent event A with no fault, then event B with one fault on its dedup claim, then redelivered B 5 s later. I read back async_jobs and workflow_execution_logs, counting distinct executions. "Before" is staging at a0526e7. Fault on B's dedup claim Before After Script runs; the connection drops before the reply 0 runs. Both jobs completed with duplicate: true; key in-progress, TTL 7196 s 1 run; key completed; the redelivery is skipped as a true duplicate Script runs; the reply is held 7 s (timeout is 5 s) 0 runs. Job failed: Command timed out; the redelivery skipped as a duplicate; key in-progress, TTL 7194 s 1 run. Claim released, job requeued (requeued: true); the retry ran once and the redelivery was skipped Control: event A, no fault 1 run 1 run App log for the late case after the change: "Released an idempotency claim with an unknown outcome", then "Requeued webhook execution after retryable setup failure". What this doesn't close: A stall that also outlasts the release. The release runs on the same connection, so it times out too when the stall is longer than about twice the command timeout. The original error is rethrown and the claim may stay orphaned until the lease ends, as on staging today. Closing that would mean letting a requeued attempt recognize its own claim (for example a claim owner derived from the executionId) or a shorter lease renewed by the running job. The database claim path (atomicallyClaimDb, used without Redis) can also leave an orphaned claim when its insert commits and the answer is lost. postgres.js doesn't replay, so the self-skip doesn't happen there. I left it unchanged; it can follow the same pattern. Tests lib/core/idempotency/claim-faults.integration.ts runs against real Redis (skipped without TEST_REDIS_URL). A TCP proxy sits between a prod-configured ioredis client (commandTimeout, maxRetriesPerRequest: 5) and Redis: the claim's reply is dropped → ioredis replays it → the operation runs once and the key ends completed; the claim's reply arrives after the timeout → RetryableSetupError, the key is released, and a retry runs; the stall also outlasts the release → the original (non-retryable) error; control: a delivery during another holder's live claim is still skipped. The proxy matches the claim script across TCP chunk boundaries, and afterAll deletes the suite's Redis keys. With the script line reverted, the lost-reply test fails with { outcome: 'in-progress' }. With the try/catch reverted, the late-reply test fails. The other tests pass either way. bun run test:integration claim-faults: 4 passed. bun run --cwd apps/sim test: all files pass. A handful of unrelated suites hit their 10 s timeout under memory pressure on my machine and pass when rerun alone. bun run lint:check, bun run type-check and bun run check:audits are clean. Checklist Bug fix Tests added and passing I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA) Generated by Opslane
Open Graph Description: Hey 👋 This bug was found by Opslane. A webhook delivery whose Redis dedup claim applied, but whose reply was lost or late, is answered 200 and never runs. Its redeliveries are skipped too, for up t...
X Description: Hey 👋 This bug was found by Opslane. A webhook delivery whose Redis dedup claim applied, but whose reply was lost or late, is answered 200 and never runs. Its redeliveries are skipped too, for up t...
Opengraph URL: https://github.com/simstudioai/sim/pull/8999
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:685a8f43-2311-930f-ea91-6f6e162f1211 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | BD5A:1BD5CA:138F06E:1ACA43E:6AFB719F |
| html-safe-nonce | cba009c9014274243ea1576fc7a520c8136cc50cc158aacba889b1df2efc6ca5 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCRDVBOjFCRDVDQToxMzhGMDZFOjFBQ0E0M0U6NkFGQjcxOUYiLCJ2aXNpdG9yX2lkIjoiNjE1MDU2Nzk1MzM2MDk3NDIzOSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | fe9debe3ad9e10809d32c05defb81c1f9fd13fe9898dc8e814fd34ef4b678e4c |
| hovercard-subject-tag | pull_request:4850787506 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/8999/files |
| twitter:image | https://avatars.githubusercontent.com/u/1710047?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/1710047?s=400&v=4 |
| og:image:alt | Hey 👋 This bug was found by Opslane. A webhook delivery whose Redis dedup claim applied, but whose reply was lost or late, is answered 200 and never runs. Its redeliveries are skipped too, for up t... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 0e29e7974fdda2feeaf5c891efd4b23b0734b23480232b62c7bdef13f40b5b48 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/simstudioai/sim git https://github.com/simstudioai/sim.git |
| octolytics-dimension-user_id | 199344406 |
| octolytics-dimension-user_login | simstudioai |
| octolytics-dimension-repository_id | 912559512 |
| octolytics-dimension-repository_nwo | simstudioai/sim |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 912559512 |
| octolytics-dimension-repository_network_root_nwo | simstudioai/sim |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 2e8872b770cc4d5d0cc1f2026afab747a268e2ce |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width