René's URL Explorer Experiment


Title: fix(webhooks): run a delivery whose dedup claim reply was lost or late by abhishekray07 · Pull Request #8999 · simstudioai/sim · GitHub

Open Graph Title: fix(webhooks): run a delivery whose dedup claim reply was lost or late by abhishekray07 · Pull Request #8999 · simstudioai/sim

X Title: fix(webhooks): run a delivery whose dedup claim reply was lost or late by abhishekray07 · Pull Request #8999 · simstudioai/sim

Description: Hey 👋 This bug was found by Opslane. A webhook delivery whose Redis dedup claim applied, but whose reply was lost or late, is answered 200 and never runs. Its redeliveries are skipped too, for up to 2 hours. The job reads completed (duplicate: true) or failed: Command timed out, and no workflow run exists. It takes one Redis hiccup while a delivery is being claimed: a connection reset after the claim script ran (failover, maintenance, a proxy or load balancer closing idle connections, a network blip); a reply slower than SHARED_COMMAND_TIMEOUT_MS (5 s), e.g. behind a slow command. It needs an idempotency key, which Generic Webhook has through idempotencyField and most providers have through a delivery-id header. Why: Lost reply. ioredis replays a command whose reply a dropped connection lost (maxRetriesPerRequest: 5). The replayed CLAIM_REDIS_LEASE_SCRIPT finds the value the first run wrote and returns it as the current owner. atomicallyClaimRedis treats any existing value as another holder's, so executeOrSkipInProgress reports in-progress, and webhook-execution logs "Skipping duplicate webhook delivery already in progress". The delivery skipped itself. Late reply. commandTimeout rejects the EVAL, but the script already ran. The claim sits outside the try/catch that releases it, and the error isn't a RetryableSetupError, so the #7038 requeue path doesn't apply. The key stays in-progress until the lease ends: the execution deadline + 5 min, or WEBHOOK_IN_PROGRESS_LEASE_SECONDS (2 h) with no async timeout. Every redelivery in that window is skipped. Fix apps/sim/lib/core/idempotency/service.ts: CLAIM_REDIS_LEASE_SCRIPT returns "claimed" when the stored value is exactly ARGV[1]. Each claim value carries a fresh claimToken, so only a replay of the same call can match. Another holder's claim still comes back as {0, current}. atomicallyClaimRedis, when the claim round-trip throws, releases this call's own claim by token (the existing DELETE_REDIS_RESULT_IF_OWNER_SCRIPT). It then throws a RetryableSetupError, and webhook-execution's existing setup-failure path requeues the delivery under the same executionId. If the release fails too, the claim may still be held, so it rethrows the original error unchanged and behaves as before. Only the Redis round-trip is wrapped. A missing Redis client still throws as before. Other IdempotencyService users (polling, Stripe webhooks, chat send) get the same two fixes. For a polling or Stripe caller, a lost reply used to mean waiting on its own claim until the lease ended. Verified end to end I ran Sim locally from each branch (next dev) against Postgres (pgvector/pg17) and Redis 7, with protocol-aware fault proxies in front of both. The workflow was a Generic Webhook (idempotencyField: event_id, no auth) followed by a 5 s Wait. Each run sent event A with no fault, then event B with one fault on its dedup claim, then redelivered B 5 s later. I read back async_jobs and workflow_execution_logs, counting distinct executions. "Before" is staging at a0526e7. Fault on B's dedup claim Before After Script runs; the connection drops before the reply 0 runs. Both jobs completed with duplicate: true; key in-progress, TTL 7196 s 1 run; key completed; the redelivery is skipped as a true duplicate Script runs; the reply is held 7 s (timeout is 5 s) 0 runs. Job failed: Command timed out; the redelivery skipped as a duplicate; key in-progress, TTL 7194 s 1 run. Claim released, job requeued (requeued: true); the retry ran once and the redelivery was skipped Control: event A, no fault 1 run 1 run App log for the late case after the change: "Released an idempotency claim with an unknown outcome", then "Requeued webhook execution after retryable setup failure". What this doesn't close: A stall that also outlasts the release. The release runs on the same connection, so it times out too when the stall is longer than about twice the command timeout. The original error is rethrown and the claim may stay orphaned until the lease ends, as on staging today. Closing that would mean letting a requeued attempt recognize its own claim (for example a claim owner derived from the executionId) or a shorter lease renewed by the running job. The database claim path (atomicallyClaimDb, used without Redis) can also leave an orphaned claim when its insert commits and the answer is lost. postgres.js doesn't replay, so the self-skip doesn't happen there. I left it unchanged; it can follow the same pattern. Tests lib/core/idempotency/claim-faults.integration.ts runs against real Redis (skipped without TEST_REDIS_URL). A TCP proxy sits between a prod-configured ioredis client (commandTimeout, maxRetriesPerRequest: 5) and Redis: the claim's reply is dropped → ioredis replays it → the operation runs once and the key ends completed; the claim's reply arrives after the timeout → RetryableSetupError, the key is released, and a retry runs; the stall also outlasts the release → the original (non-retryable) error; control: a delivery during another holder's live claim is still skipped. The proxy matches the claim script across TCP chunk boundaries, and afterAll deletes the suite's Redis keys. With the script line reverted, the lost-reply test fails with { outcome: 'in-progress' }. With the try/catch reverted, the late-reply test fails. The other tests pass either way. bun run test:integration claim-faults: 4 passed. bun run --cwd apps/sim test: all files pass. A handful of unrelated suites hit their 10 s timeout under memory pressure on my machine and pass when rerun alone. bun run lint:check, bun run type-check and bun run check:audits are clean. Checklist Bug fix Tests added and passing I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA) Generated by Opslane

Open Graph Description: Hey 👋 This bug was found by Opslane. A webhook delivery whose Redis dedup claim applied, but whose reply was lost or late, is answered 200 and never runs. Its redeliveries are skipped too, for up t...

X Description: Hey 👋 This bug was found by Opslane. A webhook delivery whose Redis dedup claim applied, but whose reply was lost or late, is answered 200 and never runs. Its redeliveries are skipped too, for up t...

Opengraph URL: https://github.com/simstudioai/sim/pull/8999

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:685a8f43-2311-930f-ea91-6f6e162f1211
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idBD5A:1BD5CA:138F06E:1ACA43E:6AFB719F
html-safe-noncecba009c9014274243ea1576fc7a520c8136cc50cc158aacba889b1df2efc6ca5
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCRDVBOjFCRDVDQToxMzhGMDZFOjFBQ0E0M0U6NkFGQjcxOUYiLCJ2aXNpdG9yX2lkIjoiNjE1MDU2Nzk1MzM2MDk3NDIzOSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmacfe9debe3ad9e10809d32c05defb81c1f9fd13fe9898dc8e814fd34ef4b678e4c
hovercard-subject-tagpull_request:4850787506
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/8999/files
twitter:imagehttps://avatars.githubusercontent.com/u/1710047?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/1710047?s=400&v=4
og:image:altHey 👋 This bug was found by Opslane. A webhook delivery whose Redis dedup claim applied, but whose reply was lost or late, is answered 200 and never runs. Its redeliveries are skipped too, for up t...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None0e29e7974fdda2feeaf5c891efd4b23b0734b23480232b62c7bdef13f40b5b48
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/simstudioai/sim git https://github.com/simstudioai/sim.git
octolytics-dimension-user_id199344406
octolytics-dimension-user_loginsimstudioai
octolytics-dimension-repository_id912559512
octolytics-dimension-repository_nwosimstudioai/sim
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id912559512
octolytics-dimension-repository_network_root_nwosimstudioai/sim
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release2e8872b770cc4d5d0cc1f2026afab747a268e2ce
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/simstudioai/sim/pull/8999/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F8999%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F8999%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=simstudioai%2Fsim
Reloadhttps://github.com/simstudioai/sim/pull/8999/files
Reloadhttps://github.com/simstudioai/sim/pull/8999/files
Reloadhttps://github.com/simstudioai/sim/pull/8999/files
simstudioaihttps://github.com/simstudioai
simhttps://github.com/simstudioai/sim
Codehttps://github.com/simstudioai/sim
Issues129 (129)https://github.com/simstudioai/sim/issues
Pull requests199 (199)https://github.com/simstudioai/sim/pulls
Actionshttps://github.com/simstudioai/sim/actions
Projectshttps://github.com/simstudioai/sim/projects
Security and qualityhttps://github.com/simstudioai/sim/security
Insightshttps://github.com/simstudioai/sim/pulse
Sign up for GitHub https://github.com/signup?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
abhishekray07https://github.com/abhishekray07
simstudioai:staginghttps://github.com/simstudioai/sim/tree/staging
abhishekray07:fix/webhook-dedup-claim-lost-replyhttps://github.com/abhishekray07/sim/tree/fix/webhook-dedup-claim-lost-reply
Conversation 8 https://github.com/simstudioai/sim/pull/8999
Commits 2 https://github.com/simstudioai/sim/pull/8999/commits
Checks 1 https://github.com/simstudioai/sim/pull/8999/checks
Files changed https://github.com/simstudioai/sim/pull/8999/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/8999/files
fix(webhooks): run a delivery whose dedup claim reply was lost or late https://github.com/simstudioai/sim/pull/8999/files#top
Show all changes 2 commits https://github.com/simstudioai/sim/pull/8999/files
8432e19 fix(webhooks): run a delivery whose dedup claim reply was lost or late abhishekray07 Oct 13, 2026 https://github.com/simstudioai/sim/pull/8999/commits/8432e19ba2dbec1159cd5ea6e30605e6d9a4b59d
493fb2d test(idempotency): match split claim markers, gate the live-holder te… abhishekray07 Oct 14, 2026 https://github.com/simstudioai/sim/pull/8999/commits/493fb2d13a108eccdd5f91bf05f9b162cf8e9c2c
Clear filters https://github.com/simstudioai/sim/pull/8999/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/8999/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/8999/files
claim-faults.integration.ts https://github.com/simstudioai/sim/pull/8999/files#diff-b31b10e94858cdba0d21cd1d5934c356d904cb42d899490624e36d0337533e98
service.ts https://github.com/simstudioai/sim/pull/8999/files#diff-6622bed9a9e8ea71519df9b69f50201260f8b7aa4742fe7973f87212b37dc024
apps/sim/lib/core/idempotency/claim-faults.integration.tshttps://github.com/simstudioai/sim/pull/8999/files#diff-b31b10e94858cdba0d21cd1d5934c356d904cb42d899490624e36d0337533e98
View file https://github.com/simstudioai/sim/blob/493fb2d13a108eccdd5f91bf05f9b162cf8e9c2c/apps/sim/lib/core/idempotency/claim-faults.integration.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/8999/{{ revealButtonHref }}
greptile-appshttps://github.com/apps/greptile-apps
Oct 13, 2026https://github.com/simstudioai/sim/pull/8999/files#r4265198100
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
https://github.com/simstudioai/sim/pull/8999/files
sourcehttps://github.com/simstudioai/sim/blob/main/CLAUDE.md
Please reload this pagehttps://github.com/simstudioai/sim/pull/8999/files
abhishekray07https://github.com/abhishekray07
Oct 14, 2026https://github.com/simstudioai/sim/pull/8999/files#r4265977596
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
493fb2dhttps://github.com/simstudioai/sim/commit/493fb2d13a108eccdd5f91bf05f9b162cf8e9c2c
Please reload this pagehttps://github.com/simstudioai/sim/pull/8999/files
greptile-appshttps://github.com/apps/greptile-apps
Oct 13, 2026https://github.com/simstudioai/sim/pull/8999/files#r4265198093
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
https://github.com/simstudioai/sim/pull/8999/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/8999/files
abhishekray07https://github.com/abhishekray07
Oct 14, 2026https://github.com/simstudioai/sim/pull/8999/files#r4265977446
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
493fb2dhttps://github.com/simstudioai/sim/commit/493fb2d13a108eccdd5f91bf05f9b162cf8e9c2c
Please reload this pagehttps://github.com/simstudioai/sim/pull/8999/files
apps/sim/lib/core/idempotency/service.tshttps://github.com/simstudioai/sim/pull/8999/files#diff-6622bed9a9e8ea71519df9b69f50201260f8b7aa4742fe7973f87212b37dc024
View file https://github.com/simstudioai/sim/blob/493fb2d13a108eccdd5f91bf05f9b162cf8e9c2c/apps/sim/lib/core/idempotency/service.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/8999/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/8999/files#diff-6622bed9a9e8ea71519df9b69f50201260f8b7aa4742fe7973f87212b37dc024
https://github.com/simstudioai/sim/pull/8999/files#diff-6622bed9a9e8ea71519df9b69f50201260f8b7aa4742fe7973f87212b37dc024
https://github.com/simstudioai/sim/pull/8999/files#diff-6622bed9a9e8ea71519df9b69f50201260f8b7aa4742fe7973f87212b37dc024
https://github.com/simstudioai/sim/pull/8999/files#diff-6622bed9a9e8ea71519df9b69f50201260f8b7aa4742fe7973f87212b37dc024
https://github.com/simstudioai/sim/pull/8999/files#diff-6622bed9a9e8ea71519df9b69f50201260f8b7aa4742fe7973f87212b37dc024
https://github.com/simstudioai/sim/pull/8999/files#diff-6622bed9a9e8ea71519df9b69f50201260f8b7aa4742fe7973f87212b37dc024
https://github.com/simstudioai/sim/pull/8999/files#diff-6622bed9a9e8ea71519df9b69f50201260f8b7aa4742fe7973f87212b37dc024
https://github.com/simstudioai/sim/pull/8999/files#diff-6622bed9a9e8ea71519df9b69f50201260f8b7aa4742fe7973f87212b37dc024
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.