René's URL Explorer Experiment


Title: fix(async-jobs): retry an inline job claim that hit a dropped connection by abhishekray07 · Pull Request #9000 · simstudioai/sim · GitHub

Open Graph Title: fix(async-jobs): retry an inline job claim that hit a dropped connection by abhishekray07 · Pull Request #9000 · simstudioai/sim

X Title: fix(async-jobs): retry an inline job claim that hit a dropped connection by abhishekray07 · Pull Request #9000 · simstudioai/sim

Description: Hey 👋 This bug was found by Opslane. On the database job backend, an accepted webhook delivery is failed for good when its job claim hits a dropped Postgres connection. The provider already got 200, so it won't retry. Nothing in Sim re-runs the job, and the workflow never runs. The job reads failed: Failed query: update "async_jobs" …. It takes one dropped connection right after a delivery is accepted: a Postgres failover or restart; PgBouncer, RDS Proxy or a load balancer closing connections; a network blip. Each event loses the deliveries whose claim was in flight on the affected connection. That's rare per delivery and certain at volume. Generic Webhook and every push provider without executionMode: 'queue' take this path (getInlineJobQueue() in lib/webhooks/processor.ts). Why (lib/core/async-jobs/backends/database.ts): Ingress inserts the job as pending, starts runInline without awaiting it, and answers 200. runInline runs startJob → runner → completeJob. Any throw, including from the claim UPDATE … SET status = 'processing' WHERE status = 'pending', lands in one catch that calls markJobFailed. markJobFailed accepts pending or processing. maxAttempts is stored but never used to re-run a job in this backend. cleanup-stale-executions only marks stale pending/processing jobs failed. The connection can drop before the UPDATE reaches Postgres or after it commits. Either way the job is failed. The fixes for dropped deliveries (#7038) and for ECONNRESET during setup run after this claim, so they don't cover it. Fix lib/core/async-jobs/backends/database.ts: claimInlineJob (new) is what runInline now calls instead of startJob. It retries a claim that threw a retryable infrastructure error (isRetryableInfrastructureError: postgres.js CONNECTION_CLOSED, ECONNRESET, 57P01 and so on). It makes up to 5 attempts with jittered backoff (backoffWithJitter, 250 ms to 2 s, about 4 s in total). Read-back before each retry. The error can arrive after the claim committed, so each retry first reads the row back: processing with this call's own claim token means the claim was ours and committed, so it proceeds; still pending means it claims again; anything else (another claimer's token, completed, cancelled) means it backs off without running. startJob takes an optional claim token and writes it to metadata.claimToken with the claim. A random token rather than started_at, because a duplicate enqueue of the same job id (e.g. Slack Search's stable ids) can race a second claimer in the same millisecond. rowToJob strips the key, so it doesn't show up in the jobs API. No schema change, and the JobQueueBackend signature is unchanged for other callers. Non-retryable errors, and a claim that still fails after the last attempt, are failed as before. Verified end to end I ran Sim locally from each branch (next dev) against Postgres (pgvector/pg17) and Redis 7, with a Postgres wire-protocol fault proxy in front of the database. The workflow was a Generic Webhook followed by a 5 s Wait. Each run sent event A with no fault, then event B, whose job-claim UPDATE the proxy let commit before cutting the connection ahead of the answer. I read back async_jobs and workflow_execution_logs, then again 60 s later. "Before" is staging at a0526e7. Before After Event B: claim commits, answer lost 0 runs; job failed (Failed query: update "async_jobs" …), attempts 1 1 run; job completed, attempts 1 Same, 60 s later still failed, 0 runs unchanged: 1 run Control: event A, no fault 1 run 1 run App log after the change: "Retrying inline job claim after a retryable error", then 300 ms later "Recovered inline job claim that committed before its error". The proxy's hit log shows the fault landed on event B's claim. What this doesn't close: A process that dies while an inline job is pending or running. Deploy, OOM or SIGTERM loses the in-memory job the same way, and the stale-job sweeper marks it failed. Fixing that means the sweeper re-running unstarted jobs while attempts < maxAttempts, keeping the executionId so the run stays exactly-once. That's a larger change I left out. An outage longer than the retry budget (about 4 s) still fails the job, as before. Tests lib/core/async-jobs/backends/inline-claim-faults.integration.ts runs against real PostgreSQL. A TCP proxy, started before @sim/db loads, sits between the app's db client and Postgres: the claim UPDATE commits and its UPDATE 1 answer is dropped → the job runs once, completed, attempts 1; the connection drops before the claim is sent → the job runs once; another claimer takes the row before this claim's dropped connection surfaces → the job is not run. The proxy matches its markers across TCP chunk boundaries, and afterAll deletes the suite's async_jobs rows. Reverting each guard turns its test red: No retry: both fault tests end failed. Retry without the read-back: the committed case is stranded in processing. Read-back without the claim-token fence: the other claimer's job runs. bun run test:integration inline-claim-faults: 3 passed. The existing database.test.ts suite passes unchanged. bun run --cwd apps/sim test: all files pass. One unrelated suite (lib/mothership/agent-cli/services.test.ts) hit its 10 s timeout under memory pressure on my machine and passes alone. bun run lint:check, bun run type-check and bun run check:audits are clean. Checklist Bug fix Tests added and passing I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA) Generated by Opslane

Open Graph Description: Hey 👋 This bug was found by Opslane. On the database job backend, an accepted webhook delivery is failed for good when its job claim hits a dropped Postgres connection. The provider already got 200...

X Description: Hey 👋 This bug was found by Opslane. On the database job backend, an accepted webhook delivery is failed for good when its job claim hits a dropped Postgres connection. The provider already got 200...

Opengraph URL: https://github.com/simstudioai/sim/pull/9000

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:80b13c43-218e-a2ab-0c76-419005133654
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idCD5E:1C135C:32F960:47A2C9:6AFB7169
html-safe-nonce6c5d76bffe3f4f848ed457e0bb7a656040f2d829047c9154418ec2addfbdf007
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDRDVFOjFDMTM1QzozMkY5NjA6NDdBMkM5OjZBRkI3MTY5IiwidmlzaXRvcl9pZCI6IjMxMjQ1Njk1NDk3ODU3NTYwMDkiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmacf8b22f6b9b757624872c9f26adf07fcdd09cb152b57525932792cb9b03cb4bd4
hovercard-subject-tagpull_request:4850787796
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/9000/files
twitter:imagehttps://avatars.githubusercontent.com/u/1710047?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/1710047?s=400&v=4
og:image:altHey 👋 This bug was found by Opslane. On the database job backend, an accepted webhook delivery is failed for good when its job claim hits a dropped Postgres connection. The provider already got 200...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None0e29e7974fdda2feeaf5c891efd4b23b0734b23480232b62c7bdef13f40b5b48
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/simstudioai/sim git https://github.com/simstudioai/sim.git
octolytics-dimension-user_id199344406
octolytics-dimension-user_loginsimstudioai
octolytics-dimension-repository_id912559512
octolytics-dimension-repository_nwosimstudioai/sim
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id912559512
octolytics-dimension-repository_network_root_nwosimstudioai/sim
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release2e8872b770cc4d5d0cc1f2026afab747a268e2ce
ui-targetcanary-1
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/simstudioai/sim/pull/9000/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F9000%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F9000%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=simstudioai%2Fsim
Reloadhttps://github.com/simstudioai/sim/pull/9000/files
Reloadhttps://github.com/simstudioai/sim/pull/9000/files
Reloadhttps://github.com/simstudioai/sim/pull/9000/files
simstudioaihttps://github.com/simstudioai
simhttps://github.com/simstudioai/sim
Codehttps://github.com/simstudioai/sim
Issues129 (129)https://github.com/simstudioai/sim/issues
Pull requests198 (198)https://github.com/simstudioai/sim/pulls
Actionshttps://github.com/simstudioai/sim/actions
Projectshttps://github.com/simstudioai/sim/projects
Security and qualityhttps://github.com/simstudioai/sim/security
Insightshttps://github.com/simstudioai/sim/pulse
Sign up for GitHub https://github.com/signup?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
abhishekray07https://github.com/abhishekray07
simstudioai:staginghttps://github.com/simstudioai/sim/tree/staging
abhishekray07:fix/inline-job-claim-dropped-connectionhttps://github.com/abhishekray07/sim/tree/fix/inline-job-claim-dropped-connection
Conversation 8 https://github.com/simstudioai/sim/pull/9000
Commits 2 https://github.com/simstudioai/sim/pull/9000/commits
Checks 1 https://github.com/simstudioai/sim/pull/9000/checks
Files changed https://github.com/simstudioai/sim/pull/9000/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/9000/files
fix(async-jobs): retry an inline job claim that hit a dropped connection https://github.com/simstudioai/sim/pull/9000/files#top
Show all changes 2 commits https://github.com/simstudioai/sim/pull/9000/files
7f9bb9b fix(async-jobs): retry an inline job claim that hit a dropped connection abhishekray07 Oct 13, 2026 https://github.com/simstudioai/sim/pull/9000/commits/7f9bb9b929ac632f4b7a9607a7b64029c7fa4655
3d7369f fix(async-jobs): fence the inline claim read-back on a claim token, n… abhishekray07 Oct 14, 2026 https://github.com/simstudioai/sim/pull/9000/commits/3d7369f0937a70ada00821eb7b420d2bb1e57ba1
Clear filters https://github.com/simstudioai/sim/pull/9000/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/9000/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/9000/files
database.ts https://github.com/simstudioai/sim/pull/9000/files#diff-fefc9fd260f80579ec85711ad364032eba5dc97e39be0933bd84841d8d83e4c6
inline-claim-faults.integration.ts https://github.com/simstudioai/sim/pull/9000/files#diff-63e045bbbd3382c4edcf4a179a2a5d66e415f04922c22790d0730d7e523bc3d3
apps/sim/lib/core/async-jobs/backends/database.tshttps://github.com/simstudioai/sim/pull/9000/files#diff-fefc9fd260f80579ec85711ad364032eba5dc97e39be0933bd84841d8d83e4c6
View file https://github.com/simstudioai/sim/blob/3d7369f0937a70ada00821eb7b420d2bb1e57ba1/apps/sim/lib/core/async-jobs/backends/database.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/9000/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/9000/files#diff-fefc9fd260f80579ec85711ad364032eba5dc97e39be0933bd84841d8d83e4c6
https://github.com/simstudioai/sim/pull/9000/files#diff-fefc9fd260f80579ec85711ad364032eba5dc97e39be0933bd84841d8d83e4c6
https://github.com/simstudioai/sim/pull/9000/files#diff-fefc9fd260f80579ec85711ad364032eba5dc97e39be0933bd84841d8d83e4c6
https://github.com/simstudioai/sim/pull/9000/files#diff-fefc9fd260f80579ec85711ad364032eba5dc97e39be0933bd84841d8d83e4c6
https://github.com/simstudioai/sim/pull/9000/files#diff-fefc9fd260f80579ec85711ad364032eba5dc97e39be0933bd84841d8d83e4c6
https://github.com/simstudioai/sim/pull/9000/files#diff-fefc9fd260f80579ec85711ad364032eba5dc97e39be0933bd84841d8d83e4c6
https://github.com/simstudioai/sim/pull/9000/files#diff-fefc9fd260f80579ec85711ad364032eba5dc97e39be0933bd84841d8d83e4c6
apps/sim/lib/core/async-jobs/backends/inline-claim-faults.integration.tshttps://github.com/simstudioai/sim/pull/9000/files#diff-63e045bbbd3382c4edcf4a179a2a5d66e415f04922c22790d0730d7e523bc3d3
View file https://github.com/simstudioai/sim/blob/3d7369f0937a70ada00821eb7b420d2bb1e57ba1/apps/sim/lib/core/async-jobs/backends/inline-claim-faults.integration.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/9000/{{ revealButtonHref }}
greptile-appshttps://github.com/apps/greptile-apps
Oct 13, 2026https://github.com/simstudioai/sim/pull/9000/files#r4265199616
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
https://github.com/simstudioai/sim/pull/9000/files
sourcehttps://github.com/simstudioai/sim/blob/main/CLAUDE.md
Please reload this pagehttps://github.com/simstudioai/sim/pull/9000/files
abhishekray07https://github.com/abhishekray07
Oct 14, 2026https://github.com/simstudioai/sim/pull/9000/files#r4265978079
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
3d7369fhttps://github.com/simstudioai/sim/commit/3d7369f0937a70ada00821eb7b420d2bb1e57ba1
Please reload this pagehttps://github.com/simstudioai/sim/pull/9000/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.