Title: fix(async-jobs): retry an inline job claim that hit a dropped connection by abhishekray07 · Pull Request #9000 · simstudioai/sim · GitHub
Open Graph Title: fix(async-jobs): retry an inline job claim that hit a dropped connection by abhishekray07 · Pull Request #9000 · simstudioai/sim
X Title: fix(async-jobs): retry an inline job claim that hit a dropped connection by abhishekray07 · Pull Request #9000 · simstudioai/sim
Description: Hey 👋 This bug was found by Opslane. On the database job backend, an accepted webhook delivery is failed for good when its job claim hits a dropped Postgres connection. The provider already got 200, so it won't retry. Nothing in Sim re-runs the job, and the workflow never runs. The job reads failed: Failed query: update "async_jobs" …. It takes one dropped connection right after a delivery is accepted: a Postgres failover or restart; PgBouncer, RDS Proxy or a load balancer closing connections; a network blip. Each event loses the deliveries whose claim was in flight on the affected connection. That's rare per delivery and certain at volume. Generic Webhook and every push provider without executionMode: 'queue' take this path (getInlineJobQueue() in lib/webhooks/processor.ts). Why (lib/core/async-jobs/backends/database.ts): Ingress inserts the job as pending, starts runInline without awaiting it, and answers 200. runInline runs startJob → runner → completeJob. Any throw, including from the claim UPDATE … SET status = 'processing' WHERE status = 'pending', lands in one catch that calls markJobFailed. markJobFailed accepts pending or processing. maxAttempts is stored but never used to re-run a job in this backend. cleanup-stale-executions only marks stale pending/processing jobs failed. The connection can drop before the UPDATE reaches Postgres or after it commits. Either way the job is failed. The fixes for dropped deliveries (#7038) and for ECONNRESET during setup run after this claim, so they don't cover it. Fix lib/core/async-jobs/backends/database.ts: claimInlineJob (new) is what runInline now calls instead of startJob. It retries a claim that threw a retryable infrastructure error (isRetryableInfrastructureError: postgres.js CONNECTION_CLOSED, ECONNRESET, 57P01 and so on). It makes up to 5 attempts with jittered backoff (backoffWithJitter, 250 ms to 2 s, about 4 s in total). Read-back before each retry. The error can arrive after the claim committed, so each retry first reads the row back: processing with this call's own claim token means the claim was ours and committed, so it proceeds; still pending means it claims again; anything else (another claimer's token, completed, cancelled) means it backs off without running. startJob takes an optional claim token and writes it to metadata.claimToken with the claim. A random token rather than started_at, because a duplicate enqueue of the same job id (e.g. Slack Search's stable ids) can race a second claimer in the same millisecond. rowToJob strips the key, so it doesn't show up in the jobs API. No schema change, and the JobQueueBackend signature is unchanged for other callers. Non-retryable errors, and a claim that still fails after the last attempt, are failed as before. Verified end to end I ran Sim locally from each branch (next dev) against Postgres (pgvector/pg17) and Redis 7, with a Postgres wire-protocol fault proxy in front of the database. The workflow was a Generic Webhook followed by a 5 s Wait. Each run sent event A with no fault, then event B, whose job-claim UPDATE the proxy let commit before cutting the connection ahead of the answer. I read back async_jobs and workflow_execution_logs, then again 60 s later. "Before" is staging at a0526e7. Before After Event B: claim commits, answer lost 0 runs; job failed (Failed query: update "async_jobs" …), attempts 1 1 run; job completed, attempts 1 Same, 60 s later still failed, 0 runs unchanged: 1 run Control: event A, no fault 1 run 1 run App log after the change: "Retrying inline job claim after a retryable error", then 300 ms later "Recovered inline job claim that committed before its error". The proxy's hit log shows the fault landed on event B's claim. What this doesn't close: A process that dies while an inline job is pending or running. Deploy, OOM or SIGTERM loses the in-memory job the same way, and the stale-job sweeper marks it failed. Fixing that means the sweeper re-running unstarted jobs while attempts < maxAttempts, keeping the executionId so the run stays exactly-once. That's a larger change I left out. An outage longer than the retry budget (about 4 s) still fails the job, as before. Tests lib/core/async-jobs/backends/inline-claim-faults.integration.ts runs against real PostgreSQL. A TCP proxy, started before @sim/db loads, sits between the app's db client and Postgres: the claim UPDATE commits and its UPDATE 1 answer is dropped → the job runs once, completed, attempts 1; the connection drops before the claim is sent → the job runs once; another claimer takes the row before this claim's dropped connection surfaces → the job is not run. The proxy matches its markers across TCP chunk boundaries, and afterAll deletes the suite's async_jobs rows. Reverting each guard turns its test red: No retry: both fault tests end failed. Retry without the read-back: the committed case is stranded in processing. Read-back without the claim-token fence: the other claimer's job runs. bun run test:integration inline-claim-faults: 3 passed. The existing database.test.ts suite passes unchanged. bun run --cwd apps/sim test: all files pass. One unrelated suite (lib/mothership/agent-cli/services.test.ts) hit its 10 s timeout under memory pressure on my machine and passes alone. bun run lint:check, bun run type-check and bun run check:audits are clean. Checklist Bug fix Tests added and passing I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA) Generated by Opslane
Open Graph Description: Hey 👋 This bug was found by Opslane. On the database job backend, an accepted webhook delivery is failed for good when its job claim hits a dropped Postgres connection. The provider already got 200...
X Description: Hey 👋 This bug was found by Opslane. On the database job backend, an accepted webhook delivery is failed for good when its job claim hits a dropped Postgres connection. The provider already got 200...
Opengraph URL: https://github.com/simstudioai/sim/pull/9000
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:80b13c43-218e-a2ab-0c76-419005133654 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | CD5E:1C135C:32F960:47A2C9:6AFB7169 |
| html-safe-nonce | 6c5d76bffe3f4f848ed457e0bb7a656040f2d829047c9154418ec2addfbdf007 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDRDVFOjFDMTM1QzozMkY5NjA6NDdBMkM5OjZBRkI3MTY5IiwidmlzaXRvcl9pZCI6IjMxMjQ1Njk1NDk3ODU3NTYwMDkiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | f8b22f6b9b757624872c9f26adf07fcdd09cb152b57525932792cb9b03cb4bd4 |
| hovercard-subject-tag | pull_request:4850787796 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/9000/files |
| twitter:image | https://avatars.githubusercontent.com/u/1710047?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/1710047?s=400&v=4 |
| og:image:alt | Hey 👋 This bug was found by Opslane. On the database job backend, an accepted webhook delivery is failed for good when its job claim hits a dropped Postgres connection. The provider already got 200... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 0e29e7974fdda2feeaf5c891efd4b23b0734b23480232b62c7bdef13f40b5b48 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/simstudioai/sim git https://github.com/simstudioai/sim.git |
| octolytics-dimension-user_id | 199344406 |
| octolytics-dimension-user_login | simstudioai |
| octolytics-dimension-repository_id | 912559512 |
| octolytics-dimension-repository_nwo | simstudioai/sim |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 912559512 |
| octolytics-dimension-repository_network_root_nwo | simstudioai/sim |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 2e8872b770cc4d5d0cc1f2026afab747a268e2ce |
| ui-target | canary-1 |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width