Title: fix(mothership): never sweep a Chat run while one of its Sim tools is executing by waleedlatif1 · Pull Request #8482 · simstudioai/sim · GitHub
Open Graph Title: fix(mothership): never sweep a Chat run while one of its Sim tools is executing by waleedlatif1 · Pull Request #8482 · simstudioai/sim
X Title: fix(mothership): never sweep a Chat run while one of its Sim tools is executing by waleedlatif1 · Pull Request #8482 · simstudioai/sim
Description: Summary Problem. The orphaned-run sweep could settle a Chat run as interrupted while one of its Sim tools was still executing, which closes tool admission under a worker that still holds the run. Root cause. The sweep treats a leased run as orphaned once it has gone past the recovery window (ORPHANED_RUN_GRACE_MS, one hour) without a status write and no controller holds its chat lock. A Sim tool call writes nothing to its run while it executes. The only sign that it is alive is its execution lease heartbeat, and the sweep never checked it. A tool call that runs longer than the window, such as a long workflow run, therefore looked idle. The default one-hour run deadline hid this. Fix. The leased-run candidate predicate now excludes any run that has an unsettled, unrevoked tool execution whose lease has not expired (execution_lease_expires_at > clock_timestamp()). The same predicate guards both candidate selection and the guarded settle update. settleRuns now locks the candidate run rows (in id order, after the chat rows, matching the controller claim's lock order) before the guarded update. Tool admission locks its run row, so an admission racing the sweep either commits a lease that the update then sees, or finds the run already settled and is refused. settleRuns then locks the candidate runs' unsettled tool execution rows, after the run rows. A lease heartbeat writes only the tool row, so a renewal that commits first is seen by the guarded update, and a later heartbeat finds the lease expired. Behaviour changes A Chat run whose Sim tool holds a live execution lease is no longer swept, however long the tool runs. It becomes eligible once the lease expires, is revoked, or settles, and the run has been idle for the recovery window. The sweep adds one row lock per candidate run inside its existing transaction. The lock order is unchanged relative to controller claims (chats first, then runs, both by id). Legacy-run handling is unchanged. Test plan New integration test: a run idle past the window with a live tool execution lease is not settled (orphaned-runs.integration.ts) New integration test: a tool admitted while the sweep waits on the run's lock is seen, and the run is not settled New integration test: never settles a run whose Sim tool lease a heartbeat renewed as the sweep settled it The new tests fail on the pre-fix orphaned-runs.ts and pass with the fix (confirmed red, then green) The full orphaned-runs integration suite passes (20/20) against disposable Postgres and Redis (bun run test:integration lib/mothership/async-runs/orphaned-runs) bun run type-check (apps/sim), biome, check:test-patterns Staging: a Chat run with a Sim tool call running longer than an hour completes without being settled as interrupted
Open Graph Description: Summary Problem. The orphaned-run sweep could settle a Chat run as interrupted while one of its Sim tools was still executing, which closes tool admission under a worker that still holds the run. R...
X Description: Summary Problem. The orphaned-run sweep could settle a Chat run as interrupted while one of its Sim tools was still executing, which closes tool admission under a worker that still holds the run. R...
Opengraph URL: https://github.com/simstudioai/sim/pull/8482
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:7a118aa9-c61b-5e93-8237-5fd6f8b2705e |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | C988:E4B58:3EC6F2:560DEB:6AFB9975 |
| html-safe-nonce | 9fffb801c82ba95fc1348eb4ff68cfb95b5a963c6e134c951014d9975b33b050 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDOTg4OkU0QjU4OjNFQzZGMjo1NjBERUI6NkFGQjk5NzUiLCJ2aXNpdG9yX2lkIjoiMTg0MzMxNDkyOTE4MDU4MDIxMyIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | e3cd0b8c7aab80c05299dc5630dcc971df92f8f001a1737a604930f3a0b0d479 |
| hovercard-subject-tag | pull_request:4696585681 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/8482/files |
| twitter:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| og:image:alt | Summary Problem. The orphaned-run sweep could settle a Chat run as interrupted while one of its Sim tools was still executing, which closes tool admission under a worker that still holds the run. R... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 3cefd19a936819d82cfb5279d50fb56fe79cb9ec30f0f807e04e89b0c58ffae1 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/simstudioai/sim git https://github.com/simstudioai/sim.git |
| octolytics-dimension-user_id | 199344406 |
| octolytics-dimension-user_login | simstudioai |
| octolytics-dimension-repository_id | 912559512 |
| octolytics-dimension-repository_nwo | simstudioai/sim |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 912559512 |
| octolytics-dimension-repository_network_root_nwo | simstudioai/sim |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 606800bdb1b81d125fc845bdcc262c9fe063fc8f |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width