René's URL Explorer Experiment


Title: fix(mothership): settle chat runs no controller owns by waleedlatif1 · Pull Request #8457 · simstudioai/sim · GitHub

Open Graph Title: fix(mothership): settle chat runs no controller owns by waleedlatif1 · Pull Request #8457 · simstudioai/sim

X Title: fix(mothership): settle chat runs no controller owns by waleedlatif1 · Pull Request #8457 · simstudioai/sim

Description: Summary A chat run could stay active or paused_waiting_for_tool forever if it never reached a terminal status. That happened when its controller died (deploy, OOM), was superseded with no successor, or was stopped when no controller remained. Nothing ever settled these runs. Production had a large backlog of them, almost all from before the controller-lease protocol. Some chats kept pointing at a dead run, so they could show as busy. cleanup-stale-executions now sweeps runs that no controller owns: Leased runs are settled when their stream holds no chat lock, their replay buffer has expired, and they have been idle for more than an hour. Liveness comes only from the heartbeated chat lock, which a live controller renews every 20 s for its whole life, so a long-running turn is never touched. Pre-lease runs (tool_execution_version < 2, no lease) are settled after 24 h idle. They keep updated_at (so the retention clock isn't reset), get completedAt = updated_at (so they don't all land on one day), and carry their own filterable error text. Current headless runs are never swept; their own lifecycle settles them. Stopping a run that no controller holds now settles it as cancelled, instead of leaving it active. cancelled requires a recorded Stop, checked inside the same UPDATE, so a run superseded by a newer turn is never mislabelled. Every settle is atomic against the controller finalizing or a successor claiming the run: status and owner token are re-checked in one UPDATE. Chats are locked before runs, in the claim path's order, so the two can't deadlock. Chat markers are cleared in one statement per batch. A cron run handles at most 5k rows, in batches of 500, with a short pause between batches. Slack-search Assistant runs, which own their run row, never recorded success or a stop: only failures were written, so a successful run stayed active. They now settle complete, cancelled or error once, through the shared run-status helper. This backlog is the only backfill; no separate one is needed. The sweep drains it gradually after deploy. Testing New orphaned-runs.integration.ts runs against real PostgreSQL and Redis: settle and skip cases: lock held, recovering controller, replay buffer present, fresh run, current headless run, legacy threshold, and never cancelling an unstopped run; sweep and Stop racing controller finalize and successor claim; a claim-vs-settle deadlock stress test. New slack-search/assistant.integration.ts (real PostgreSQL): completed, stopped and failed Slack-search turns settle complete, cancelled and error. Each guard was reverted and its test went red (Stop-row check, unfinished-status check, owner-token check, lock order). bun run lint, bun run type-check, bun run check:audits, unit tests for the cron route and lib/mothership/request, and the replay-budget integration suite all pass.

Open Graph Description: Summary A chat run could stay active or paused_waiting_for_tool forever if it never reached a terminal status. That happened when its controller died (deploy, OOM), was superseded with no successor...

X Description: Summary A chat run could stay active or paused_waiting_for_tool forever if it never reached a terminal status. That happened when its controller died (deploy, OOM), was superseded with no successor...

Opengraph URL: https://github.com/simstudioai/sim/pull/8457

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:b5d0b023-78c9-1b80-86ec-dfab88963ebe
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idCAAE:D16D6:194CFC0:229498B:6AFB90B8
html-safe-nonceae687e789bcdf02f255587faad0cc0db2536a7ace04757ca45c49787593fa14a
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDQUFFOkQxNkQ2OjE5NENGQzA6MjI5NDk4Qjo2QUZCOTBCOCIsInZpc2l0b3JfaWQiOiIzODI3MTk0OTAxOTg0Njc4MDcyIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0=
visitor-hmac9ddb1d181c732805116744ef282714a3b05982d3a885f7c564d21b8935926e6c
hovercard-subject-tagpull_request:4688270072
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/8457/files
twitter:imagehttps://avatars.githubusercontent.com/u/40672544?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/40672544?s=400&v=4
og:image:altSummary A chat run could stay active or paused_waiting_for_tool forever if it never reached a terminal status. That happened when its controller died (deploy, OOM), was superseded with no successor...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None4f01efe06174a5ba6342fd58e42d02295e0675bedbb465d6bef2dbaec8e46556
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/simstudioai/sim git https://github.com/simstudioai/sim.git
octolytics-dimension-user_id199344406
octolytics-dimension-user_loginsimstudioai
octolytics-dimension-repository_id912559512
octolytics-dimension-repository_nwosimstudioai/sim
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id912559512
octolytics-dimension-repository_network_root_nwosimstudioai/sim
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
releasea52db8866b6543b4e83730cd6a6b1360f1a9e4dc
ui-targetcanary-1
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/simstudioai/sim/pull/8457/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F8457%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F8457%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=simstudioai%2Fsim
Reloadhttps://github.com/simstudioai/sim/pull/8457/files
Reloadhttps://github.com/simstudioai/sim/pull/8457/files
Reloadhttps://github.com/simstudioai/sim/pull/8457/files
simstudioaihttps://github.com/simstudioai
simhttps://github.com/simstudioai/sim
Codehttps://github.com/simstudioai/sim
Issues129 (129)https://github.com/simstudioai/sim/issues
Pull requests198 (198)https://github.com/simstudioai/sim/pulls
Actionshttps://github.com/simstudioai/sim/actions
Projectshttps://github.com/simstudioai/sim/projects
Security and qualityhttps://github.com/simstudioai/sim/security
Insightshttps://github.com/simstudioai/sim/pulse
Sign up for GitHub https://github.com/signup?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
waleedlatif1https://github.com/waleedlatif1
staginghttps://github.com/simstudioai/sim/tree/staging
fix/orphaned-run-settlementhttps://github.com/simstudioai/sim/tree/fix/orphaned-run-settlement
Conversation 28 https://github.com/simstudioai/sim/pull/8457
Commits 7 https://github.com/simstudioai/sim/pull/8457/commits
Checks 31 https://github.com/simstudioai/sim/pull/8457/checks
Files changed https://github.com/simstudioai/sim/pull/8457/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/8457/files
fix(mothership): settle chat runs no controller owns https://github.com/simstudioai/sim/pull/8457/files#top
Show all changes 7 commits https://github.com/simstudioai/sim/pull/8457/files
d78d860 fix(mothership): settle Chat runs no controller will finish waleedlatif1 Sep 30, 2026 https://github.com/simstudioai/sim/pull/8457/commits/d78d860690263b5d9a2921db5b519e52f4706806
bc9369c fix(mothership): lock chats before runs when settling orphans, and la… waleedlatif1 Sep 30, 2026 https://github.com/simstudioai/sim/pull/8457/commits/bc9369c634375ce2ec3fbcf05f40f06187218d5b
f1d878a fix(mothership): never sweep a current headless run waleedlatif1 Sep 30, 2026 https://github.com/simstudioai/sim/pull/8457/commits/f1d878a428cd721da5468e0a64010d21172fd627
fd4fc18 refactor(mothership): require the recorded Stop inside the stopped-ru… waleedlatif1 Sep 30, 2026 https://github.com/simstudioai/sim/pull/8457/commits/fd4fc18187074bfe6ba3e939753a665a3c5cf780
34768f4 fix(mothership): fence orphan settlement on the chat lock and resume … waleedlatif1 Sep 30, 2026 https://github.com/simstudioai/sim/pull/8457/commits/34768f4a32c42a29d0fac653cbee0598f8598c73
7c023cf fix(knowledge): record a terminal status for every Slack Assistant run waleedlatif1 Sep 30, 2026 https://github.com/simstudioai/sim/pull/8457/commits/7c023cf9a274b79a236f502146b5ef1be7e110f9
3ff5a3f fix(knowledge): record the Slack run's status after its turn is saved waleedlatif1 Sep 30, 2026 https://github.com/simstudioai/sim/pull/8457/commits/3ff5a3f8d5d70c747d7d5bc21ca0932ae09e6eda
Clear filters https://github.com/simstudioai/sim/pull/8457/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/8457/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/8457/files
route.ts https://github.com/simstudioai/sim/pull/8457/files#diff-3c5077959e4545be2034b5b99e76849a74f2d3bd0e41e546514214def8fe3563
assistant.integration.ts https://github.com/simstudioai/sim/pull/8457/files#diff-7a600e888e68d6a1334da19c6cea1bd09793a524eb24351b3b623e3a668d0218
assistant.ts https://github.com/simstudioai/sim/pull/8457/files#diff-6dae8277716c704eb50784b81fdc479303e9fb7da26d3dcd4758c01965c4b7c7
lifecycle.ts https://github.com/simstudioai/sim/pull/8457/files#diff-dbf57f6d994f4694017802dca8e2463eb2a5083cebe0a28bcde187ef0b5d954e
orphaned-runs.integration.ts https://github.com/simstudioai/sim/pull/8457/files#diff-ec44c1c6e5915437ecd7e01af1911844cdd02ebc730abd5c8c8fb34498fdbb07
orphaned-runs.ts https://github.com/simstudioai/sim/pull/8457/files#diff-384a5854871f26d3d52fc6cf89369ea8fe0ad64d1870bcbd403a08053f4e1482
repository.ts https://github.com/simstudioai/sim/pull/8457/files#diff-bdd330821217c2b7b7e64b969c999f2d2d57e7cb2f0cd376efcdc1bea63f95fa
controls.ts https://github.com/simstudioai/sim/pull/8457/files#diff-99343ce422b25af55834e446458d5d048b56514d2bf23fe923f4609015fe51bf
buffer.ts https://github.com/simstudioai/sim/pull/8457/files#diff-3523442e19f7ca7ea7476233be9cb95707b375b5cc5164fc889a592fedbbc44f
apps/sim/app/api/cron/cleanup-stale-executions/route.tshttps://github.com/simstudioai/sim/pull/8457/files#diff-3c5077959e4545be2034b5b99e76849a74f2d3bd0e41e546514214def8fe3563
View file https://github.com/simstudioai/sim/blob/3ff5a3f8d5d70c747d7d5bc21ca0932ae09e6eda/apps/sim/app/api/cron/cleanup-stale-executions/route.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/8457/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/8457/files#diff-3c5077959e4545be2034b5b99e76849a74f2d3bd0e41e546514214def8fe3563
https://github.com/simstudioai/sim/pull/8457/files#diff-3c5077959e4545be2034b5b99e76849a74f2d3bd0e41e546514214def8fe3563
https://github.com/simstudioai/sim/pull/8457/files#diff-3c5077959e4545be2034b5b99e76849a74f2d3bd0e41e546514214def8fe3563
https://github.com/simstudioai/sim/pull/8457/files#diff-3c5077959e4545be2034b5b99e76849a74f2d3bd0e41e546514214def8fe3563
https://github.com/simstudioai/sim/pull/8457/files#diff-3c5077959e4545be2034b5b99e76849a74f2d3bd0e41e546514214def8fe3563
https://github.com/simstudioai/sim/pull/8457/files#diff-3c5077959e4545be2034b5b99e76849a74f2d3bd0e41e546514214def8fe3563
apps/sim/lib/knowledge/application/slack-search/assistant.integration.tshttps://github.com/simstudioai/sim/pull/8457/files#diff-7a600e888e68d6a1334da19c6cea1bd09793a524eb24351b3b623e3a668d0218
View file https://github.com/simstudioai/sim/blob/3ff5a3f8d5d70c747d7d5bc21ca0932ae09e6eda/apps/sim/lib/knowledge/application/slack-search/assistant.integration.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/8457/{{ revealButtonHref }}
apps/sim/lib/knowledge/application/slack-search/assistant.tshttps://github.com/simstudioai/sim/pull/8457/files#diff-6dae8277716c704eb50784b81fdc479303e9fb7da26d3dcd4758c01965c4b7c7
View file https://github.com/simstudioai/sim/blob/3ff5a3f8d5d70c747d7d5bc21ca0932ae09e6eda/apps/sim/lib/knowledge/application/slack-search/assistant.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/8457/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/8457/files#diff-6dae8277716c704eb50784b81fdc479303e9fb7da26d3dcd4758c01965c4b7c7
https://github.com/simstudioai/sim/pull/8457/files#diff-6dae8277716c704eb50784b81fdc479303e9fb7da26d3dcd4758c01965c4b7c7
https://github.com/simstudioai/sim/pull/8457/files#diff-6dae8277716c704eb50784b81fdc479303e9fb7da26d3dcd4758c01965c4b7c7
https://github.com/simstudioai/sim/pull/8457/files#diff-6dae8277716c704eb50784b81fdc479303e9fb7da26d3dcd4758c01965c4b7c7
https://github.com/simstudioai/sim/pull/8457/files#diff-6dae8277716c704eb50784b81fdc479303e9fb7da26d3dcd4758c01965c4b7c7
https://github.com/simstudioai/sim/pull/8457/files#diff-6dae8277716c704eb50784b81fdc479303e9fb7da26d3dcd4758c01965c4b7c7
https://github.com/simstudioai/sim/pull/8457/files#diff-6dae8277716c704eb50784b81fdc479303e9fb7da26d3dcd4758c01965c4b7c7
https://github.com/simstudioai/sim/pull/8457/files#diff-6dae8277716c704eb50784b81fdc479303e9fb7da26d3dcd4758c01965c4b7c7
apps/sim/lib/mothership/async-runs/lifecycle.tshttps://github.com/simstudioai/sim/pull/8457/files#diff-dbf57f6d994f4694017802dca8e2463eb2a5083cebe0a28bcde187ef0b5d954e
View file https://github.com/simstudioai/sim/blob/3ff5a3f8d5d70c747d7d5bc21ca0932ae09e6eda/apps/sim/lib/mothership/async-runs/lifecycle.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/8457/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/8457/files#diff-dbf57f6d994f4694017802dca8e2463eb2a5083cebe0a28bcde187ef0b5d954e
https://github.com/simstudioai/sim/pull/8457/files#diff-dbf57f6d994f4694017802dca8e2463eb2a5083cebe0a28bcde187ef0b5d954e
Please reload this pagehttps://github.com/simstudioai/sim/pull/8457/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/8457/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.