Title: fix(mothership): re-sync Chat reconnects from the worker log and trim the replay ring by bytes by waleedlatif1 · Pull Request #8469 · simstudioai/sim · GitHub
Open Graph Title: fix(mothership): re-sync Chat reconnects from the worker log and trim the replay ring by bytes by waleedlatif1 · Pull Request #8469 · simstudioai/sim
X Title: fix(mothership): re-sync Chat reconnects from the worker log and trim the replay ring by bytes by waleedlatif1 · Pull Request #8469 · simstudioai/sim
Description: Summary This PR stops the replay ring from ending long Chat turns. The bug it fixes: byte-budget refusals seen in production. A long run's replay ring reached its owner byte ceiling, and the refused write ended the turn. The byte-trimmed ring below fixes that. Enabling work: once the ring trims by bytes, it can lose its head, so reconnects past a trimmed head must be re-synced. The replay_gap re-sync is that enabling work. It is not a fix for a replay_gap observed in production. Merge order It builds on #8463 (stream-leg continuity), which is already in staging. It needs mothership#555's read-only replay endpoint (POST /api/streams/replay) deployed first. Its protocol types are synced here with bun run contracts:sync. Worker without the endpoint, or a key that may not call it: it answers 404, 401 or 403, and a reconnect keeps today's behaviour: the replay_gap terminal. Unreachable worker: the route answers 503, and the client retries with backoff. Byte-trimmed replay ring The ring now trims by bytes as well as by count. The owner counter tracks exactly the retained bytes, so a long run slides instead of hitting the owner budget and failing. Re-sync from the worker's log The ring can't serve a reconnect when: it lost its head (a fresh tab or reload from cursor 0 included); the cursor is behind it; its numbering restarted. In those cases Sim re-syncs the reader from the worker's durable log through the read-only replay. Response: frames carry this response's own cursors, starting at 1, and the header x-mothership-stream-replay: log. The client rebuilds the turn from an empty response. The reader stays on the log: from then on the client sends source=log, and the ring never serves that reader again. The ring and the log share no position to join on. Endings: A terminal or the worker's 5-minute cap ends the response at once. A park holds it while the run waits on Sim, and ends it as soon as the run resumes. Any held response ends as soon as the run reaches a terminal. A stall, a cut connection, or a park Sim has not yet marked holds it at least 10 s, so a reader replays the log at most every 10 s while nothing moves. Batch reads the ring can't serve return no ring events. Live tails: a live tail whose ring loses its head or restarts ends without a terminal event, so it re-attaches and is re-synced. replay_end frames are parsed with a schema tied to the protocol's reasons. An unknown reason ends the replay and is never forwarded as a run event. Recovery and snapshots on a ring that lost its head Recovery: a run whose Sim controller died (for example, across a deploy) recovers from an empty context, as it would from an expired ring. The new controller re-attaches with an empty receipt. The worker re-sends the whole response and re-hands its parked calls. The turn then finalizes with its full content persisted. Usage: usage stays with the worker's per-run settlement. A re-attach under the same message identity is never a second run. Snapshots: both chat snapshot routes skip the snapshot, through one shared reader, so the client re-syncs through the reconnect route. Cost A reconnect on a run whose ring lost its head replays up to the whole log. A tab that stays attached replays it again about every 5 minutes, when the worker's replay connection reaches its cap. It also replays at most every 10 s while a run is stalled. Sharing replays between readers is a worker follow-up. Known limits Worker deadline: the worker's run deadline still ends runs until its flag is turned off. Restarted rings in recovery and snapshots: these check only that the ring's first retained seq is 1. A ring that expired and restarted at 1 still looks complete to them. The heartbeat TTL refresh makes this rare. Duplicated text in the ring: after an empty-context recovery, the ring holds the re-sent response after its old tail. Readers never see it twice: a trimmed ring always re-syncs from the log, and live tails leave the ring when it loses its head. Inbox runs: Chat runs started from the inbox still run in a Trigger.dev task with a 90-minute maxDuration. Next step: one durable read path Key the ring by the worker's durable seq and serve every reader, live or late, from one read path. Cursors would then name positions in the durable log, and nothing would be re-synced or joined. Test plan Integration suites against real Postgres and Redis (bun run test:integration, all suites). The worker is mocked only at the HTTP boundary. replay-gap.integration.ts: re-sync from cursor 0, from behind the ring and from a restarted ring; source=log readers stay on the log; parked and stalled holds, and an unknown end reason; live tails leaving a restarted or headless ring; batch reads; the 404, 401 and 403 fallbacks. stream-recovery.integration.ts: a controller dies on a run whose ring lost its head. The new controller recovers with the real lifecycle. The turn persists its full content once, after one re-attach with an empty receipt. A replayed tool and a re-handed call the dead controller already ran each appear once. The re-handed call is resumed with its stored result, never run again. replay-budget.integration.ts: a stream far past the owner budget; recovery and snapshots on a ring that lost its head. buffer-ttl.integration.ts. Unit and DOM tests, including the client rebuild and source=log. bun run lint, bun run type-check, bun run check:audits, full apps/sim vitest.
Open Graph Description: Summary This PR stops the replay ring from ending long Chat turns. The bug it fixes: byte-budget refusals seen in production. A long run's replay ring reached its owner byte ceiling, and the r...
X Description: Summary This PR stops the replay ring from ending long Chat turns. The bug it fixes: byte-budget refusals seen in production. A long run's replay ring reached its owner byte ceiling, and t...
Opengraph URL: https://github.com/simstudioai/sim/pull/8469
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:ed89cb1c-b1ac-1619-ed50-8a5a0c302508 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | E71E:1743E0:3C1BBD:523157:6AFB9973 |
| html-safe-nonce | 7f579a57baee703cca96aa2c6f639465679a44e059debe4056d88ba70fb937e0 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJFNzFFOjE3NDNFMDozQzFCQkQ6NTIzMTU3OjZBRkI5OTczIiwidmlzaXRvcl9pZCI6IjE4Mzg3ODkxMzMxNjIyMjM5ODciLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | ca23d3c6a8df17d666b5490eb79a426ea2e840d98849f2273dea8f4d6b42de75 |
| hovercard-subject-tag | pull_request:4693432919 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/8469/files |
| twitter:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| og:image:alt | Summary This PR stops the replay ring from ending long Chat turns. The bug it fixes: byte-budget refusals seen in production. A long run's replay ring reached its owner byte ceiling, and the r... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 3cefd19a936819d82cfb5279d50fb56fe79cb9ec30f0f807e04e89b0c58ffae1 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/simstudioai/sim git https://github.com/simstudioai/sim.git |
| octolytics-dimension-user_id | 199344406 |
| octolytics-dimension-user_login | simstudioai |
| octolytics-dimension-repository_id | 912559512 |
| octolytics-dimension-repository_nwo | simstudioai/sim |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 912559512 |
| octolytics-dimension-repository_network_root_nwo | simstudioai/sim |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 606800bdb1b81d125fc845bdcc262c9fe063fc8f |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width