Title: perf(outbox): resume each retention batch where the previous run's batch ended by waleedlatif1 · Pull Request #9446 · simstudioai/sim · GitHub
Open Graph Title: perf(outbox): resume each retention batch where the previous run's batch ended by waleedlatif1 · Pull Request #9446 · simstudioai/sim
X Title: perf(outbox): resume each retention batch where the previous run's batch ended by waleedlatif1 · Pull Request #9446 · simstudioai/sim
Description: Summary The completed-outbox prune deletes one oldest-first batch per type per run, and every batch started its index scan at the oldest row of its type. Rows deleted by earlier batches stay in outbox_event_type_created_idx until vacuum, so while a backlog drains each batch first walks everything deleted since the last vacuum (about 0.6s before the first live row on a read replica; a batch can reach the 10s statement timeout) Each batch now resumes after the (created_at, id) of the last row the previous run's batch deleted, the same resume shape as the idempotency sweep. Batches order by (created_at, id) so ties split exactly; the cursor keeps PostgreSQL's microsecond text form. The row comparison plans as a created_at >= index bound with the full comparison as a filter (plain EXPLAIN checked); resuming, the same batch took about 20ms The prune runs once per processor run, so the cursor outlives the run as a Redis hint (one MGET before the batches, one pipeline after). The cursor clears when a batch comes up short of the cutoff, and a pass older than an hour starts over from the oldest row, so a row the pass went by before it completed, or while an overlapping run held its lock, is still pruned by the next pass Without Redis, or with an unreadable cursor, a batch starts from the oldest row exactly as before. Batch size, statement timeout, cadence and eligibility (type list, completed, retention window) are unchanged Type of Change Performance improvement Testing lib/core/outbox/retention.integration.ts against real PostgreSQL and Redis: a row completed behind the cursor between runs is left for the next pass and then pruned (fails on the previous code, which takes it at once); one creation time split across batches with pending rows and non-prunable types mixed in deletes exactly the eligible rows; a pass over an hour old and an unreadable cursor both restart from the oldest row. Reverting the short-batch reset, the id tiebreak, or the pass age limit each turns its case red Existing retention cases still pass without Redis; lib/core/outbox unit tests bun run lint, bun run type-check (apps/sim), bun run check:audits (61 audits) Checklist Code follows project style guidelines Self-reviewed my changes Tests added/updated and passing (new tests pass the test-audit authoring gate) No new warnings introduced I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)
Open Graph Description: Summary The completed-outbox prune deletes one oldest-first batch per type per run, and every batch started its index scan at the oldest row of its type. Rows deleted by earlier batches stay in ou...
X Description: Summary The completed-outbox prune deletes one oldest-first batch per type per run, and every batch started its index scan at the oldest row of its type. Rows deleted by earlier batches stay in ou...
Opengraph URL: https://github.com/simstudioai/sim/pull/9446
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:535949a3-8f75-f8f5-f7db-68e73fa88127 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | D560:1ABF0E:E50DF:14D82B:6AFB1D3C |
| html-safe-nonce | 79a35008f45db2040daad78dc0db1bdbbd3f88ccdc75eb619ec7b55957e25f19 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJENTYwOjFBQkYwRTpFNTBERjoxNEQ4MkI6NkFGQjFEM0MiLCJ2aXNpdG9yX2lkIjoiMjI3OTg0NzM4NTQxNTc1NDg0IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | f8123e8453b85c4556f16e344558a5ed3427e5174514cdb303034fe00dabc911 |
| hovercard-subject-tag | pull_request:5073370221 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/9446/files |
| twitter:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| og:image:alt | Summary The completed-outbox prune deletes one oldest-first batch per type per run, and every batch started its index scan at the oldest row of its type. Rows deleted by earlier batches stay in ou... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 24933791aba0f10b2f189f8b953a1452dd79933933ac13a0a67b8ad3acee3d9f |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/simstudioai/sim git https://github.com/simstudioai/sim.git |
| octolytics-dimension-user_id | 199344406 |
| octolytics-dimension-user_login | simstudioai |
| octolytics-dimension-repository_id | 912559512 |
| octolytics-dimension-repository_nwo | simstudioai/sim |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 912559512 |
| octolytics-dimension-repository_network_root_nwo | simstudioai/sim |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 030711cc180b563614a434f4376acf7da5d445fa |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width