Title: perf(knowledge): reconcile a completed listing in one pass over its windows by waleedlatif1 · Pull Request #9449 · simstudioai/sim · GitHub
Open Graph Title: perf(knowledge): reconcile a completed listing in one pass over its windows by waleedlatif1 · Pull Request #9449 · simstudioai/sim
X Title: perf(knowledge): reconcile a completed listing in one pass over its windows by waleedlatif1 · Pull Request #9449 · simstudioai/sim
Description: Summary A completed listing's reconciliation read every window of the connector to count, then the ACL, soft-delete and hard-delete walks each read the same windows again: up to four passes over a connector whenever any removal was due, the count pass cold and the walks warm right after it tallyReconciliationWindows replaces countReconciliationWindows: the same one-statement-per-window keyset walk over doc_connector_reconciliation_v2_idx now also returns, per condition, the ids (and tombstone flag) of the matching rows, each condition evaluated once per row visitReconciliationMatches pages those retained rows to the existing removal callbacks, so a completed listing reads each window once; the walks stay only as the fallback for a condition matching more than RECONCILIATION_RETAINED_MATCHES (10,000), which keeps memory and each window's result bounded Every decision is unchanged: the same four counts feed the suspect-listing guard, the per-generation cap and its hold notice, ACL revocation still runs when removals are held, and every removal still re-checks its target in its own statement (soft and ACL writes re-check absence, hard deletes keep the connector/exclusion/archive guard). The rows acted on are now exactly the rows counted, rather than whatever a later walk found Prod (plain EXPLAIN and one replica EXPLAIN ANALYZE of a full window): the tally keeps the old plan, a 5,000-step recursive Limit 1 probe of doc_connector_reconciliation_v2_idx with no other document access, plus an in-memory sort of the window. A cold window is I/O-bound (~1.3s, nearly all of it reads), a warm one ~47ms Effect: a completed listing with removals due issues one statement per window instead of two to four. Over the last day of query insights, the soft- and hard-delete walk windows this removes accounted for ~60% of the rows reconciliation windows read and ~10% of their DB time. The cold count pass itself remains, since it has to read every owned row's seen stamp Not changed: window size, the index, source_seen_at stays out of every index (HOT updates), member-resurrection walks, statement timeouts Type of Change Other: performance Testing sync-content-pass.integration.ts (real Postgres) gains a suite for the removal decisions: under, at and over the cap for soft, hard and both together (exact hold notices, ACLs revoked either way), full sync past the cap, an empty connector, a connector of three windows (exact soft/hard/ACL sets, 3 tally statements and no walks), more than 10,000 removals (walk fallback, each purged exactly once in pages of 25), a user excluding a tallied document mid-pass (skipped) and one in a later window (never counted), and a deadline cut mid-removal resumed by the next pass (each removed once) The same suite run against the pre-change code: every decision test passes unchanged; only the statement-count assertion fails (3 tallies + 9 walk statements before, 3 + 0 now) Mutation checks: acting on truncated rows past the bound, keeping only the first window's rows, and an off-by-one bound each turn a test red listing-continuation.integration.ts reconciliation-window tests now assert no walks and keep the EXPLAIN ANALYZE bound (every tally statement reads at most one window) on the real schema Unit: reconciliation-window.test.ts (sums and retains across windows, bound edge), plus the updated sync-content-pass, sync-engine and member-sync-engine tests; vitest run lib/knowledge (1,793 tests) bun run lint, bun run type-check, bun run check:audits, check:test-types, block-registry check Checklist Code follows project style guidelines Self-reviewed my changes Tests added/updated and passing (new tests pass the test-audit authoring gate) No new warnings introduced I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)
Open Graph Description: Summary A completed listing's reconciliation read every window of the connector to count, then the ACL, soft-delete and hard-delete walks each read the same windows again: up to four passes ov...
X Description: Summary A completed listing's reconciliation read every window of the connector to count, then the ACL, soft-delete and hard-delete walks each read the same windows again: up to four passe...
Opengraph URL: https://github.com/simstudioai/sim/pull/9449
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:7b7d5929-c24e-50bf-a8cf-eb0235a612f5 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | 8004:25EFCC:8E425:CF1C1:6AFB192F |
| html-safe-nonce | 3545ce2e669542717ae9fe96f144a827bbafefede20790563e3c9fa538a1db46 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI4MDA0OjI1RUZDQzo4RTQyNTpDRjFDMTo2QUZCMTkyRiIsInZpc2l0b3JfaWQiOiI1NzYxMTM4Mjg2NDcwNTY4MjM5IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | a1b6f0af13bfaa5421f257f80ac99c925c3485b4a6b32607e4beb3089dc9edc7 |
| hovercard-subject-tag | pull_request:5073525764 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/9449/files |
| twitter:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| og:image:alt | Summary A completed listing's reconciliation read every window of the connector to count, then the ACL, soft-delete and hard-delete walks each read the same windows again: up to four passes ov... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 24933791aba0f10b2f189f8b953a1452dd79933933ac13a0a67b8ad3acee3d9f |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/simstudioai/sim git https://github.com/simstudioai/sim.git |
| octolytics-dimension-user_id | 199344406 |
| octolytics-dimension-user_login | simstudioai |
| octolytics-dimension-repository_id | 912559512 |
| octolytics-dimension-repository_nwo | simstudioai/sim |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 912559512 |
| octolytics-dimension-repository_network_root_nwo | simstudioai/sim |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 030711cc180b563614a434f4376acf7da5d445fa |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width