René's URL Explorer Experiment


Title: perf(knowledge): reconcile a completed listing in one pass over its windows by waleedlatif1 · Pull Request #9449 · simstudioai/sim · GitHub

Open Graph Title: perf(knowledge): reconcile a completed listing in one pass over its windows by waleedlatif1 · Pull Request #9449 · simstudioai/sim

X Title: perf(knowledge): reconcile a completed listing in one pass over its windows by waleedlatif1 · Pull Request #9449 · simstudioai/sim

Description: Summary A completed listing's reconciliation read every window of the connector to count, then the ACL, soft-delete and hard-delete walks each read the same windows again: up to four passes over a connector whenever any removal was due, the count pass cold and the walks warm right after it tallyReconciliationWindows replaces countReconciliationWindows: the same one-statement-per-window keyset walk over doc_connector_reconciliation_v2_idx now also returns, per condition, the ids (and tombstone flag) of the matching rows, each condition evaluated once per row visitReconciliationMatches pages those retained rows to the existing removal callbacks, so a completed listing reads each window once; the walks stay only as the fallback for a condition matching more than RECONCILIATION_RETAINED_MATCHES (10,000), which keeps memory and each window's result bounded Every decision is unchanged: the same four counts feed the suspect-listing guard, the per-generation cap and its hold notice, ACL revocation still runs when removals are held, and every removal still re-checks its target in its own statement (soft and ACL writes re-check absence, hard deletes keep the connector/exclusion/archive guard). The rows acted on are now exactly the rows counted, rather than whatever a later walk found Prod (plain EXPLAIN and one replica EXPLAIN ANALYZE of a full window): the tally keeps the old plan, a 5,000-step recursive Limit 1 probe of doc_connector_reconciliation_v2_idx with no other document access, plus an in-memory sort of the window. A cold window is I/O-bound (~1.3s, nearly all of it reads), a warm one ~47ms Effect: a completed listing with removals due issues one statement per window instead of two to four. Over the last day of query insights, the soft- and hard-delete walk windows this removes accounted for ~60% of the rows reconciliation windows read and ~10% of their DB time. The cold count pass itself remains, since it has to read every owned row's seen stamp Not changed: window size, the index, source_seen_at stays out of every index (HOT updates), member-resurrection walks, statement timeouts Type of Change Other: performance Testing sync-content-pass.integration.ts (real Postgres) gains a suite for the removal decisions: under, at and over the cap for soft, hard and both together (exact hold notices, ACLs revoked either way), full sync past the cap, an empty connector, a connector of three windows (exact soft/hard/ACL sets, 3 tally statements and no walks), more than 10,000 removals (walk fallback, each purged exactly once in pages of 25), a user excluding a tallied document mid-pass (skipped) and one in a later window (never counted), and a deadline cut mid-removal resumed by the next pass (each removed once) The same suite run against the pre-change code: every decision test passes unchanged; only the statement-count assertion fails (3 tallies + 9 walk statements before, 3 + 0 now) Mutation checks: acting on truncated rows past the bound, keeping only the first window's rows, and an off-by-one bound each turn a test red listing-continuation.integration.ts reconciliation-window tests now assert no walks and keep the EXPLAIN ANALYZE bound (every tally statement reads at most one window) on the real schema Unit: reconciliation-window.test.ts (sums and retains across windows, bound edge), plus the updated sync-content-pass, sync-engine and member-sync-engine tests; vitest run lib/knowledge (1,793 tests) bun run lint, bun run type-check, bun run check:audits, check:test-types, block-registry check Checklist Code follows project style guidelines Self-reviewed my changes Tests added/updated and passing (new tests pass the test-audit authoring gate) No new warnings introduced I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

Open Graph Description: Summary A completed listing's reconciliation read every window of the connector to count, then the ACL, soft-delete and hard-delete walks each read the same windows again: up to four passes ov...

X Description: Summary A completed listing's reconciliation read every window of the connector to count, then the ACL, soft-delete and hard-delete walks each read the same windows again: up to four passe...

Opengraph URL: https://github.com/simstudioai/sim/pull/9449

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:7b7d5929-c24e-50bf-a8cf-eb0235a612f5
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-id8004:25EFCC:8E425:CF1C1:6AFB192F
html-safe-nonce3545ce2e669542717ae9fe96f144a827bbafefede20790563e3c9fa538a1db46
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI4MDA0OjI1RUZDQzo4RTQyNTpDRjFDMTo2QUZCMTkyRiIsInZpc2l0b3JfaWQiOiI1NzYxMTM4Mjg2NDcwNTY4MjM5IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0=
visitor-hmaca1b6f0af13bfaa5421f257f80ac99c925c3485b4a6b32607e4beb3089dc9edc7
hovercard-subject-tagpull_request:5073525764
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/9449/files
twitter:imagehttps://avatars.githubusercontent.com/u/40672544?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/40672544?s=400&v=4
og:image:altSummary A completed listing's reconciliation read every window of the connector to count, then the ACL, soft-delete and hard-delete walks each read the same windows again: up to four passes ov...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None24933791aba0f10b2f189f8b953a1452dd79933933ac13a0a67b8ad3acee3d9f
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/simstudioai/sim git https://github.com/simstudioai/sim.git
octolytics-dimension-user_id199344406
octolytics-dimension-user_loginsimstudioai
octolytics-dimension-repository_id912559512
octolytics-dimension-repository_nwosimstudioai/sim
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id912559512
octolytics-dimension-repository_network_root_nwosimstudioai/sim
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release030711cc180b563614a434f4376acf7da5d445fa
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/simstudioai/sim/pull/9449/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F9449%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F9449%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=simstudioai%2Fsim
Reloadhttps://github.com/simstudioai/sim/pull/9449/files
Reloadhttps://github.com/simstudioai/sim/pull/9449/files
Reloadhttps://github.com/simstudioai/sim/pull/9449/files
simstudioaihttps://github.com/simstudioai
simhttps://github.com/simstudioai/sim
Codehttps://github.com/simstudioai/sim
Issues129 (129)https://github.com/simstudioai/sim/issues
Pull requests198 (198)https://github.com/simstudioai/sim/pulls
Actionshttps://github.com/simstudioai/sim/actions
Projectshttps://github.com/simstudioai/sim/projects
Security and qualityhttps://github.com/simstudioai/sim/security
Insightshttps://github.com/simstudioai/sim/pulse
Sign up for GitHub https://github.com/signup?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
waleedlatif1https://github.com/waleedlatif1
staginghttps://github.com/simstudioai/sim/tree/staging
perf/knowledge-reconciliation-single-passhttps://github.com/simstudioai/sim/tree/perf/knowledge-reconciliation-single-pass
Conversation 2 https://github.com/simstudioai/sim/pull/9449
Commits 1 https://github.com/simstudioai/sim/pull/9449/commits
Checks 67 https://github.com/simstudioai/sim/pull/9449/checks
Files changed https://github.com/simstudioai/sim/pull/9449/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/9449/files
perf(knowledge): reconcile a completed listing in one pass over its windows https://github.com/simstudioai/sim/pull/9449/files#top
Show all changes 1 commit https://github.com/simstudioai/sim/pull/9449/files
e0c5fa9 perf(knowledge): reconcile a completed listing in one pass over its w… waleedlatif1 Oct 29, 2026 https://github.com/simstudioai/sim/pull/9449/commits/e0c5fa9b0e3eb01e261d2955e8775fe069218f99
Clear filters https://github.com/simstudioai/sim/pull/9449/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/9449/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/9449/files
listing-continuation.integration.ts https://github.com/simstudioai/sim/pull/9449/files#diff-e0cfe9e757a63c3c668fa797a9a26e31db2e1773cbbedf4f4ee7066714447105
member-sync-engine-content-credential.test.ts https://github.com/simstudioai/sim/pull/9449/files#diff-40f1876e4f1f511d3de4d450ad9f97ffb6d2b500a4017fab27b899af27484d57
reconciliation-window.test-helpers.ts https://github.com/simstudioai/sim/pull/9449/files#diff-282d1098b5c70b8070973c142c45337e07a924f7968c290f485228dba6fbbf8c
reconciliation-window.test.ts https://github.com/simstudioai/sim/pull/9449/files#diff-9ab666c3026278a7667792073aadc9403cf860f94e4c70eb95fd67e3b348d45d
reconciliation-window.ts https://github.com/simstudioai/sim/pull/9449/files#diff-a397298150e20079f419649d2cd5491b535ffde39ab744149eea033f5e8455e4
sync-content-pass.integration.ts https://github.com/simstudioai/sim/pull/9449/files#diff-fa75e5999c2488ccdf3c7523b17b435e7ee27fea3954a774ff1e37ebe2b30341
sync-content-pass.test.ts https://github.com/simstudioai/sim/pull/9449/files#diff-8d35d55800e2821b9e7cd19c76e2db80b43c20e2e53ab4485115f2abfd24d5d7
sync-content-pass.ts https://github.com/simstudioai/sim/pull/9449/files#diff-b6bd8cd1ab36d6b8e1b29f63847ffc4e4ee912f70e19de773896d753cc73d866
sync-engine.test.ts https://github.com/simstudioai/sim/pull/9449/files#diff-021afebf7e100833ecdc940e307263436964ce85ef3dfd40fc90307a1d115534
apps/sim/lib/knowledge/__integration__/listing-continuation.integration.tshttps://github.com/simstudioai/sim/pull/9449/files#diff-e0cfe9e757a63c3c668fa797a9a26e31db2e1773cbbedf4f4ee7066714447105
View file https://github.com/simstudioai/sim/blob/e0c5fa9b0e3eb01e261d2955e8775fe069218f99/apps/sim/lib/knowledge/__integration__/listing-continuation.integration.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/9449/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/9449/files#diff-e0cfe9e757a63c3c668fa797a9a26e31db2e1773cbbedf4f4ee7066714447105
https://github.com/simstudioai/sim/pull/9449/files#diff-e0cfe9e757a63c3c668fa797a9a26e31db2e1773cbbedf4f4ee7066714447105
https://github.com/simstudioai/sim/pull/9449/files#diff-e0cfe9e757a63c3c668fa797a9a26e31db2e1773cbbedf4f4ee7066714447105
https://github.com/simstudioai/sim/pull/9449/files#diff-e0cfe9e757a63c3c668fa797a9a26e31db2e1773cbbedf4f4ee7066714447105
https://github.com/simstudioai/sim/pull/9449/files#diff-e0cfe9e757a63c3c668fa797a9a26e31db2e1773cbbedf4f4ee7066714447105
https://github.com/simstudioai/sim/pull/9449/files#diff-e0cfe9e757a63c3c668fa797a9a26e31db2e1773cbbedf4f4ee7066714447105
https://github.com/simstudioai/sim/pull/9449/files#diff-e0cfe9e757a63c3c668fa797a9a26e31db2e1773cbbedf4f4ee7066714447105
https://github.com/simstudioai/sim/pull/9449/files#diff-e0cfe9e757a63c3c668fa797a9a26e31db2e1773cbbedf4f4ee7066714447105
https://github.com/simstudioai/sim/pull/9449/files#diff-e0cfe9e757a63c3c668fa797a9a26e31db2e1773cbbedf4f4ee7066714447105
https://github.com/simstudioai/sim/pull/9449/files#diff-e0cfe9e757a63c3c668fa797a9a26e31db2e1773cbbedf4f4ee7066714447105
apps/sim/lib/knowledge/connectors/member-sync-engine-content-credential.test.tshttps://github.com/simstudioai/sim/pull/9449/files#diff-40f1876e4f1f511d3de4d450ad9f97ffb6d2b500a4017fab27b899af27484d57
View file https://github.com/simstudioai/sim/blob/e0c5fa9b0e3eb01e261d2955e8775fe069218f99/apps/sim/lib/knowledge/connectors/member-sync-engine-content-credential.test.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/9449/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/9449/files#diff-40f1876e4f1f511d3de4d450ad9f97ffb6d2b500a4017fab27b899af27484d57
https://github.com/simstudioai/sim/pull/9449/files#diff-40f1876e4f1f511d3de4d450ad9f97ffb6d2b500a4017fab27b899af27484d57
https://github.com/simstudioai/sim/pull/9449/files#diff-40f1876e4f1f511d3de4d450ad9f97ffb6d2b500a4017fab27b899af27484d57
https://github.com/simstudioai/sim/pull/9449/files#diff-40f1876e4f1f511d3de4d450ad9f97ffb6d2b500a4017fab27b899af27484d57
https://github.com/simstudioai/sim/pull/9449/files#diff-40f1876e4f1f511d3de4d450ad9f97ffb6d2b500a4017fab27b899af27484d57
https://github.com/simstudioai/sim/pull/9449/files#diff-40f1876e4f1f511d3de4d450ad9f97ffb6d2b500a4017fab27b899af27484d57
https://github.com/simstudioai/sim/pull/9449/files#diff-40f1876e4f1f511d3de4d450ad9f97ffb6d2b500a4017fab27b899af27484d57
https://github.com/simstudioai/sim/pull/9449/files#diff-40f1876e4f1f511d3de4d450ad9f97ffb6d2b500a4017fab27b899af27484d57
apps/sim/lib/knowledge/connectors/reconciliation-window.test-helpers.tshttps://github.com/simstudioai/sim/pull/9449/files#diff-282d1098b5c70b8070973c142c45337e07a924f7968c290f485228dba6fbbf8c
View file https://github.com/simstudioai/sim/blob/e0c5fa9b0e3eb01e261d2955e8775fe069218f99/apps/sim/lib/knowledge/connectors/reconciliation-window.test-helpers.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/9449/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/9449/files#diff-282d1098b5c70b8070973c142c45337e07a924f7968c290f485228dba6fbbf8c
apps/sim/lib/knowledge/connectors/reconciliation-window.test.tshttps://github.com/simstudioai/sim/pull/9449/files#diff-9ab666c3026278a7667792073aadc9403cf860f94e4c70eb95fd67e3b348d45d
View file https://github.com/simstudioai/sim/blob/e0c5fa9b0e3eb01e261d2955e8775fe069218f99/apps/sim/lib/knowledge/connectors/reconciliation-window.test.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/9449/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/9449/files#diff-9ab666c3026278a7667792073aadc9403cf860f94e4c70eb95fd67e3b348d45d
Please reload this pagehttps://github.com/simstudioai/sim/pull/9449/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.