René's URL Explorer Experiment


Title: fix(knowledge): run the projection source/ACL backfill in the background instead of the deploy by waleedlatif1 · Pull Request #8058 · simstudioai/sim · GitHub

Open Graph Title: fix(knowledge): run the projection source/ACL backfill in the background instead of the deploy by waleedlatif1 · Pull Request #8058 · simstudioai/sim

X Title: fix(knowledge): run the projection source/ACL backfill in the background instead of the deploy by waleedlatif1 · Pull Request #8058 · simstudioai/sim

Description: Summary Script migration 0021_embedding_search_connector backfilled the ranking projections' connector_id and acl columns synchronously, 500 rows per page under a 60s statement timeout. On embedding_search every filled row is re-inserted into each HNSW index, so a page's cost is index maintenance rather than the plan: one 500-row page ran ~68s cold (~130 ms per row), the first page hit the timeout (57014), and the migration and the deploy failed with it. No page size or timeout makes a synchronous backfill of an HNSW-indexed projection fit inside a deploy job. Search no longer depends on the backfill: projectionCandidateAccessCondition decides a row whose acl IS NULL on its document — acl IS NULL AND EXISTS (SELECT 1 FROM document WHERE document.id = .document_id AND ) — the per-candidate join every row paid before the columns existed, so bounded candidate pools (vector walk, exact slice, Tin window) hold only rows hydration will keep. A filled row is still decided on its mirrored columns. knowledgeCandidateAccessConditionForConnectors at hydration is unchanged and remains the security gate. Covers both the vector walk and the Tin keyword window, which share the function. The registered migration is now 0022_projection_source_acl_backfill, which supersedes 0021 (the shape 0016 uses over 0015): it installs the triggers and builds the indexes only (CREATE INDEX CONCURRENTLY IF NOT EXISTS, on one reserved connection so the session-scoped lock timeout covers every build; plus a partial index on (id) WHERE acl IS NULL per projection so the unfilled branch stays an index probe beside the ACL GIN and the backfill's keyset pages read from it). Both are idempotent, so the re-run on the next deploy completes quickly, and a database that already recorded 0021 still gains the unfilled indexes. The 0021 file keeps the shared helpers and its direct db:push entry point, which still backfills inline. backfillProjectionSourceAcl stays exported and is now resumable: keyset pages of 100 unfilled rows, each its own transaction with a per-page statement timeout (exported as PROJECTION_SOURCE_ACL_PAGE_TIMEOUT_MS), one statement per page with scalar binds only, a pause between pages (sleep), a time budget that returns the cursor, progress logging, and an ANALYZE on completion. New projection-source-acl-backfill Trigger.dev task (apps/sim/background/), modeled on table-backfill: one run at a time, fills for up to an hour, then triggers its continuation from the cursor until both projections are filled. Safe to start repeatedly — a run only fills rows still unset. Started after the deploy with bun apps/sim/scripts/backfill-projection-source-acl.ts, which enqueues the task the way the table backfill is enqueued (tasks.trigger + region when a Trigger worker is configured, runDetached of the same runner in-process otherwise). Documented in the migration's TSDoc as the operator path. Type of Change Bug fix Testing predicate.test.ts: renders the on-row predicate with the document branch for unfilled rows; predicate.postgres.test.ts (real database): an unfilled chunk of a document the caller cannot read is refused on the row, unfilled chunks of readable documents are admitted; queries.test.ts: the Tin window carries the branch. Each fails with the predicate change reverted. New 0021_embedding_search_connector.postgres.test.ts: up re-runs cleanly and builds the unfilled indexes; the backfill fills only unset rows in pages, skips a chunk whose document changed, stops at its budget with a cursor and resumes after it; a rerun writes nothing. The index and budget tests fail with those changes reverted. New projection-source-acl-backfill.test.ts: projection order, cursor resume, budget return, connection close on failure, Trigger vs detached dispatch. packages/db registry and push tests, bun run type-check in apps/sim and packages/db, bun run lint, bun run check:audits (47 audits), docs-manifest:check, block-registry check: all pass. Checklist Code follows project style guidelines Self-reviewed my changes Tests added/updated and passing No new warnings introduced I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA) https://claude.ai/code/session_01XU6c7pKRpa5CMoMHKDdqxX

Open Graph Description: Summary Script migration 0021_embedding_search_connector backfilled the ranking projections' connector_id and acl columns synchronously, 500 rows per page under a 60s statement timeout. On emb...

X Description: Summary Script migration 0021_embedding_search_connector backfilled the ranking projections' connector_id and acl columns synchronously, 500 rows per page under a 60s statement timeout. On...

Opengraph URL: https://github.com/simstudioai/sim/pull/8058

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:656328fb-bff0-5ec2-75f4-d86917380a7e
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idA926:219171:C63D06:1178EC4:6AFB5DB9
html-safe-noncea99d94de0dd4b4a45e38ef3447d46e87eb5cb8d1fcc63fb8e2c5241867b90374
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBOTI2OjIxOTE3MTpDNjNEMDY6MTE3OEVDNDo2QUZCNURCOSIsInZpc2l0b3JfaWQiOiI1NzAyMjYyMTYxODEyNDQ2NjQ5IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0=
visitor-hmac42124de24ac55f667936d233640c1dabd0e92be03b00abaa380eae724c5d33f1
hovercard-subject-tagpull_request:4585100326
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/8058/files
twitter:imagehttps://avatars.githubusercontent.com/u/40672544?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/40672544?s=400&v=4
og:image:altSummary Script migration 0021_embedding_search_connector backfilled the ranking projections' connector_id and acl columns synchronously, 500 rows per page under a 60s statement timeout. On emb...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
Noneea33d13c96d0ff66bd8fa906b193d2213536b78cb06b0c50a4632c58b1418f94
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/simstudioai/sim git https://github.com/simstudioai/sim.git
octolytics-dimension-user_id199344406
octolytics-dimension-user_loginsimstudioai
octolytics-dimension-repository_id912559512
octolytics-dimension-repository_nwosimstudioai/sim
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id912559512
octolytics-dimension-repository_network_root_nwosimstudioai/sim
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release59ef97164c52cf59bd10558ddec9c4f482763988
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/simstudioai/sim/pull/8058/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F8058%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F8058%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=simstudioai%2Fsim
Reloadhttps://github.com/simstudioai/sim/pull/8058/files
Reloadhttps://github.com/simstudioai/sim/pull/8058/files
Reloadhttps://github.com/simstudioai/sim/pull/8058/files
simstudioaihttps://github.com/simstudioai
simhttps://github.com/simstudioai/sim
Codehttps://github.com/simstudioai/sim
Issues129 (129)https://github.com/simstudioai/sim/issues
Pull requests197 (197)https://github.com/simstudioai/sim/pulls
Actionshttps://github.com/simstudioai/sim/actions
Projectshttps://github.com/simstudioai/sim/projects
Security and qualityhttps://github.com/simstudioai/sim/security
Insightshttps://github.com/simstudioai/sim/pulse
Sign up for GitHub https://github.com/signup?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
waleedlatif1https://github.com/waleedlatif1
staginghttps://github.com/simstudioai/sim/tree/staging
fix/projection-acl-backfill-asynchttps://github.com/simstudioai/sim/tree/fix/projection-acl-backfill-async
Conversation 42 https://github.com/simstudioai/sim/pull/8058
Commits 6 https://github.com/simstudioai/sim/pull/8058/commits
Checks 33 https://github.com/simstudioai/sim/pull/8058/checks
Files changed https://github.com/simstudioai/sim/pull/8058/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/8058/files
fix(knowledge): run the projection source/ACL backfill in the background instead of the deploy https://github.com/simstudioai/sim/pull/8058/files#top
Show all changes 6 commits https://github.com/simstudioai/sim/pull/8058/files
1dddb8b fix(knowledge): run the projection source/ACL backfill in the backgro… waleedlatif1 Sep 20, 2026 https://github.com/simstudioai/sim/pull/8058/commits/1dddb8bd662f0e3e82c8873aad71f3e2027b9e39
8a64dd1 fix(knowledge): decide unfilled projection rows on their document and… waleedlatif1 Sep 20, 2026 https://github.com/simstudioai/sim/pull/8058/commits/8a64dd1cb83041963f4f7b4532490093ab079798
ecbda39 fix(knowledge): keep the outbox event pending until a deployment with… waleedlatif1 Sep 20, 2026 https://github.com/simstudioai/sim/pull/8058/commits/ecbda392048c5e43920751f33a409f37f1bcda46
526f659 fix(knowledge): start the projection backfill the way the table backf… waleedlatif1 Sep 20, 2026 https://github.com/simstudioai/sim/pull/8058/commits/526f659f2d55299b7ba3884b7f3a83820cd07df5
9435d51 fix(knowledge): refuse a projection backfill page size that is not a … waleedlatif1 Sep 20, 2026 https://github.com/simstudioai/sim/pull/8058/commits/9435d510fdd65fa8cec044626709b534bedfec02
ac644c0 fix(knowledge): check the projection backfill budget after the pause … waleedlatif1 Sep 20, 2026 https://github.com/simstudioai/sim/pull/8058/commits/ac644c06666acf4d53e494816d54284d87f29e33
Clear filters https://github.com/simstudioai/sim/pull/8058/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/8058/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/8058/files
projection-source-acl-backfill.ts https://github.com/simstudioai/sim/pull/8058/files#diff-509276fe69dcb1c189a64523121a52900d803c98e4f2a68eed016d7775a496af
predicate.postgres.test.ts https://github.com/simstudioai/sim/pull/8058/files#diff-0594c8e9ccb3d01589226b3540ac725c18ddafe355b80b2472f9905586c1161e
predicate.test.ts https://github.com/simstudioai/sim/pull/8058/files#diff-cbeeaa5cc835fb1fedc629c869b2a5f952eb1c7343e44960b6a8dfe126660ff1
predicate.ts https://github.com/simstudioai/sim/pull/8058/files#diff-720ce8ebbff1c232b2a35c2b75ac2bb1983110156c6d866f4127172e619c4df3
projection-source-acl-backfill.test.ts https://github.com/simstudioai/sim/pull/8058/files#diff-bef812b85dc282fb8c3741a0125810e9aa01bfee59c41d360c08343edecf7caa
projection-source-acl-backfill.ts https://github.com/simstudioai/sim/pull/8058/files#diff-7ac6330ea0a7cf0177d32cf3e57641976a88161cf7eaef1e38bad30fbe0c31ca
queries.test.ts https://github.com/simstudioai/sim/pull/8058/files#diff-b5bc7cdb4cf5dd93b0dc5beb76436fb9e70fc036e89833eaf8d9defb8bc83c3b
queries.ts https://github.com/simstudioai/sim/pull/8058/files#diff-5f10054baeef2f06350b32a6520d8f841f86a2fc908435296dd8dcb097497ba9
backfill-projection-source-acl.ts https://github.com/simstudioai/sim/pull/8058/files#diff-a1a8d5bdd73171ab6c7b8e66b28b70139ed93463a861c7cbce06ca6f4e66b2af
script-migrations-paused-billing-attribution.test.ts https://github.com/simstudioai/sim/pull/8058/files#diff-509458c3951fa9cd2e97a7942fd977a83ef520c679d29dc242a5b2cacfb02743
0016_backfill_search_vectors.postgres.test.ts https://github.com/simstudioai/sim/pull/8058/files#diff-356dbbd8454d0f54e3461aeedfa2b0bec37bfa5930f8272878b0455ab14d6711
0021_embedding_search_connector.postgres.test.ts https://github.com/simstudioai/sim/pull/8058/files#diff-ec4e0f10a764b6d454d1944e270af3ba066294b238c60f388301dab9ee93b538
0021_embedding_search_connector.test.ts https://github.com/simstudioai/sim/pull/8058/files#diff-7030952f8a72f4a67628574075fa8162bd41bf9481491a8c4282d0b4569a77b6
0021_embedding_search_connector.ts https://github.com/simstudioai/sim/pull/8058/files#diff-053cef46b422b9c5acf830b1c477ac221deb525da87b191e907a27c482525674
0022_projection_source_acl_backfill.ts https://github.com/simstudioai/sim/pull/8058/files#diff-494c46c703abe040199f2860d79226c70aaf38211df221737d945da96409752d
index.ts https://github.com/simstudioai/sim/pull/8058/files#diff-4c8058a0b924ce061c5ce010c1308148ea03720952dda10d03cee092b61adbd6
apps/sim/background/projection-source-acl-backfill.tshttps://github.com/simstudioai/sim/pull/8058/files#diff-509276fe69dcb1c189a64523121a52900d803c98e4f2a68eed016d7775a496af
View file https://github.com/simstudioai/sim/blob/ac644c06666acf4d53e494816d54284d87f29e33/apps/sim/background/projection-source-acl-backfill.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/8058/{{ revealButtonHref }}
apps/sim/lib/knowledge/access/predicate.postgres.test.tshttps://github.com/simstudioai/sim/pull/8058/files#diff-0594c8e9ccb3d01589226b3540ac725c18ddafe355b80b2472f9905586c1161e
View file https://github.com/simstudioai/sim/blob/ac644c06666acf4d53e494816d54284d87f29e33/apps/sim/lib/knowledge/access/predicate.postgres.test.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/8058/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/8058/files#diff-0594c8e9ccb3d01589226b3540ac725c18ddafe355b80b2472f9905586c1161e
https://github.com/simstudioai/sim/pull/8058/files#diff-0594c8e9ccb3d01589226b3540ac725c18ddafe355b80b2472f9905586c1161e
apps/sim/lib/knowledge/access/predicate.test.tshttps://github.com/simstudioai/sim/pull/8058/files#diff-cbeeaa5cc835fb1fedc629c869b2a5f952eb1c7343e44960b6a8dfe126660ff1
View file https://github.com/simstudioai/sim/blob/ac644c06666acf4d53e494816d54284d87f29e33/apps/sim/lib/knowledge/access/predicate.test.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/8058/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/8058/files#diff-cbeeaa5cc835fb1fedc629c869b2a5f952eb1c7343e44960b6a8dfe126660ff1
https://github.com/simstudioai/sim/pull/8058/files#diff-cbeeaa5cc835fb1fedc629c869b2a5f952eb1c7343e44960b6a8dfe126660ff1
apps/sim/lib/knowledge/access/predicate.tshttps://github.com/simstudioai/sim/pull/8058/files#diff-720ce8ebbff1c232b2a35c2b75ac2bb1983110156c6d866f4127172e619c4df3
View file https://github.com/simstudioai/sim/blob/ac644c06666acf4d53e494816d54284d87f29e33/apps/sim/lib/knowledge/access/predicate.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/8058/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/8058/files#diff-720ce8ebbff1c232b2a35c2b75ac2bb1983110156c6d866f4127172e619c4df3
https://github.com/simstudioai/sim/pull/8058/files#diff-720ce8ebbff1c232b2a35c2b75ac2bb1983110156c6d866f4127172e619c4df3
https://github.com/simstudioai/sim/pull/8058/files#diff-720ce8ebbff1c232b2a35c2b75ac2bb1983110156c6d866f4127172e619c4df3
apps/sim/lib/knowledge/search/projection-source-acl-backfill.test.tshttps://github.com/simstudioai/sim/pull/8058/files#diff-bef812b85dc282fb8c3741a0125810e9aa01bfee59c41d360c08343edecf7caa
View file https://github.com/simstudioai/sim/blob/ac644c06666acf4d53e494816d54284d87f29e33/apps/sim/lib/knowledge/search/projection-source-acl-backfill.test.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/8058/{{ revealButtonHref }}
apps/sim/lib/knowledge/search/projection-source-acl-backfill.tshttps://github.com/simstudioai/sim/pull/8058/files#diff-7ac6330ea0a7cf0177d32cf3e57641976a88161cf7eaef1e38bad30fbe0c31ca
View file https://github.com/simstudioai/sim/blob/ac644c06666acf4d53e494816d54284d87f29e33/apps/sim/lib/knowledge/search/projection-source-acl-backfill.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/8058/{{ revealButtonHref }}
apps/sim/lib/knowledge/search/queries.test.tshttps://github.com/simstudioai/sim/pull/8058/files#diff-b5bc7cdb4cf5dd93b0dc5beb76436fb9e70fc036e89833eaf8d9defb8bc83c3b
View file https://github.com/simstudioai/sim/blob/ac644c06666acf4d53e494816d54284d87f29e33/apps/sim/lib/knowledge/search/queries.test.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/8058/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/8058/files#diff-b5bc7cdb4cf5dd93b0dc5beb76436fb9e70fc036e89833eaf8d9defb8bc83c3b
https://github.com/simstudioai/sim/pull/8058/files#diff-b5bc7cdb4cf5dd93b0dc5beb76436fb9e70fc036e89833eaf8d9defb8bc83c3b
Please reload this pagehttps://github.com/simstudioai/sim/pull/8058/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/8058/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.