René's URL Explorer Experiment


Title: fix(knowledge): optimize search and trace pipeline latency by icecrasher321 · Pull Request #7782 · simstudioai/sim · GitHub

Open Graph Title: fix(knowledge): optimize search and trace pipeline latency by icecrasher321 · Pull Request #7782 · simstudioai/sim

X Title: fix(knowledge): optimize search and trace pipeline latency by icecrasher321 · Pull Request #7782 · simstudioai/sim

Description: Summary Keep broad permission-filtered vector searches on HNSW in the shared query layer. Probe visible IDs without reading vectors, rank small scopes by those bounded IDs, and restart exact fallback from its own cursor for underfilled approximate pages. Deduplicate previously considered candidates and preserve distance ranking across refills. Live source authorization still gates content. Add correlated pipeline timings for Assistant and Search, including active-stage reports for stalled calls, provenance, billing, provider authorization, SQL, and result presentation. Record UTF-8 output/passage byte counts without content, and correct the final tool trace to measure bytes rather than characters. Add an opt-in PostgreSQL/pgvector harness that calls the real Assistant tool and compares identical Search-tab requests across a 30,000-chunk synthetic corpus, selective permissions, revocation, concurrent calls, and organization scope. Correct the ACL-transition test to cover pending and completed rewrites for candidate and final-read predicates. Type of Change Bug fix Testing Synthetic paired SQL benchmark: median vector time including the new probe fell from 81.5 ms to 3.8 ms; buffer accesses fell from 123,278 to 2,403. All 50 returned candidates matched exact ranking in this corpus. These are local SQL measurements, not end-to-end or deployment latency claims. 8 real PostgreSQL/pgvector harness scenarios passed again after review fixes, including saved pipeline diagnostics, passage-size assertions, and matching Search-tab/Assistant SQL. The synthetic 15-passage response was about 54 KB (45 KB of text); local presentation and measurement took 0.4–1.7 ms. No model latency claim is made from this provider-controlled harness. 198 search, tool-execution, and diagnostic tests passed for the review fixes, including exact-cursor restarts, deduplication, cross-page ranking, and UTF-8 size reporting. 224 search/application/diagnostic unit tests, 20 database mock tests, and 66 broader access integration tests passed; one existing integration test was skipped. Full lint, all 26 workspace type checks, all 46 repository audits (including API validation), block-registry validation, generated artifacts, and docs-manifest checks passed. Run the harness from the repository root with Docker running: KNOWLEDGE_SEARCH_PERFORMANCE_REPORT_FILE=/tmp/search-performance.json bun run test:search-performance Checklist Code follows project style guidelines Self-reviewed my changes Tests added/updated and passing No new warnings introduced I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

Open Graph Description: Summary Keep broad permission-filtered vector searches on HNSW in the shared query layer. Probe visible IDs without reading vectors, rank small scopes by those bounded IDs, and restart exact fallb...

X Description: Summary Keep broad permission-filtered vector searches on HNSW in the shared query layer. Probe visible IDs without reading vectors, rank small scopes by those bounded IDs, and restart exact fallb...

Opengraph URL: https://github.com/simstudioai/sim/pull/7782

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:c00319f8-1284-8309-82b5-e2d432196ae9
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idE484:1C8E04:5B04C0:7E0E7A:6AFB3EB2
html-safe-nonce107bcc45b8bf8f5919ebde073b8c925d3babbc9945a69e4801cebae38ba74968
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJFNDg0OjFDOEUwNDo1QjA0QzA6N0UwRTdBOjZBRkIzRUIyIiwidmlzaXRvcl9pZCI6IjE3MTc2MTUxODIxODU0NDcwOTAiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmacbd48e11501d8c40f84390944569c2b39a14887cb573a19ce0cd0d1f91b363310
hovercard-subject-tagpull_request:4509200867
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/7782/files
twitter:imagehttps://avatars.githubusercontent.com/u/17950982?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/17950982?s=400&v=4
og:image:altSummary Keep broad permission-filtered vector searches on HNSW in the shared query layer. Probe visible IDs without reading vectors, rank small scopes by those bounded IDs, and restart exact fallb...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None2169b0dfbef67ed6b128074a5ed45bb7d5d1859783b10178b09a7765ad5f3693
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/simstudioai/sim git https://github.com/simstudioai/sim.git
octolytics-dimension-user_id199344406
octolytics-dimension-user_loginsimstudioai
octolytics-dimension-repository_id912559512
octolytics-dimension-repository_nwosimstudioai/sim
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id912559512
octolytics-dimension-repository_network_root_nwosimstudioai/sim
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release3b4fc8df5ef1c6757e733a23e6b81b6725021823
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/simstudioai/sim/pull/7782/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F7782%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F7782%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=simstudioai%2Fsim
Reloadhttps://github.com/simstudioai/sim/pull/7782/files
Reloadhttps://github.com/simstudioai/sim/pull/7782/files
Reloadhttps://github.com/simstudioai/sim/pull/7782/files
simstudioaihttps://github.com/simstudioai
simhttps://github.com/simstudioai/sim
Codehttps://github.com/simstudioai/sim
Issues129 (129)https://github.com/simstudioai/sim/issues
Pull requests202 (202)https://github.com/simstudioai/sim/pulls
Actionshttps://github.com/simstudioai/sim/actions
Projectshttps://github.com/simstudioai/sim/projects
Security and qualityhttps://github.com/simstudioai/sim/security
Insightshttps://github.com/simstudioai/sim/pulse
Sign up for GitHub https://github.com/signup?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
waleedlatif1https://github.com/waleedlatif1
staginghttps://github.com/simstudioai/sim/tree/staging
codex/fix-document-search-latencyhttps://github.com/simstudioai/sim/tree/codex/fix-document-search-latency
Conversation 9 https://github.com/simstudioai/sim/pull/7782
Commits 3 https://github.com/simstudioai/sim/pull/7782/commits
Checks 41 https://github.com/simstudioai/sim/pull/7782/checks
Files changed https://github.com/simstudioai/sim/pull/7782/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/7782/files
fix(knowledge): optimize search and trace pipeline latency https://github.com/simstudioai/sim/pull/7782/files#top
Show all changes 3 commits https://github.com/simstudioai/sim/pull/7782/files
e118ce8 fix(knowledge): optimize search and trace pipeline latency Sep 11, 2026 https://github.com/simstudioai/sim/pull/7782/commits/e118ce80aac17415bcff03ecf106e423b0639732
cd1b6d8 docs: remove standalone search latency notes Sep 11, 2026 https://github.com/simstudioai/sim/pull/7782/commits/cd1b6d8814adc1ae70abbd9a54c114943ecd4fd6
617ce71 fix(knowledge): preserve fallback ranking and measure result bytes Sep 11, 2026 https://github.com/simstudioai/sim/pull/7782/commits/617ce71e9c61a6424a7baa1da20d5b31d58d2347
Clear filters https://github.com/simstudioai/sim/pull/7782/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/7782/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/7782/files
executor.test.ts https://github.com/simstudioai/sim/pull/7782/files#diff-7539d2e1e1a23e035c0b4aada8a99a35520a8afe81c7a358f94c7fb69cf96185
executor.ts https://github.com/simstudioai/sim/pull/7782/files#diff-055301ea08ccdebe72fc75f6a483db681d15088e4f77e56358000f3575e065d6
workspace-search.test.ts https://github.com/simstudioai/sim/pull/7782/files#diff-ee4bd8f28eaa3e65861abfd901c46cf7cb56c8ed04d54e1b388be07cfd65e7eb
workspace-search.ts https://github.com/simstudioai/sim/pull/7782/files#diff-5e7ab9e936b0780a19edfeb0c6b4b82fec582920f0247555582003b07b79c26f
application-acl.integration.ts https://github.com/simstudioai/sim/pull/7782/files#diff-853351f6b5e867c5a2de04a9dfe0284f05330e6d2e290c48f56a235e13bde8c9
search-latency.integration.ts https://github.com/simstudioai/sim/pull/7782/files#diff-2a5bd119422fb2a9a479531e9e0ea926dbaef06eb2a9ae45e1f786fac32385e8
seed-search-reader-fixture.ts https://github.com/simstudioai/sim/pull/7782/files#diff-486e5d1eb8de1746c08e47e8be8347f256762f9c1aa5d788f6737743dbbf8e56
search-diagnostics.ts https://github.com/simstudioai/sim/pull/7782/files#diff-f0f854fc56b62af3fc9d1069eb09a789c89bc07e23862a03d717e6ccf1902caa
search.ts https://github.com/simstudioai/sim/pull/7782/files#diff-6d92dc3a4dd3235b5c805a3ef81a2a271e98cc2632caf3274d8a1ac938644934
workspace-search.ts https://github.com/simstudioai/sim/pull/7782/files#diff-9580ec7774638e234982040c96120c35ac8b57a4271198bcc23bebd524c1d740
diagnostics.test.ts https://github.com/simstudioai/sim/pull/7782/files#diff-3bb11f053a684271914ea6f2d76600b881044ccee5d9152eefbc1e9b26100ea8
diagnostics.ts https://github.com/simstudioai/sim/pull/7782/files#diff-454cb97c4d1f9ca41cc9ec69e41039731b15cf7250ff59d784f19495ea0caea6
queries.test.ts https://github.com/simstudioai/sim/pull/7782/files#diff-b5bc7cdb4cf5dd93b0dc5beb76436fb9e70fc036e89833eaf8d9defb8bc83c3b
queries.ts https://github.com/simstudioai/sim/pull/7782/files#diff-5f10054baeef2f06350b32a6520d8f841f86a2fc908435296dd8dcb097497ba9
package.json https://github.com/simstudioai/sim/pull/7782/files#diff-7ae45ad102eab3b6d7e7896acd08c427a9b25b346470d7bc6507b6481575d519
database.mock.test.ts https://github.com/simstudioai/sim/pull/7782/files#diff-d1ce09bbd60d07080ae08563dcbe64479f30aa36ca3e4366493794678ceae63a
database.mock.ts https://github.com/simstudioai/sim/pull/7782/files#diff-859f381ad1b6a93489bef08c44a1044a2c3cc6f9b98b8d973bb5b0ef893ba42b
test-knowledge-acls.ts https://github.com/simstudioai/sim/pull/7782/files#diff-11d2eed992824bef9b61574a683fb54ee2e34caa0bf7a62ece280a56250db1af
apps/sim/lib/copilot/request/tools/executor.test.tshttps://github.com/simstudioai/sim/pull/7782/files#diff-7539d2e1e1a23e035c0b4aada8a99a35520a8afe81c7a358f94c7fb69cf96185
View file https://github.com/simstudioai/sim/blob/617ce71e9c61a6424a7baa1da20d5b31d58d2347/apps/sim/lib/copilot/request/tools/executor.test.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/7782/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/7782/files#diff-7539d2e1e1a23e035c0b4aada8a99a35520a8afe81c7a358f94c7fb69cf96185
https://github.com/simstudioai/sim/pull/7782/files#diff-7539d2e1e1a23e035c0b4aada8a99a35520a8afe81c7a358f94c7fb69cf96185
apps/sim/lib/copilot/request/tools/executor.tshttps://github.com/simstudioai/sim/pull/7782/files#diff-055301ea08ccdebe72fc75f6a483db681d15088e4f77e56358000f3575e065d6
View file https://github.com/simstudioai/sim/blob/617ce71e9c61a6424a7baa1da20d5b31d58d2347/apps/sim/lib/copilot/request/tools/executor.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/7782/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/7782/files#diff-055301ea08ccdebe72fc75f6a483db681d15088e4f77e56358000f3575e065d6
https://github.com/simstudioai/sim/pull/7782/files#diff-055301ea08ccdebe72fc75f6a483db681d15088e4f77e56358000f3575e065d6
https://github.com/simstudioai/sim/pull/7782/files#diff-055301ea08ccdebe72fc75f6a483db681d15088e4f77e56358000f3575e065d6
apps/sim/lib/copilot/tools/server/knowledge/workspace-search.test.tshttps://github.com/simstudioai/sim/pull/7782/files#diff-ee4bd8f28eaa3e65861abfd901c46cf7cb56c8ed04d54e1b388be07cfd65e7eb
View file https://github.com/simstudioai/sim/blob/617ce71e9c61a6424a7baa1da20d5b31d58d2347/apps/sim/lib/copilot/tools/server/knowledge/workspace-search.test.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/7782/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/7782/files#diff-ee4bd8f28eaa3e65861abfd901c46cf7cb56c8ed04d54e1b388be07cfd65e7eb
https://github.com/simstudioai/sim/pull/7782/files#diff-ee4bd8f28eaa3e65861abfd901c46cf7cb56c8ed04d54e1b388be07cfd65e7eb
https://github.com/simstudioai/sim/pull/7782/files#diff-ee4bd8f28eaa3e65861abfd901c46cf7cb56c8ed04d54e1b388be07cfd65e7eb
apps/sim/lib/copilot/tools/server/knowledge/workspace-search.tshttps://github.com/simstudioai/sim/pull/7782/files#diff-5e7ab9e936b0780a19edfeb0c6b4b82fec582920f0247555582003b07b79c26f
View file https://github.com/simstudioai/sim/blob/617ce71e9c61a6424a7baa1da20d5b31d58d2347/apps/sim/lib/copilot/tools/server/knowledge/workspace-search.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/7782/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/7782/files#diff-5e7ab9e936b0780a19edfeb0c6b4b82fec582920f0247555582003b07b79c26f
https://github.com/simstudioai/sim/pull/7782/files#diff-5e7ab9e936b0780a19edfeb0c6b4b82fec582920f0247555582003b07b79c26f
https://github.com/simstudioai/sim/pull/7782/files#diff-5e7ab9e936b0780a19edfeb0c6b4b82fec582920f0247555582003b07b79c26f
apps/sim/lib/knowledge/__integration__/application-acl.integration.tshttps://github.com/simstudioai/sim/pull/7782/files#diff-853351f6b5e867c5a2de04a9dfe0284f05330e6d2e290c48f56a235e13bde8c9
View file https://github.com/simstudioai/sim/blob/617ce71e9c61a6424a7baa1da20d5b31d58d2347/apps/sim/lib/knowledge/__integration__/application-acl.integration.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/7782/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/7782/files#diff-853351f6b5e867c5a2de04a9dfe0284f05330e6d2e290c48f56a235e13bde8c9
https://github.com/simstudioai/sim/pull/7782/files#diff-853351f6b5e867c5a2de04a9dfe0284f05330e6d2e290c48f56a235e13bde8c9
https://github.com/simstudioai/sim/pull/7782/files#diff-853351f6b5e867c5a2de04a9dfe0284f05330e6d2e290c48f56a235e13bde8c9
https://github.com/simstudioai/sim/pull/7782/files#diff-853351f6b5e867c5a2de04a9dfe0284f05330e6d2e290c48f56a235e13bde8c9
https://github.com/simstudioai/sim/pull/7782/files#diff-853351f6b5e867c5a2de04a9dfe0284f05330e6d2e290c48f56a235e13bde8c9
Please reload this pagehttps://github.com/simstudioai/sim/pull/7782/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/7782/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.