René's URL Explorer Experiment


Title: fix(search): preserve indexed vector retrieval and literal source text by icecrasher321 · Pull Request #7899 · simstudioai/sim · GitHub

Open Graph Title: fix(search): preserve indexed vector retrieval and literal source text by icecrasher321 · Pull Request #7899 · simstudioai/sim

X Title: fix(search): preserve indexed vector retrieval and literal source text by icecrasher321 · Pull Request #7899 · simstudioai/sim

Description: Summary Keep broad vector retrieval driven by the nearest-neighbor index when permission estimates are stale. Preserve authorization before the candidate limit, existing ANN settings, full-vector reranking, and a compact exact fallback for underfilled scans. Preserve literal source text in canonical stored .txt artifacts so HTML application shells and RTF snippets can be indexed without being interpreted as rendered documents. Extend the existing retrieval harness to cover the 512-dimensional candidate path, interleaved tenants, stale permission estimates, concurrent organization searches, and selective-scope recall. Type of Change Bug fix Testing The 206,000-chunk synthetic integration benchmark passed 26 tests on 2 CPUs / 1 GiB: 40 normal searches with no retrieval timeouts, 100% recall in all 11 exact-reference comparisons, and a maximum vector stage of 669 ms. Both filled ANN scans and selective exact fallbacks were exercised. Organization searches after shared-buffer eviction took 3.45–3.99 s overall, dominated by keyword retrieval. The operating-system cache was not cleared; provider HTTP responses were controlled while PostgreSQL, Redis, authorization, and application search ran normally. Both concurrency scenarios passed again with distinct query vectors. Independent SQL checks verified unused fallback CTEs on filled ANN scans and a single compact scoring scan with a spilling join at 64 KiB work_mem. Search query, budget, and diagnostics tests: 68 passed. Parser, document ingestion, and connector regression suites: 1,045 passed. Focused tests after merging latest staging: 254 passed. App type checking, repository lint, all 46 audits, block registry audit, and generated-artifact checks passed. Checklist Code follows project style guidelines Self-reviewed my changes Tests added/updated and passing No new warnings introduced I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

Open Graph Description: Summary Keep broad vector retrieval driven by the nearest-neighbor index when permission estimates are stale. Preserve authorization before the candidate limit, existing ANN settings, full-vector ...

X Description: Summary Keep broad vector retrieval driven by the nearest-neighbor index when permission estimates are stale. Preserve authorization before the candidate limit, existing ANN settings, full-vector ...

Opengraph URL: https://github.com/simstudioai/sim/pull/7899

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:a29b8c33-60b9-c223-f82e-9b55efe109bc
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idAC2A:277BD5:1A0D43:2443AB:6AFB2EE6
html-safe-nonceb93df3f1aa897fc4e61e2e1b191247d320cc0b5371ab892d7217267af8122022
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBQzJBOjI3N0JENToxQTBENDM6MjQ0M0FCOjZBRkIyRUU2IiwidmlzaXRvcl9pZCI6IjE1MzY3NDM4MDU3MjQzMzk5NDIiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmacdc446d8dee2712dbcfe9b482afd7ad6d5aeb54f9c553e91b800be6191b966c06
hovercard-subject-tagpull_request:4553072876
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/7899/files
twitter:imagehttps://avatars.githubusercontent.com/u/17950982?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/17950982?s=400&v=4
og:image:altSummary Keep broad vector retrieval driven by the nearest-neighbor index when permission estimates are stale. Preserve authorization before the candidate limit, existing ANN settings, full-vector ...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None40c7ffe00e8bbb3ca8403c83d7ef110379c9eba2cbc7a7c333b037ffd940d4e4
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/simstudioai/sim git https://github.com/simstudioai/sim.git
octolytics-dimension-user_id199344406
octolytics-dimension-user_loginsimstudioai
octolytics-dimension-repository_id912559512
octolytics-dimension-repository_nwosimstudioai/sim
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id912559512
octolytics-dimension-repository_network_root_nwosimstudioai/sim
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
releasef494d77c37809e03e18d49c6e9945c57e0b64070
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/simstudioai/sim/pull/7899/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F7899%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F7899%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=simstudioai%2Fsim
Reloadhttps://github.com/simstudioai/sim/pull/7899/files
Reloadhttps://github.com/simstudioai/sim/pull/7899/files
Reloadhttps://github.com/simstudioai/sim/pull/7899/files
simstudioaihttps://github.com/simstudioai
simhttps://github.com/simstudioai/sim
Codehttps://github.com/simstudioai/sim
Issues129 (129)https://github.com/simstudioai/sim/issues
Pull requests198 (198)https://github.com/simstudioai/sim/pulls
Actionshttps://github.com/simstudioai/sim/actions
Projectshttps://github.com/simstudioai/sim/projects
Security and qualityhttps://github.com/simstudioai/sim/security
Insightshttps://github.com/simstudioai/sim/pulse
Sign up for GitHub https://github.com/signup?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
icecrasher321https://github.com/icecrasher321
staginghttps://github.com/simstudioai/sim/tree/staging
codex/bound-vector-candidate-retrievalhttps://github.com/simstudioai/sim/tree/codex/bound-vector-candidate-retrieval
Conversation 21 https://github.com/simstudioai/sim/pull/7899
Commits 5 https://github.com/simstudioai/sim/pull/7899/commits
Checks 36 https://github.com/simstudioai/sim/pull/7899/checks
Files changed https://github.com/simstudioai/sim/pull/7899/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/7899/files
fix(search): preserve indexed vector retrieval and literal source text https://github.com/simstudioai/sim/pull/7899/files#top
Show all changes 5 commits https://github.com/simstudioai/sim/pull/7899/files
8d9c765 fix(search): preserve indexed vector retrieval and literal source text Sep 16, 2026 https://github.com/simstudioai/sim/pull/7899/commits/8d9c765d42481d8ddb03c6e7fc8aa1deacfb87d6
42372e2 Merge remote-tracking branch 'origin/staging' into codex/bound-vector… Sep 16, 2026 https://github.com/simstudioai/sim/pull/7899/commits/42372e27a3d8fc1d038540ab3b228c64a1482ed7
045e2c8 fix(search): keep benchmark corpus defaults within valid bounds Sep 16, 2026 https://github.com/simstudioai/sim/pull/7899/commits/045e2c806bdf41d70f84e8513ad9b4d44050a486
c117587 Merge remote-tracking branch 'origin/staging' into codex/bound-vector… Sep 16, 2026 https://github.com/simstudioai/sim/pull/7899/commits/c1175875407d18462457f3bb1ebb4fac9c7f0a9f
44cc97c fix(search): retain ANN recall settings and verify distinct queries Sep 17, 2026 https://github.com/simstudioai/sim/pull/7899/commits/44cc97c906c6b316d49677071539239068eb7d13
Clear filters https://github.com/simstudioai/sim/pull/7899/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/7899/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/7899/files
index.ts https://github.com/simstudioai/sim/pull/7899/files#diff-5340b2eaee20e0208a9b187e5946c1b1bf273be94c0e2da72d79adedc2a399f7
sniff.test.ts https://github.com/simstudioai/sim/pull/7899/files#diff-2b84ef868484ac582e11ec57c7a311a6469f43e82f18adab4483dfc2f66df717
sniff.ts https://github.com/simstudioai/sim/pull/7899/files#diff-ea3755f7fdef864432ca1291ef5babbfd014006964fc6317de9d69bb025153e9
types.ts https://github.com/simstudioai/sim/pull/7899/files#diff-65f59f17af217bd0758197a935b918001153deab15a0fcdf21618a6ca1c7633d
search-latency.integration.ts https://github.com/simstudioai/sim/pull/7899/files#diff-2a5bd119422fb2a9a479531e9e0ea926dbaef06eb2a9ae45e1f786fac32385e8
document-processor.ts https://github.com/simstudioai/sim/pull/7899/files#diff-caef18c7f5680f8ad0ec1e40b90e7635fe8f182b55d4fd20ef78fbe881f11205
stored-artifact-extension.test.ts https://github.com/simstudioai/sim/pull/7899/files#diff-0c3fc04759a119d0e72255e5752d417f5903716f2ddc443324f95f5e19e9a9f6
queries.test.ts https://github.com/simstudioai/sim/pull/7899/files#diff-b5bc7cdb4cf5dd93b0dc5beb76436fb9e70fc036e89833eaf8d9defb8bc83c3b
queries.ts https://github.com/simstudioai/sim/pull/7899/files#diff-5f10054baeef2f06350b32a6520d8f841f86a2fc908435296dd8dcb097497ba9
apps/sim/lib/file-parsers/index.tshttps://github.com/simstudioai/sim/pull/7899/files#diff-5340b2eaee20e0208a9b187e5946c1b1bf273be94c0e2da72d79adedc2a399f7
View file https://github.com/simstudioai/sim/blob/44cc97c906c6b316d49677071539239068eb7d13/apps/sim/lib/file-parsers/index.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/7899/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/7899/files#diff-5340b2eaee20e0208a9b187e5946c1b1bf273be94c0e2da72d79adedc2a399f7
https://github.com/simstudioai/sim/pull/7899/files#diff-5340b2eaee20e0208a9b187e5946c1b1bf273be94c0e2da72d79adedc2a399f7
apps/sim/lib/file-parsers/sniff.test.tshttps://github.com/simstudioai/sim/pull/7899/files#diff-2b84ef868484ac582e11ec57c7a311a6469f43e82f18adab4483dfc2f66df717
View file https://github.com/simstudioai/sim/blob/44cc97c906c6b316d49677071539239068eb7d13/apps/sim/lib/file-parsers/sniff.test.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/7899/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/7899/files#diff-2b84ef868484ac582e11ec57c7a311a6469f43e82f18adab4483dfc2f66df717
https://github.com/simstudioai/sim/pull/7899/files#diff-2b84ef868484ac582e11ec57c7a311a6469f43e82f18adab4483dfc2f66df717
apps/sim/lib/file-parsers/sniff.tshttps://github.com/simstudioai/sim/pull/7899/files#diff-ea3755f7fdef864432ca1291ef5babbfd014006964fc6317de9d69bb025153e9
View file https://github.com/simstudioai/sim/blob/44cc97c906c6b316d49677071539239068eb7d13/apps/sim/lib/file-parsers/sniff.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/7899/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/7899/files#diff-ea3755f7fdef864432ca1291ef5babbfd014006964fc6317de9d69bb025153e9
https://github.com/simstudioai/sim/pull/7899/files#diff-ea3755f7fdef864432ca1291ef5babbfd014006964fc6317de9d69bb025153e9
https://github.com/simstudioai/sim/pull/7899/files#diff-ea3755f7fdef864432ca1291ef5babbfd014006964fc6317de9d69bb025153e9
apps/sim/lib/file-parsers/types.tshttps://github.com/simstudioai/sim/pull/7899/files#diff-65f59f17af217bd0758197a935b918001153deab15a0fcdf21618a6ca1c7633d
View file https://github.com/simstudioai/sim/blob/44cc97c906c6b316d49677071539239068eb7d13/apps/sim/lib/file-parsers/types.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/7899/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/7899/files#diff-65f59f17af217bd0758197a935b918001153deab15a0fcdf21618a6ca1c7633d
https://github.com/simstudioai/sim/pull/7899/files#diff-65f59f17af217bd0758197a935b918001153deab15a0fcdf21618a6ca1c7633d
Please reload this pagehttps://github.com/simstudioai/sim/pull/7899/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/7899/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.