René's URL Explorer Experiment


Title: fix(knowledge): apply the source and date filters inside a resolved scope's ranking by waleedlatif1 · Pull Request #8059 · simstudioai/sim · GitHub

Open Graph Title: fix(knowledge): apply the source and date filters inside a resolved scope's ranking by waleedlatif1 · Pull Request #8059 · simstudioai/sim

X Title: fix(knowledge): apply the source and date filters inside a resolved scope's ranking by waleedlatif1 · Pull Request #8059 · simstudioai/sim

Description: Summary The Search page's source and updated filters were not applied inside the ranking on the resolved-scope path — only at hydration — so a filtered search ranked candidates the filter then refused. On a production-shaped copy, updated: past week returned 0 rows although thousands of documents qualified, a small-source filter returned 4, and every filtered search's keyword leg ran to its deadline (this predates #8040) A source filter now confines the access plan itself: every on-row predicate, the reach, and the sources the legs walk or rank are that kind of source alone; upload keeps only source-less documents A date filter enumerates the documents it admits off a new (knowledge_base_id, source_modified_at) partial index and ranks them exactly while the planner estimates the window within the probe's limit; a wider window is walked with the date tested through the document, on the row and on the ranked keyword row A date-bounded set is ranked exactly even when a member source has its own index, since a walk cannot see the date; the reach is keyed and counted by the plan's sources A keyword leg whose deadline passes before its ranking is resolved is short rather than failed Measurements Same copy, same reader (reaches ~63% of the index), complete results; wall time over a ~80 ms round trip: filter share of the index it admits before after source (large) ~95% 20 rows, 2.9–5.1 s 20 rows, 2.1–2.7 s source (small) ~0.2% 4–20 rows, ~8 s (keyword timed out) 20 rows, 3.0–3.6 s updated: past month ~12% 20 rows, ~8 s (keyword timed out) 20 rows, 2.3–4.5 s updated: past week ~0.4% 0 rows, ~8 s (keyword timed out) 20 rows, 2.8–4.0 s Server side the new paths are hundreds of milliseconds: past week probes off the date index (~300 ms) then ranks the bounded set exactly (~200 ms); past month is one on-row walk at 340–450 ms. Without the index the date probe was a 320–640 ms sequential scan. Type of Change Bug fix Testing Unit tests for the plan restriction, the uploads flag in both predicates, the reach keyed by sources, the date probe (filter-first, estimate-gated), the on-row date test in walks and the Tin page, exact ranking of a date-bounded set, and the keyword-leg deadline; each verified to fail with its fix reverted Real-Postgres predicate equivalence test extended with source- and upload-restricted plans bun run check:migrations origin/staging passes; lint, block registry, check:audits, docs manifest pass Checklist Code follows project style guidelines Self-reviewed my changes Tests added/updated and passing No new warnings introduced I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

Open Graph Description: Summary The Search page's source and updated filters were not applied inside the ranking on the resolved-scope path — only at hydration — so a filtered search ranked candidates the filter then...

X Description: Summary The Search page's source and updated filters were not applied inside the ranking on the resolved-scope path — only at hydration — so a filtered search ranked candidates the filter ...

Opengraph URL: https://github.com/simstudioai/sim/pull/8059

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:72beaa44-7dc8-e77a-c642-f050fc5c3b99
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idDB2C:AAABF:5042C4:6FFC0C:6AFB5DB8
html-safe-noncec91941c36d16ec7fa61cfad1740c96b0590b3ca5f2d7b185678be44835226dad
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJEQjJDOkFBQUJGOjUwNDJDNDo2RkZDMEM6NkFGQjVEQjgiLCJ2aXNpdG9yX2lkIjoiMzkwMjk4NjY2MDg0ODgxOTY0MCIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmac8c00cee2406e43d2297c61b69456d89c5da323491f49b8f926318604a55a5fa2
hovercard-subject-tagpull_request:4585115391
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/8059/files
twitter:imagehttps://avatars.githubusercontent.com/u/40672544?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/40672544?s=400&v=4
og:image:altSummary The Search page's source and updated filters were not applied inside the ranking on the resolved-scope path — only at hydration — so a filtered search ranked candidates the filter then...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
Noneea33d13c96d0ff66bd8fa906b193d2213536b78cb06b0c50a4632c58b1418f94
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/simstudioai/sim git https://github.com/simstudioai/sim.git
octolytics-dimension-user_id199344406
octolytics-dimension-user_loginsimstudioai
octolytics-dimension-repository_id912559512
octolytics-dimension-repository_nwosimstudioai/sim
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id912559512
octolytics-dimension-repository_network_root_nwosimstudioai/sim
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release59ef97164c52cf59bd10558ddec9c4f482763988
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/simstudioai/sim/pull/8059/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F8059%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F8059%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=simstudioai%2Fsim
Reloadhttps://github.com/simstudioai/sim/pull/8059/files
Reloadhttps://github.com/simstudioai/sim/pull/8059/files
Reloadhttps://github.com/simstudioai/sim/pull/8059/files
simstudioaihttps://github.com/simstudioai
simhttps://github.com/simstudioai/sim
Codehttps://github.com/simstudioai/sim
Issues129 (129)https://github.com/simstudioai/sim/issues
Pull requests197 (197)https://github.com/simstudioai/sim/pulls
Actionshttps://github.com/simstudioai/sim/actions
Projectshttps://github.com/simstudioai/sim/projects
Security and qualityhttps://github.com/simstudioai/sim/security
Insightshttps://github.com/simstudioai/sim/pulse
Sign up for GitHub https://github.com/signup?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
waleedlatif1https://github.com/waleedlatif1
staginghttps://github.com/simstudioai/sim/tree/staging
improvement/search-filters-on-rowhttps://github.com/simstudioai/sim/tree/improvement/search-filters-on-row
Conversation 28 https://github.com/simstudioai/sim/pull/8059
Commits 4 https://github.com/simstudioai/sim/pull/8059/commits
Checks 33 https://github.com/simstudioai/sim/pull/8059/checks
Files changed https://github.com/simstudioai/sim/pull/8059/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/8059/files
fix(knowledge): apply the source and date filters inside a resolved scope's ranking https://github.com/simstudioai/sim/pull/8059/files#top
Show all changes 4 commits https://github.com/simstudioai/sim/pull/8059/files
a35fac6 fix(knowledge): apply the source and date filters inside a resolved s… waleedlatif1 Sep 20, 2026 https://github.com/simstudioai/sim/pull/8059/commits/a35fac667a4862553d80005bd78242f435222d1d
04f17dc fix(knowledge): enumerate a small filtered source so both legs rank i… waleedlatif1 Sep 20, 2026 https://github.com/simstudioai/sim/pull/8059/commits/04f17dcb749d08c20986d649736e4aea521a9389
1b59f04 chore(db): mark the document date lookup index concurrent in the schema waleedlatif1 Sep 20, 2026 https://github.com/simstudioai/sim/pull/8059/commits/1b59f04c202125eb26eab5848e7c329adbd3825c
8ee7f85 fix(knowledge): estimate a filter's size under a short deadline of it… waleedlatif1 Sep 20, 2026 https://github.com/simstudioai/sim/pull/8059/commits/8ee7f85d8a7a85078dd7508fe7f09d8e4ae97b96
Clear filters https://github.com/simstudioai/sim/pull/8059/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/8059/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/8059/files
connector-eligibility.ts https://github.com/simstudioai/sim/pull/8059/files#diff-b31b54f4344ef60b28f4b657981442701838254fd3f64a22e06dfabbe7ebc0ce
predicate.postgres.test.ts https://github.com/simstudioai/sim/pull/8059/files#diff-0594c8e9ccb3d01589226b3540ac725c18ddafe355b80b2472f9905586c1161e
predicate.test.ts https://github.com/simstudioai/sim/pull/8059/files#diff-cbeeaa5cc835fb1fedc629c869b2a5f952eb1c7343e44960b6a8dfe126660ff1
predicate.ts https://github.com/simstudioai/sim/pull/8059/files#diff-720ce8ebbff1c232b2a35c2b75ac2bb1983110156c6d866f4127172e619c4df3
queries.test.ts https://github.com/simstudioai/sim/pull/8059/files#diff-b5bc7cdb4cf5dd93b0dc5beb76436fb9e70fc036e89833eaf8d9defb8bc83c3b
queries.ts https://github.com/simstudioai/sim/pull/8059/files#diff-5f10054baeef2f06350b32a6520d8f841f86a2fc908435296dd8dcb097497ba9
0371_document_modified_lookup.sql https://github.com/simstudioai/sim/pull/8059/files#diff-a0206041b8e835dbb8d1496a9cec44c75c40d26e091a4f7951066f9666405d9a
0371_snapshot.json https://github.com/simstudioai/sim/pull/8059/files#diff-9e12a5fd4217d96fad38f49eb024e51e0c8d598a8a203b5821a14c59f596c712
_journal.json https://github.com/simstudioai/sim/pull/8059/files#diff-37150cc65ad16ee6fbc5fa318fc096be7730e16d63eaf863e3df5521b7030649
schema.ts https://github.com/simstudioai/sim/pull/8059/files#diff-9a959e2ce3cb4c86957bbae021d06eae364f7b0cb18964f5873bab53220af928
apps/sim/lib/knowledge/access/connector-eligibility.tshttps://github.com/simstudioai/sim/pull/8059/files#diff-b31b54f4344ef60b28f4b657981442701838254fd3f64a22e06dfabbe7ebc0ce
View file https://github.com/simstudioai/sim/blob/8ee7f85d8a7a85078dd7508fe7f09d8e4ae97b96/apps/sim/lib/knowledge/access/connector-eligibility.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/8059/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/8059/files#diff-b31b54f4344ef60b28f4b657981442701838254fd3f64a22e06dfabbe7ebc0ce
https://github.com/simstudioai/sim/pull/8059/files#diff-b31b54f4344ef60b28f4b657981442701838254fd3f64a22e06dfabbe7ebc0ce
https://github.com/simstudioai/sim/pull/8059/files#diff-b31b54f4344ef60b28f4b657981442701838254fd3f64a22e06dfabbe7ebc0ce
https://github.com/simstudioai/sim/pull/8059/files#diff-b31b54f4344ef60b28f4b657981442701838254fd3f64a22e06dfabbe7ebc0ce
apps/sim/lib/knowledge/access/predicate.postgres.test.tshttps://github.com/simstudioai/sim/pull/8059/files#diff-0594c8e9ccb3d01589226b3540ac725c18ddafe355b80b2472f9905586c1161e
View file https://github.com/simstudioai/sim/blob/8ee7f85d8a7a85078dd7508fe7f09d8e4ae97b96/apps/sim/lib/knowledge/access/predicate.postgres.test.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/8059/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/8059/files#diff-0594c8e9ccb3d01589226b3540ac725c18ddafe355b80b2472f9905586c1161e
https://github.com/simstudioai/sim/pull/8059/files#diff-0594c8e9ccb3d01589226b3540ac725c18ddafe355b80b2472f9905586c1161e
https://github.com/simstudioai/sim/pull/8059/files#diff-0594c8e9ccb3d01589226b3540ac725c18ddafe355b80b2472f9905586c1161e
https://github.com/simstudioai/sim/pull/8059/files#diff-0594c8e9ccb3d01589226b3540ac725c18ddafe355b80b2472f9905586c1161e
https://github.com/simstudioai/sim/pull/8059/files#diff-0594c8e9ccb3d01589226b3540ac725c18ddafe355b80b2472f9905586c1161e
https://github.com/simstudioai/sim/pull/8059/files#diff-0594c8e9ccb3d01589226b3540ac725c18ddafe355b80b2472f9905586c1161e
apps/sim/lib/knowledge/access/predicate.test.tshttps://github.com/simstudioai/sim/pull/8059/files#diff-cbeeaa5cc835fb1fedc629c869b2a5f952eb1c7343e44960b6a8dfe126660ff1
View file https://github.com/simstudioai/sim/blob/8ee7f85d8a7a85078dd7508fe7f09d8e4ae97b96/apps/sim/lib/knowledge/access/predicate.test.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/8059/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/8059/files#diff-cbeeaa5cc835fb1fedc629c869b2a5f952eb1c7343e44960b6a8dfe126660ff1
https://github.com/simstudioai/sim/pull/8059/files#diff-cbeeaa5cc835fb1fedc629c869b2a5f952eb1c7343e44960b6a8dfe126660ff1
https://github.com/simstudioai/sim/pull/8059/files#diff-cbeeaa5cc835fb1fedc629c869b2a5f952eb1c7343e44960b6a8dfe126660ff1
https://github.com/simstudioai/sim/pull/8059/files#diff-cbeeaa5cc835fb1fedc629c869b2a5f952eb1c7343e44960b6a8dfe126660ff1
https://github.com/simstudioai/sim/pull/8059/files#diff-cbeeaa5cc835fb1fedc629c869b2a5f952eb1c7343e44960b6a8dfe126660ff1
apps/sim/lib/knowledge/access/predicate.tshttps://github.com/simstudioai/sim/pull/8059/files#diff-720ce8ebbff1c232b2a35c2b75ac2bb1983110156c6d866f4127172e619c4df3
View file https://github.com/simstudioai/sim/blob/8ee7f85d8a7a85078dd7508fe7f09d8e4ae97b96/apps/sim/lib/knowledge/access/predicate.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/8059/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/8059/files#diff-720ce8ebbff1c232b2a35c2b75ac2bb1983110156c6d866f4127172e619c4df3
https://github.com/simstudioai/sim/pull/8059/files#diff-720ce8ebbff1c232b2a35c2b75ac2bb1983110156c6d866f4127172e619c4df3
https://github.com/simstudioai/sim/pull/8059/files#diff-720ce8ebbff1c232b2a35c2b75ac2bb1983110156c6d866f4127172e619c4df3
https://github.com/simstudioai/sim/pull/8059/files#diff-720ce8ebbff1c232b2a35c2b75ac2bb1983110156c6d866f4127172e619c4df3
https://github.com/simstudioai/sim/pull/8059/files#diff-720ce8ebbff1c232b2a35c2b75ac2bb1983110156c6d866f4127172e619c4df3
https://github.com/simstudioai/sim/pull/8059/files#diff-720ce8ebbff1c232b2a35c2b75ac2bb1983110156c6d866f4127172e619c4df3
Please reload this pagehttps://github.com/simstudioai/sim/pull/8059/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/8059/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.