René's URL Explorer Experiment


Title: improvement(knowledge): finish Confluence passes past unreadable attachments, rank unauthenticated keyword search in one statement by waleedlatif1 · Pull Request #8975 · simstudioai/sim · GitHub

Open Graph Title: improvement(knowledge): finish Confluence passes past unreadable attachments, rank unauthenticated keyword search in one statement by waleedlatif1 · Pull Request #8975 · simstudioai/sim

X Title: improvement(knowledge): finish Confluence passes past unreadable attachments, rank unauthenticated keyword search in one statement by waleedlatif1 · Pull Request #8975 · simstudioai/sim

Description: Summary Confluence: a 403/404 on one parent's attachments no longer fails the listing listConfluenceAttachments used to record attachment_access_denied (403/404 on /pages/{id}/attachments) as a listing failure. Any pass with one permanently restricted page therefore stayed incomplete: the sync was partial, lastSyncAt never advanced, and every slice recounted the whole connector. That parent is now skipped and logged as a warning. The page still returns reconciliationSafe: false and sets syncContext.reconciliationUnsafe. Because checkpoint.unsafe stays set for the rest of the pass, reconciliation never deletes attachments it did not list. This is the same deletion behavior as before. Missing scope (401 attachment_scope_mismatch) and server failures (attachment_listing_unavailable, plus the 3-in-a-row abort) are still listing failures. A failure listing the pages themselves still throws. Connector document count The member engine now uses the same gate the content engine has had since #8944. Once a pass has finished, an unfinished slice keeps the recorded count instead of scanning the whole connector. Until a pass finishes, every slice still counts, so the first-pass fix from ed2e401 still holds. countLiveConnectorDocuments and the gate (keepsRecordedDocCount) moved to sync-primitives so both engines share them. With the Confluence fix, a connector stuck on restricted attachments now finishes passes, so the existing gate applies to it. Keyword search for callers with no person behind them (workflow, API, Copilot) These callers now rank through keywordCandidateRankingQuery, the materialized match → readable documents → rank statement signed-in readers already use. The old ORDER BY ts_rank_cd over a per-chunk embedding ⋈ document join (with a per-base fan-out for many bases) is gone. Over 7 days in prod the old shape ran p50 ~400 ms / p99 ~9.2 s; the CTE shape ran p50 ~27 ms / p99 ~1.2 s. The ACL predicate is the same read.rankCondition, and hydration is unchanged. The ordering is still a global ts_rank_cd DESC over every readable match. Ties are now broken by chunk id instead of arbitrarily. The candidate side also requires document.knowledge_base_id to be in the searched bases, which only makes the filter stricter. The search budget is deliberately unchanged: these callers still run unbudgeted. Adding a deadline would turn slow searches into partial results for API callers, which is a behavior change. Not changed (deliberately) I did not add a "skip the count when a slice changed no membership" gate for connectors that have never finished a pass. I could not prove it exact. Restores (deletedAt: null on relisted tombstones) and resurrecting updates are not counted separately. Out-of-band excludes, deletes and restores never touch lastSyncDocCount. Without a recorded count timestamp there is also no bounded-staleness variant. Test plan attachments.test.ts: a 403/404 parent is skipped, the listing stays complete, no listingFailures, and it is still reconciliation-unsafe. Fails on the old code. member-sync-engine-content-credential.test.ts: an unfinished slice after a finished pass writes no lastSyncDocCount. Fails on the old code. Focused suites pass: sync-engine, member-sync-engine (both files), confluence, search queries, application search. CI: type-check, audits, integration (filtered-search / scale cover keyword search end to end).

Open Graph Description: Summary Confluence: a 403/404 on one parent's attachments no longer fails the listing listConfluenceAttachments used to record attachment_access_denied (403/404 on /pages/{id}/attachments) as ...

X Description: Summary Confluence: a 403/404 on one parent's attachments no longer fails the listing listConfluenceAttachments used to record attachment_access_denied (403/404 on /pages/{id}/attachments)...

Opengraph URL: https://github.com/simstudioai/sim/pull/8975

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:c13d379b-f6fd-41ca-d8da-38893541ac6b
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idD32A:2C34A2:16F387:2039B8:6AFB4E01
html-safe-nonce60c30a07b9270167f07675937e0759144b0f4b7a938dd05a8a5abe87b45ac2e3
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJEMzJBOjJDMzRBMjoxNkYzODc6MjAzOUI4OjZBRkI0RTAxIiwidmlzaXRvcl9pZCI6IjI5NTgwODYwMzAyOTg0MDIzMDUiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmaceead42efc5018417e85876d873308e9cfe278b56af32c4bc766a9a4b9c7d64f4
hovercard-subject-tagpull_request:4847693702
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/8975/files
twitter:imagehttps://avatars.githubusercontent.com/u/40672544?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/40672544?s=400&v=4
og:image:altSummary Confluence: a 403/404 on one parent's attachments no longer fails the listing listConfluenceAttachments used to record attachment_access_denied (403/404 on /pages/{id}/attachments) as ...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
Noneacf7616d06cc051644705b379878c9bb13dc5e70ecf4ffd9b34140f8ac01d7e1
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/simstudioai/sim git https://github.com/simstudioai/sim.git
octolytics-dimension-user_id199344406
octolytics-dimension-user_loginsimstudioai
octolytics-dimension-repository_id912559512
octolytics-dimension-repository_nwosimstudioai/sim
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id912559512
octolytics-dimension-repository_network_root_nwosimstudioai/sim
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release26504db649b58525278f77a52b80ba5f10d5f4de
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/simstudioai/sim/pull/8975/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F8975%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F8975%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=simstudioai%2Fsim
Reloadhttps://github.com/simstudioai/sim/pull/8975/files
Reloadhttps://github.com/simstudioai/sim/pull/8975/files
Reloadhttps://github.com/simstudioai/sim/pull/8975/files
simstudioaihttps://github.com/simstudioai
simhttps://github.com/simstudioai/sim
Codehttps://github.com/simstudioai/sim
Issues129 (129)https://github.com/simstudioai/sim/issues
Pull requests197 (197)https://github.com/simstudioai/sim/pulls
Actionshttps://github.com/simstudioai/sim/actions
Projectshttps://github.com/simstudioai/sim/projects
Security and qualityhttps://github.com/simstudioai/sim/security
Insightshttps://github.com/simstudioai/sim/pulse
Sign up for GitHub https://github.com/signup?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
waleedlatif1https://github.com/waleedlatif1
staginghttps://github.com/simstudioai/sim/tree/staging
perf/kb-spikeshttps://github.com/simstudioai/sim/tree/perf/kb-spikes
Conversation 5 https://github.com/simstudioai/sim/pull/8975
Commits 2 https://github.com/simstudioai/sim/pull/8975/commits
Checks 50 https://github.com/simstudioai/sim/pull/8975/checks
Files changed https://github.com/simstudioai/sim/pull/8975/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/8975/files
improvement(knowledge): finish Confluence passes past unreadable attachments, rank unauthenticated keyword search in one statement https://github.com/simstudioai/sim/pull/8975/files#top
Show all changes 2 commits https://github.com/simstudioai/sim/pull/8975/files
08d3b4d improvement(knowledge): finish Confluence passes past unreadable atta… waleedlatif1 Oct 13, 2026 https://github.com/simstudioai/sim/pull/8975/commits/08d3b4db0b7f97bd1eae0ed892602b013e456c95
8bdc00f test(knowledge): compare single-statement keyword ranking against the… waleedlatif1 Oct 13, 2026 https://github.com/simstudioai/sim/pull/8975/commits/8bdc00f1b23448b98b1a090c3ea7dbff28115a37
Clear filters https://github.com/simstudioai/sim/pull/8975/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/8975/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/8975/files
attachments.test.ts https://github.com/simstudioai/sim/pull/8975/files#diff-f53714e26f7a718edfbe7e0835a1195d04280eb8ab351cfd0845301b38cd9b72
attachments.ts https://github.com/simstudioai/sim/pull/8975/files#diff-613dd06e1326ec452f035068e9df221bc3755a29b7618995a0b633d3ff418216
keyword-ranking.integration.ts https://github.com/simstudioai/sim/pull/8975/files#diff-dbc1b7d20728cb75a89040787a45199d84a9d2184eceb740400de2960f7f7476
member-sync-engine-content-credential.test.ts https://github.com/simstudioai/sim/pull/8975/files#diff-40f1876e4f1f511d3de4d450ad9f97ffb6d2b500a4017fab27b899af27484d57
member-sync-engine.ts https://github.com/simstudioai/sim/pull/8975/files#diff-f5784431ef7cb3961aec5ea7513b6af82c03f47030b45533077f1be9233550d0
sync-engine.ts https://github.com/simstudioai/sim/pull/8975/files#diff-277f76464f02682e44e6de58ebd215c927f0f6a3a7165af669a0717b593790e9
sync-primitives.ts https://github.com/simstudioai/sim/pull/8975/files#diff-f03702ab1eda626483984539879fd0475efc6ee2b6bf9c1df86cca4ead80a1fb
queries.ts https://github.com/simstudioai/sim/pull/8975/files#diff-5f10054baeef2f06350b32a6520d8f841f86a2fc908435296dd8dcb097497ba9
apps/sim/connectors/confluence/attachments.test.tshttps://github.com/simstudioai/sim/pull/8975/files#diff-f53714e26f7a718edfbe7e0835a1195d04280eb8ab351cfd0845301b38cd9b72
View file https://github.com/simstudioai/sim/blob/8bdc00f1b23448b98b1a090c3ea7dbff28115a37/apps/sim/connectors/confluence/attachments.test.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/8975/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/8975/files#diff-f53714e26f7a718edfbe7e0835a1195d04280eb8ab351cfd0845301b38cd9b72
https://github.com/simstudioai/sim/pull/8975/files#diff-f53714e26f7a718edfbe7e0835a1195d04280eb8ab351cfd0845301b38cd9b72
apps/sim/connectors/confluence/attachments.tshttps://github.com/simstudioai/sim/pull/8975/files#diff-613dd06e1326ec452f035068e9df221bc3755a29b7618995a0b633d3ff418216
View file https://github.com/simstudioai/sim/blob/8bdc00f1b23448b98b1a090c3ea7dbff28115a37/apps/sim/connectors/confluence/attachments.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/8975/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/8975/files#diff-613dd06e1326ec452f035068e9df221bc3755a29b7618995a0b633d3ff418216
https://github.com/simstudioai/sim/pull/8975/files#diff-613dd06e1326ec452f035068e9df221bc3755a29b7618995a0b633d3ff418216
https://github.com/simstudioai/sim/pull/8975/files#diff-613dd06e1326ec452f035068e9df221bc3755a29b7618995a0b633d3ff418216
https://github.com/simstudioai/sim/pull/8975/files#diff-613dd06e1326ec452f035068e9df221bc3755a29b7618995a0b633d3ff418216
https://github.com/simstudioai/sim/pull/8975/files#diff-613dd06e1326ec452f035068e9df221bc3755a29b7618995a0b633d3ff418216
https://github.com/simstudioai/sim/pull/8975/files#diff-613dd06e1326ec452f035068e9df221bc3755a29b7618995a0b633d3ff418216
https://github.com/simstudioai/sim/pull/8975/files#diff-613dd06e1326ec452f035068e9df221bc3755a29b7618995a0b633d3ff418216
Please reload this pagehttps://github.com/simstudioai/sim/pull/8975/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/8975/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.