René's URL Explorer Experiment


Title: feat(connectors): add 9 knowledge base connectors by waleedlatif1 · Pull Request #6699 · simstudioai/sim · GitHub

Open Graph Title: feat(connectors): add 9 knowledge base connectors by waleedlatif1 · Pull Request #6699 · simstudioai/sim

X Title: feat(connectors): add 9 knowledge base connectors by waleedlatif1 · Pull Request #6699 · simstudioai/sim

Description: What Adds 9 knowledge base connectors, closing the highest-frequency gaps in our catalog. Every service already shipped a block, so OAuth providers, credentials, and icons were reused — no new auth plumbing. Connector Auth Syncs Box OAuth box Files, recursive from a folder or the whole account Zoho Desk OAuth zoho-desk Help Center articles + tickets with conversations PagerDuty API key Incidents, notes, timeline log entries Trello OAuth trello Cards with descriptions, checklists, comments, attachments Microsoft Excel OAuth microsoft-excel Worksheet data Google Slides OAuth google-drive Slide text + speaker notes Google Vault OAuth google-vault Matters, holds, saved queries Mintlify API key (optional) Documentation pages via llms.txt SFTP API key Remote file trees Selected by surveying the published connector catalogs of Glean, Onyx, Dust, Vectara, Writer, Guru, Elastic, Microsoft 365 Copilot, Notion AI, Unstructured, and Airbyte, then intersecting the gaps with services that already have a Sim block. Box was the single biggest gap — 7-8 of ~10 catalogs, and the default content store in financial services, life sciences, and government. Validation Every connector was validated against live provider documentation twice, independently. The second pass was told to treat the first pass's conclusions as unproven — which mattered, because it found defects in the first pass's own fixes. Data-loss class The sync engine hard-deletes documents past a cap not flagged with listingCapped. Paths found and fixed across both passes: Mintlify — three independent listing-collapse paths. An empty discovery (both llms.txt and sitemap.xml 404, or an HTTP-200 HTML interstitial, which docs.cursor.com really serves) reconciled the whole KB away. A sitemap-index child that 404s was silently skipped despite a comment claiming failure was fatal. The origin-level index returned early with 13 of 306 pages for a sub-path site. Trello — cards in archived lists vanished from an unflagged listing; listingCapped latched permanently on any list of 1000+ cards; then the fix for that stopped flagging multi-page lists while resting on an ordering guarantee Trello does not document. Zoho Desk — the flag never fired at the default caps (500/50 and 500/100 are exact multiples); then the fix's probe walked into Zoho's hard from <= 4999 ceiling and 422'd the whole listing at the documented 5000 maximum. PagerDuty — more: true with zero incidents ended the listing unflagged. SFTP — depth-limit pruning set the flag on every sync of a deep tree, permanently suppressing legitimate deletions. Because five of ten audited connectors had some path here, the engine also gained a backstop: an empty or collapsed listing now blocks deletion reconciliation until the same observation repeats on a consecutive sync. No migration — the previous run is reconstructed from existing sync-log counters. Corroboration is only trusted from a run that could not have been incremental, otherwise a bad listing confirms itself. Correctness Zoho Desk — desk.zoho.ca does not resolve. Canada is desk.zohocloud.ca. All 11 data centers were then re-verified by DNS and an authenticated probe returning 401 UNAUTHORIZED, which proves the host serves the Desk v1 API. Singapore and UAE were missing entirely. Zoho Desk — modifiedTime is absent from GET /tickets, so the stub and hydrated hashes never matched and every ticket re-embedded on every sync, forever. Trello — Atlassian documents that some edits bump dateLastActivity without creating an action, and never published which. The hash now folds in badges counters, which arrive with the listing at no extra cost. Microsoft Excel — dates and currency indexed as raw serial numbers (42019, not 1/15/2015), unlike Google Sheets. Graph documents that Range.text is independent of cell width, so the ####### objection does not apply. A content-format token forces one re-index, since the metadata-only hash would otherwise keep stale content. PagerDuty — the 10,000-record ceiling bounds offset + limit, so the guard admitted a request that 400s after any short page. The incremental window was also only half pinned. Google Vault — one call per matter per kind truncated at ~249 matters, silently never indexed on any run. Now ~3,500. SFTP — the incremental cutoff compared against the remote clock, permanently skipping same-second writes. Box — extracted_text gave up before polling when a representation was not yet generated. Mintlify — a site whose .md route 404s hydrated every page to nothing while validateConfig passed. The HTML fallback then indexed