Title: fix(connectors): index Office documents and PDFs from SharePoint and OneDrive by waleedlatif1 · Pull Request #6785 · simstudioai/sim · GitHub
Open Graph Title: fix(connectors): index Office documents and PDFs from SharePoint and OneDrive by waleedlatif1 · Pull Request #6785 · simstudioai/sim
X Title: fix(connectors): index Office documents and PDFs from SharePoint and OneDrive by waleedlatif1 · Pull Request #6785 · simstudioai/sim
Description: Problem A customer pointed a SharePoint connector at a document library of SOPs and got 0 documents on every sync — no error, no failed rows, status: completed. Three days were spent re-checking the site URL, the folder path, and the credential, all of which were correct. The SharePoint and OneDrive connectors filtered their listings against a 12-item plain-text extension whitelist: .txt .md .html .htm .csv .json .xml .yaml .yml .log .rst .tsv Anything else — .docx, .pdf, .xlsx, .pptx — failed isSupportedTextFile during listing and was dropped before it could become a document. The folder resolved correctly; it just had nothing the filter would accept, so the sync legitimately reported success with zero documents. That outcome is indistinguishable from a wrong folder path, which is what made it so hard to diagnose. Two things worth noting: This is not a regression. Both whitelists are byte-identical to the versions that shipped with the connectors (SharePoint in #3411, 2026-03-13) and have never been edited. Sim already parses every one of these formats. lib/file-parsers covers pdf doc docx xls xlsx ppt pptx, and the same customer had a .docx sitting in the same knowledge base, uploaded manually and processed successfully. The parsers existed; the connectors just never called them — downloadFileContent did a raw buffer.toString('utf8'). Confirming evidence from the affected workspace: the customer's working connector on the same site and credential holds 64 documents, and all 64 are .csv, .htm, or .txt. Zero Office documents across the entire library tree — exactly the shape the whitelist imposes. Change Adds three helpers to apps/sim/connectors/utils.ts, next to the existing CONNECTOR_MAX_FILE_BYTES that already aligns the connector size cap with the upload cap: CONNECTOR_INDEXABLE_EXTENSIONS / isIndexableConnectorFile(name) — the union of the text formats connectors already accepted and the binary document formats the knowledge base can parse. extractConnectorText(buffer, fileName) — UTF-8 for text formats, htmlToPlainText for HTML, and parseBuffer for binary document formats. sharepoint.ts and onedrive.ts now use those instead of their local whitelists, and their downloadFileContent returns a Buffer rather than a pre-decoded string. The change is deliberately additive. CONNECTOR_TEXT_EXTENSIONS and CONNECTOR_PARSED_EXTENSIONS are kept as separate lists precisely so a format that synced yesterday takes the identical path today. Routing .csv through CsvParser or .json through the JSON parser would reformat content that is already embedded, changing every existing connector document on its next re-index — so those keep bypassing the parsers. Only the newly-accepted binary formats reach parseBuffer, which is also where the OOXML zip-bomb guard and each parser's own extraction limits (MAX_PDF_TEXT_CHARS, the 60s PDF deadline) apply. Also adds a per-page logger.info counting files skipped for an unsupported extension, with the distinct extensions seen. Unsupported files are counted, not turned into failed document rows — a library full of images would otherwise flood the knowledge base UI. The goal is only to make "0 documents" explicable in the run trace. Tests apps/sim/connectors/utils.test.ts: Office/PDF formats accepted; previously-supported text formats still accepted; .png/.mp4/.zip still rejected; extensionless and trailing-dot names rejected; case-insensitive .docx routes through parseBuffer with the right extension regression guard: .txt/.csv/.yaml/.tsv/.xml decode as UTF-8 and never invoke a parser HTML reduces to plain text without a parser; unknown extension falls back to UTF-8; a parser failure propagates so the sync records a failed document apps/sim/connectors/sharepoint/sharepoint.test.ts: an end-to-end listing containing .docx/.pdf/.xlsx/.pptx/.txt returns all five (this is the reported bug) a listing of .png/.mp4/.txt still returns only the .txt Verified the new tests can fail: narrowing CONNECTOR_INDEXABLE_EXTENSIONS back to the text-only set turns exactly 3 of them red. vitest run connectors/ — 695 passed (25 files) tsgo --noEmit -p apps/sim/tsconfig.json — clean apart from a pre-existing unrelated Cannot find module 'mssql' bun run check:audits — 29/29 pass Follow-ups not in this PR PDF text bombs. Enabling .pdf in connectors puts the sync path behind the same PDF extraction concern tracked separately for the upload path. The parser's own char/time caps apply and the size cap already matches manual upload, so this does not widen the limit — but it does add a second entry point worth resolving there. Other connectors. box, dropbox, s3, sftp, and google-drive have their own file-type handling that was not touched here. If they carry the same gap, they can adopt extractConnectorText in a follow-up.
Open Graph Description: Problem A customer pointed a SharePoint connector at a document library of SOPs and got 0 documents on every sync — no error, no failed rows, status: completed. Three days were spent re-checking th...
X Description: Problem A customer pointed a SharePoint connector at a document library of SOPs and got 0 documents on every sync — no error, no failed rows, status: completed. Three days were spent re-checking th...
Opengraph URL: https://github.com/simstudioai/sim/pull/6785
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:7fcaed97-5a6f-ef92-b79f-3adfd0ef9bc3 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | AFD8:19FA2E:93625F:D0F314:6AFB5153 |
| html-safe-nonce | c01886a16464ef49e54ac81a6cd1f07a207b52ea7587078fd9e474401370ce67 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBRkQ4OjE5RkEyRTo5MzYyNUY6RDBGMzE0OjZBRkI1MTUzIiwidmlzaXRvcl9pZCI6IjcwNjM2ODA5ODk3ODc4MDM5ODciLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | e5f1f48c246202fb1a2bba90ed043f5730152b3674a2a43ce6f4283ede92d4c9 |
| hovercard-subject-tag | pull_request:4299397761 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/6785/files |
| twitter:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| og:image:alt | Problem A customer pointed a SharePoint connector at a document library of SOPs and got 0 documents on every sync — no error, no failed rows, status: completed. Three days were spent re-checking th... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 7122bb414b542dba14d71a3f0b9f6fd2abce715b9f5912d082800563e6c806d1 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/simstudioai/sim git https://github.com/simstudioai/sim.git |
| octolytics-dimension-user_id | 199344406 |
| octolytics-dimension-user_login | simstudioai |
| octolytics-dimension-repository_id | 912559512 |
| octolytics-dimension-repository_nwo | simstudioai/sim |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 912559512 |
| octolytics-dimension-repository_network_root_nwo | simstudioai/sim |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | cbf66c024bb6d10e6955baa7413c54cf13a201bb |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width