René's URL Explorer Experiment


Title: fix(connectors): index Office documents and PDFs from SharePoint and OneDrive by waleedlatif1 · Pull Request #6785 · simstudioai/sim · GitHub

Open Graph Title: fix(connectors): index Office documents and PDFs from SharePoint and OneDrive by waleedlatif1 · Pull Request #6785 · simstudioai/sim

X Title: fix(connectors): index Office documents and PDFs from SharePoint and OneDrive by waleedlatif1 · Pull Request #6785 · simstudioai/sim

Description: Problem A customer pointed a SharePoint connector at a document library of SOPs and got 0 documents on every sync — no error, no failed rows, status: completed. Three days were spent re-checking the site URL, the folder path, and the credential, all of which were correct. The SharePoint and OneDrive connectors filtered their listings against a 12-item plain-text extension whitelist: .txt .md .html .htm .csv .json .xml .yaml .yml .log .rst .tsv Anything else — .docx, .pdf, .xlsx, .pptx — failed isSupportedTextFile during listing and was dropped before it could become a document. The folder resolved correctly; it just had nothing the filter would accept, so the sync legitimately reported success with zero documents. That outcome is indistinguishable from a wrong folder path, which is what made it so hard to diagnose. Two things worth noting: This is not a regression. Both whitelists are byte-identical to the versions that shipped with the connectors (SharePoint in #3411, 2026-03-13) and have never been edited. Sim already parses every one of these formats. lib/file-parsers covers pdf doc docx xls xlsx ppt pptx, and the same customer had a .docx sitting in the same knowledge base, uploaded manually and processed successfully. The parsers existed; the connectors just never called them — downloadFileContent did a raw buffer.toString('utf8'). Confirming evidence from the affected workspace: the customer's working connector on the same site and credential holds 64 documents, and all 64 are .csv, .htm, or .txt. Zero Office documents across the entire library tree — exactly the shape the whitelist imposes. Change Adds three helpers to apps/sim/connectors/utils.ts, next to the existing CONNECTOR_MAX_FILE_BYTES that already aligns the connector size cap with the upload cap: CONNECTOR_INDEXABLE_EXTENSIONS / isIndexableConnectorFile(name) — the union of the text formats connectors already accepted and the binary document formats the knowledge base can parse. extractConnectorText(buffer, fileName) — UTF-8 for text formats, htmlToPlainText for HTML, and parseBuffer for binary document formats. sharepoint.ts and onedrive.ts now use those instead of their local whitelists, and their downloadFileContent returns a Buffer rather than a pre-decoded string. The change is deliberately additive. CONNECTOR_TEXT_EXTENSIONS and CONNECTOR_PARSED_EXTENSIONS are kept as separate lists precisely so a format that synced yesterday takes the identical path today. Routing .csv through CsvParser or .json through the JSON parser would reformat content that is already embedded, changing every existing connector document on its next re-index — so those keep bypassing the parsers. Only the newly-accepted binary formats reach parseBuffer, which is also where the OOXML zip-bomb guard and each parser's own extraction limits (MAX_PDF_TEXT_CHARS, the 60s PDF deadline) apply. Also adds a per-page logger.info counting files skipped for an unsupported extension, with the distinct extensions seen. Unsupported files are counted, not turned into failed document rows — a library full of images would otherwise flood the knowledge base UI. The goal is only to make "0 documents" explicable in the run trace. Tests apps/sim/connectors/utils.test.ts: Office/PDF formats accepted; previously-supported text formats still accepted; .png/.mp4/.zip still rejected; extensionless and trailing-dot names rejected; case-insensitive .docx routes through parseBuffer with the right extension regression guard: .txt/.csv/.yaml/.tsv/.xml decode as UTF-8 and never invoke a parser HTML reduces to plain text without a parser; unknown extension falls back to UTF-8; a parser failure propagates so the sync records a failed document apps/sim/connectors/sharepoint/sharepoint.test.ts: an end-to-end listing containing .docx/.pdf/.xlsx/.pptx/.txt returns all five (this is the reported bug) a listing of .png/.mp4/.txt still returns only the .txt Verified the new tests can fail: narrowing CONNECTOR_INDEXABLE_EXTENSIONS back to the text-only set turns exactly 3 of them red. vitest run connectors/ — 695 passed (25 files) tsgo --noEmit -p apps/sim/tsconfig.json — clean apart from a pre-existing unrelated Cannot find module 'mssql' bun run check:audits — 29/29 pass Follow-ups not in this PR PDF text bombs. Enabling .pdf in connectors puts the sync path behind the same PDF extraction concern tracked separately for the upload path. The parser's own char/time caps apply and the size cap already matches manual upload, so this does not widen the limit — but it does add a second entry point worth resolving there. Other connectors. box, dropbox, s3, sftp, and google-drive have their own file-type handling that was not touched here. If they carry the same gap, they can adopt extractConnectorText in a follow-up.

Open Graph Description: Problem A customer pointed a SharePoint connector at a document library of SOPs and got 0 documents on every sync — no error, no failed rows, status: completed. Three days were spent re-checking th...

X Description: Problem A customer pointed a SharePoint connector at a document library of SOPs and got 0 documents on every sync — no error, no failed rows, status: completed. Three days were spent re-checking th...

Opengraph URL: https://github.com/simstudioai/sim/pull/6785

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:7fcaed97-5a6f-ef92-b79f-3adfd0ef9bc3
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idAFD8:19FA2E:93625F:D0F314:6AFB5153
html-safe-noncec01886a16464ef49e54ac81a6cd1f07a207b52ea7587078fd9e474401370ce67
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBRkQ4OjE5RkEyRTo5MzYyNUY6RDBGMzE0OjZBRkI1MTUzIiwidmlzaXRvcl9pZCI6IjcwNjM2ODA5ODk3ODc4MDM5ODciLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmace5f1f48c246202fb1a2bba90ed043f5730152b3674a2a43ce6f4283ede92d4c9
hovercard-subject-tagpull_request:4299397761
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/6785/files
twitter:imagehttps://avatars.githubusercontent.com/u/40672544?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/40672544?s=400&v=4
og:image:altProblem A customer pointed a SharePoint connector at a document library of SOPs and got 0 documents on every sync — no error, no failed rows, status: completed. Three days were spent re-checking th...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None7122bb414b542dba14d71a3f0b9f6fd2abce715b9f5912d082800563e6c806d1
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/simstudioai/sim git https://github.com/simstudioai/sim.git
octolytics-dimension-user_id199344406
octolytics-dimension-user_loginsimstudioai
octolytics-dimension-repository_id912559512
octolytics-dimension-repository_nwosimstudioai/sim
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id912559512
octolytics-dimension-repository_network_root_nwosimstudioai/sim
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
releasecbf66c024bb6d10e6955baa7413c54cf13a201bb
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/simstudioai/sim/pull/6785/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F6785%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F6785%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=simstudioai%2Fsim
Reloadhttps://github.com/simstudioai/sim/pull/6785/files
Reloadhttps://github.com/simstudioai/sim/pull/6785/files
Reloadhttps://github.com/simstudioai/sim/pull/6785/files
simstudioaihttps://github.com/simstudioai
simhttps://github.com/simstudioai/sim
Codehttps://github.com/simstudioai/sim
Issues129 (129)https://github.com/simstudioai/sim/issues
Pull requests197 (197)https://github.com/simstudioai/sim/pulls
Actionshttps://github.com/simstudioai/sim/actions
Projectshttps://github.com/simstudioai/sim/projects
Security and qualityhttps://github.com/simstudioai/sim/security
Insightshttps://github.com/simstudioai/sim/pulse
Sign up for GitHub https://github.com/signup?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
waleedlatif1https://github.com/waleedlatif1
staginghttps://github.com/simstudioai/sim/tree/staging
fix/connector-office-document-parsinghttps://github.com/simstudioai/sim/tree/fix/connector-office-document-parsing
Conversation 12 https://github.com/simstudioai/sim/pull/6785
Commits 4 https://github.com/simstudioai/sim/pull/6785/commits
Checks 30 https://github.com/simstudioai/sim/pull/6785/checks
Files changed https://github.com/simstudioai/sim/pull/6785/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/6785/files
fix(connectors): index Office documents and PDFs from SharePoint and OneDrive https://github.com/simstudioai/sim/pull/6785/files#top
Show all changes 4 commits https://github.com/simstudioai/sim/pull/6785/files
f03f62b fix(connectors): index Office documents and PDFs from SharePoint and … waleedlatif1 Aug 17, 2026 https://github.com/simstudioai/sim/pull/6785/commits/f03f62bcbe0868747dcdf76fec4c5f863dc3a474
dd083bf fix(connectors): never index a degraded document extraction waleedlatif1 Aug 17, 2026 https://github.com/simstudioai/sim/pull/6785/commits/dd083bf6f791813f7d29d72ffc7fdab004fe907a
0cbab90 fix(file-parsers): register the document variants the parsers already… waleedlatif1 Aug 17, 2026 https://github.com/simstudioai/sim/pull/6785/commits/0cbab9050750dfa7e4bbc6f6bc9766ab8e4cacc4
1bb1b2b fix(file-parsers): resolve the parser registry through a Map, not obj… waleedlatif1 Aug 17, 2026 https://github.com/simstudioai/sim/pull/6785/commits/1bb1b2b543c823fb371e4bcf136747b8b5aa9b9c
Clear filters https://github.com/simstudioai/sim/pull/6785/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/6785/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/6785/files
onedrive.ts https://github.com/simstudioai/sim/pull/6785/files#diff-60e130e26008d9eb6e671ee3195a984336ebd186565dced245cd820268143da1
sharepoint.test.ts https://github.com/simstudioai/sim/pull/6785/files#diff-fd83e8cefcb2e1781421448d77ccd616be9a128f6a135da4784ea2e3bf2e89f7
sharepoint.ts https://github.com/simstudioai/sim/pull/6785/files#diff-a39a21bb15e6c27b68de3dc7899e31ba8e4ee2fa37e7c34ce1c93a23ab25c625
utils.test.ts https://github.com/simstudioai/sim/pull/6785/files#diff-2405bfb9d79e30136f8e29677391cd1f1164759188077003e27518634e26bbe9
utils.ts https://github.com/simstudioai/sim/pull/6785/files#diff-c27db14947d25b8915e236c873ccf485a2e0497266dde4f61bd295f93e7f3920
doc-parser.ts https://github.com/simstudioai/sim/pull/6785/files#diff-cdfc927b24e9d2666bd93713a077b86bbded2c33e752f1a6f298e0fcf5c48032
index.ts https://github.com/simstudioai/sim/pull/6785/files#diff-5340b2eaee20e0208a9b187e5946c1b1bf273be94c0e2da72d79adedc2a399f7
opendocument-parser.ts https://github.com/simstudioai/sim/pull/6785/files#diff-468a9173aaa19c962f8bed8feddc9d6a035abc94abea3958cd47cda7486fdc68
parser-formats.test.ts https://github.com/simstudioai/sim/pull/6785/files#diff-ea445cd678ea9a7c3170a19e43213ab8822e71b49f85c733cc0520b01ef37658
pptx-parser.ts https://github.com/simstudioai/sim/pull/6785/files#diff-c950c4de7c124d97ac12dad2b9ce3813fde4e0f18100a4ca7a6ab3d42b53d71a
registry.test.ts https://github.com/simstudioai/sim/pull/6785/files#diff-c92a9f9ac215ebc4b49afa907f1dfc3a0dada1745f4ab80560bd50fc4f7a90e7
types.ts https://github.com/simstudioai/sim/pull/6785/files#diff-65f59f17af217bd0758197a935b918001153deab15a0fcdf21618a6ca1c7633d
apps/sim/connectors/onedrive/onedrive.tshttps://github.com/simstudioai/sim/pull/6785/files#diff-60e130e26008d9eb6e671ee3195a984336ebd186565dced245cd820268143da1
View file https://github.com/simstudioai/sim/blob/1bb1b2b543c823fb371e4bcf136747b8b5aa9b9c/apps/sim/connectors/onedrive/onedrive.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/6785/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/6785/files#diff-60e130e26008d9eb6e671ee3195a984336ebd186565dced245cd820268143da1
https://github.com/simstudioai/sim/pull/6785/files#diff-60e130e26008d9eb6e671ee3195a984336ebd186565dced245cd820268143da1
https://github.com/simstudioai/sim/pull/6785/files#diff-60e130e26008d9eb6e671ee3195a984336ebd186565dced245cd820268143da1
https://github.com/simstudioai/sim/pull/6785/files#diff-60e130e26008d9eb6e671ee3195a984336ebd186565dced245cd820268143da1
https://github.com/simstudioai/sim/pull/6785/files#diff-60e130e26008d9eb6e671ee3195a984336ebd186565dced245cd820268143da1
https://github.com/simstudioai/sim/pull/6785/files#diff-60e130e26008d9eb6e671ee3195a984336ebd186565dced245cd820268143da1
https://github.com/simstudioai/sim/pull/6785/files#diff-60e130e26008d9eb6e671ee3195a984336ebd186565dced245cd820268143da1
https://github.com/simstudioai/sim/pull/6785/files#diff-60e130e26008d9eb6e671ee3195a984336ebd186565dced245cd820268143da1
https://github.com/simstudioai/sim/pull/6785/files#diff-60e130e26008d9eb6e671ee3195a984336ebd186565dced245cd820268143da1
https://github.com/simstudioai/sim/pull/6785/files#diff-60e130e26008d9eb6e671ee3195a984336ebd186565dced245cd820268143da1
https://github.com/simstudioai/sim/pull/6785/files#diff-60e130e26008d9eb6e671ee3195a984336ebd186565dced245cd820268143da1
apps/sim/connectors/sharepoint/sharepoint.test.tshttps://github.com/simstudioai/sim/pull/6785/files#diff-fd83e8cefcb2e1781421448d77ccd616be9a128f6a135da4784ea2e3bf2e89f7
View file https://github.com/simstudioai/sim/blob/1bb1b2b543c823fb371e4bcf136747b8b5aa9b9c/apps/sim/connectors/sharepoint/sharepoint.test.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/6785/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/6785/files#diff-fd83e8cefcb2e1781421448d77ccd616be9a128f6a135da4784ea2e3bf2e89f7
https://github.com/simstudioai/sim/pull/6785/files#diff-fd83e8cefcb2e1781421448d77ccd616be9a128f6a135da4784ea2e3bf2e89f7
https://github.com/simstudioai/sim/pull/6785/files#diff-fd83e8cefcb2e1781421448d77ccd616be9a128f6a135da4784ea2e3bf2e89f7
https://github.com/simstudioai/sim/pull/6785/files#diff-fd83e8cefcb2e1781421448d77ccd616be9a128f6a135da4784ea2e3bf2e89f7
https://github.com/simstudioai/sim/pull/6785/files#diff-fd83e8cefcb2e1781421448d77ccd616be9a128f6a135da4784ea2e3bf2e89f7
https://github.com/simstudioai/sim/pull/6785/files#diff-fd83e8cefcb2e1781421448d77ccd616be9a128f6a135da4784ea2e3bf2e89f7
https://github.com/simstudioai/sim/pull/6785/files#diff-fd83e8cefcb2e1781421448d77ccd616be9a128f6a135da4784ea2e3bf2e89f7
Please reload this pagehttps://github.com/simstudioai/sim/pull/6785/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/6785/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.