René's URL Explorer Experiment


Title: feat(connectors): hand source files to the document pipeline instead of extracting them by waleedlatif1 · Pull Request #6821 · simstudioai/sim · GitHub

Open Graph Title: feat(connectors): hand source files to the document pipeline instead of extracting them by waleedlatif1 · Pull Request #6821 · simstudioai/sim

X Title: feat(connectors): hand source files to the document pipeline instead of extracting them by waleedlatif1 · Pull Request #6821 · simstudioai/sim

Description: Connector PDFs never reached OCR. This makes connectors hand their source files to the shared document pipeline instead of extracting text themselves, so a synced file is parsed exactly the way an upload of that same file is. Why they bypassed OCR parseDocument gates OCR on mimeType === 'application/pdf'. Connector documents were stored as text/plain, so the branch could never fire. That mimeType was not wrong — it truthfully described what was stored. The problem was what we chose to store. The connector downloaded the PDF, extracted text with the local parser, and discarded the original before the pipeline that knows about OCR ever saw it. The result was two parsing implementations, with connectors on the weaker one: upload connector (before) PDF Mistral OCR local unpdf scanned PDF readable unindexable Office formats shared parsers a second copy in connectors/utils.ts What changed ExternalDocument gains an optional sourceFile carrier (bytes + fileName + mimeType as one object, so they cannot disagree). SharePoint and OneDrive populate it for any format the knowledge base can parse; the sync engine stores those bytes under the file's own name and type. Formats that are already text stay on the existing path — HTML still reduces to plain text, everything else is a UTF-8 decode — so nothing already indexed changes representation. Only the formats that were being parsed twice move. Two details that matter for correctness: MIME is derived from the extension, not from the source's declaration, so a provider that omits or mislabels it cannot strand a PDF on the non-OCR path. Re-sync now rewrites mimeType. The update path previously never set it, so a document first stored as connector-extracted text would have kept declaring text/plain forever and never routed to OCR even after this change. This removes the duplicate path rather than leaving both: extractConnectorText is text-only now, and ConnectorTextExtractionError / extractionFailedSkipReason are deleted rather than left unused. The guard that had to move with it DocParser and PptxParser never throw. On a legacy OLE binary or a deck with no text they return a placeholder sentence or scraped archive bytes, flagged degraded. That guard lived in the connector; since parsing moved, it moves too — parseWithFileParser now treats degraded exactly like empty output and fails the document with the same actionable message, naming the modern container for legacy formats (Re-save it as DOCX). Without this the change would have regressed into indexing placeholder text. Constraints checked rather than assumed Memory: estimateOpSizeBytes already reads the true source size from listing metadata, so batching has always reserved against the real file and merely over-reserved while only text was stored. chunkOpsByByteBudget explicitly lets a single oversized file form its own chunk, so the per-file cap exceeding the in-flight budget is not a blocker. Change detection: contentHash is connector-provided and independent of representation, so nothing re-syncs spuriously. Self-hosted: OCR degrades to defaultProvider: 'local' when unconfigured, so those deployments get today's behavior; deployments that pay for OCR now actually get it for connector files. Trade-offs Storage grows — raw files instead of extracted text. OCR is an external per-PDF call (1,000-page cap, batched) that does not appear to flow through the metered model-cost path, so the spend is worth confirming before this reaches large libraries. Existing connector documents need a re-sync to convert; they are not migrated by this change. Testing vitest run connectors/ lib/knowledge/ lib/uploads/ lib/file-parsers/ app/api/knowledge/ — 1,988 passed (129 files) New unreadable-document.test.ts pins the relocated degraded guard; verified it fails without the guard (2 of 4 red) Connector tests rewritten for the new contract: a PDF is declared application/pdf so OCR can route it, an Office file is delivered as its source file, and a text file is still extracted in-connector tsgo --noEmit clean apart from two pre-existing errors on staging (mssql, @sim/testing) bun run check:audits — 29/29

Open Graph Description: Connector PDFs never reached OCR. This makes connectors hand their source files to the shared document pipeline instead of extracting text themselves, so a synced file is parsed exactly the way an ...

X Description: Connector PDFs never reached OCR. This makes connectors hand their source files to the shared document pipeline instead of extracting text themselves, so a synced file is parsed exactly the way an ...

Opengraph URL: https://github.com/simstudioai/sim/pull/6821

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:18123bbd-403a-fdb9-9e69-deee430ce9cc
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-id95DE:1BD5CA:EA7DB4:1417615:6AFB5F15
html-safe-nonce1b4dd23e8049cbe6fff9d65c302098f5bdb5eb83c876d48cca1147abd881978e
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5NURFOjFCRDVDQTpFQTdEQjQ6MTQxNzYxNTo2QUZCNUYxNSIsInZpc2l0b3JfaWQiOiI2NjA4MzY4MDYyMTc1MDc2MTE3IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0=
visitor-hmaca390a64bf03092cc1628abdf8a8d6b8c2f46192a33d56e1f5135014b6aa879ba
hovercard-subject-tagpull_request:4308518740
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/6821/files
twitter:imagehttps://avatars.githubusercontent.com/u/40672544?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/40672544?s=400&v=4
og:image:altConnector PDFs never reached OCR. This makes connectors hand their source files to the shared document pipeline instead of extracting text themselves, so a synced file is parsed exactly the way an ...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
Noneea33d13c96d0ff66bd8fa906b193d2213536b78cb06b0c50a4632c58b1418f94
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/simstudioai/sim git https://github.com/simstudioai/sim.git
octolytics-dimension-user_id199344406
octolytics-dimension-user_loginsimstudioai
octolytics-dimension-repository_id912559512
octolytics-dimension-repository_nwosimstudioai/sim
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id912559512
octolytics-dimension-repository_network_root_nwosimstudioai/sim
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release59ef97164c52cf59bd10558ddec9c4f482763988
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/simstudioai/sim/pull/6821/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F6821%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F6821%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=simstudioai%2Fsim
Reloadhttps://github.com/simstudioai/sim/pull/6821/files
Reloadhttps://github.com/simstudioai/sim/pull/6821/files
Reloadhttps://github.com/simstudioai/sim/pull/6821/files
simstudioaihttps://github.com/simstudioai
simhttps://github.com/simstudioai/sim
Codehttps://github.com/simstudioai/sim
Issues129 (129)https://github.com/simstudioai/sim/issues
Pull requests197 (197)https://github.com/simstudioai/sim/pulls
Actionshttps://github.com/simstudioai/sim/actions
Projectshttps://github.com/simstudioai/sim/projects
Security and qualityhttps://github.com/simstudioai/sim/security
Insightshttps://github.com/simstudioai/sim/pulse
Sign up for GitHub https://github.com/signup?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
waleedlatif1https://github.com/waleedlatif1
staginghttps://github.com/simstudioai/sim/tree/staging
feat/connector-raw-source-fileshttps://github.com/simstudioai/sim/tree/feat/connector-raw-source-files
Conversation 14 https://github.com/simstudioai/sim/pull/6821
Commits 4 https://github.com/simstudioai/sim/pull/6821/commits
Checks 30 https://github.com/simstudioai/sim/pull/6821/checks
Files changed https://github.com/simstudioai/sim/pull/6821/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/6821/files
feat(connectors): hand source files to the document pipeline instead of extracting them https://github.com/simstudioai/sim/pull/6821/files#top
Show all changes 4 commits https://github.com/simstudioai/sim/pull/6821/files
a7dad95 feat(connectors): hand source files to the document pipeline instead … waleedlatif1 Aug 18, 2026 https://github.com/simstudioai/sim/pull/6821/commits/a7dad9526f998c0610d0531130a4dea859f9f315
22b441e fix(knowledge): guard every parser against empty output, not just the… waleedlatif1 Aug 18, 2026 https://github.com/simstudioai/sim/pull/6821/commits/22b441e4656218133dff02d1dac78e3967bec00a
bdb9979 fix(knowledge): validate the stored artifact against the parser registry waleedlatif1 Aug 18, 2026 https://github.com/simstudioai/sim/pull/6821/commits/bdb9979ee532951690c83aca46df515107cf5b00
d3dfab0 fix(knowledge): carry the MIME type through hydration waleedlatif1 Aug 18, 2026 https://github.com/simstudioai/sim/pull/6821/commits/d3dfab063e5a47306744efc9cec21b57a24b1f64
Clear filters https://github.com/simstudioai/sim/pull/6821/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/6821/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/6821/files
onedrive.ts https://github.com/simstudioai/sim/pull/6821/files#diff-60e130e26008d9eb6e671ee3195a984336ebd186565dced245cd820268143da1
sharepoint.test.ts https://github.com/simstudioai/sim/pull/6821/files#diff-fd83e8cefcb2e1781421448d77ccd616be9a128f6a135da4784ea2e3bf2e89f7
sharepoint.ts https://github.com/simstudioai/sim/pull/6821/files#diff-a39a21bb15e6c27b68de3dc7899e31ba8e4ee2fa37e7c34ce1c93a23ab25c625
types.ts https://github.com/simstudioai/sim/pull/6821/files#diff-f2cc63aee0c2986b6b2d30205f9f261e33b282d0cd330baf9f8bb0a1cd307178
utils.test.ts https://github.com/simstudioai/sim/pull/6821/files#diff-2405bfb9d79e30136f8e29677391cd1f1164759188077003e27518634e26bbe9
utils.ts https://github.com/simstudioai/sim/pull/6821/files#diff-c27db14947d25b8915e236c873ccf485a2e0497266dde4f61bd295f93e7f3920
sync-engine.test.ts https://github.com/simstudioai/sim/pull/6821/files#diff-021afebf7e100833ecdc940e307263436964ce85ef3dfd40fc90307a1d115534
sync-engine.ts https://github.com/simstudioai/sim/pull/6821/files#diff-277f76464f02682e44e6de58ebd215c927f0f6a3a7165af669a0717b593790e9
document-processor-secret-provenance.test.ts https://github.com/simstudioai/sim/pull/6821/files#diff-0c29ef408fce32f11d0455ec151b9dc1646f04cbe2351a3863fbf52def2b40d9
document-processor.ts https://github.com/simstudioai/sim/pull/6821/files#diff-caef18c7f5680f8ad0ec1e40b90e7635fe8f182b55d4fd20ef78fbe881f11205
parser-extension.ts https://github.com/simstudioai/sim/pull/6821/files#diff-e9fe609c97ac485c94aed161a03a69764fb5ebf7f0c7928e5fc2910f9bd5ac99
stored-artifact-extension.test.ts https://github.com/simstudioai/sim/pull/6821/files#diff-0c3fc04759a119d0e72255e5752d417f5903716f2ddc443324f95f5e19e9a9f6
unreadable-document.test.ts https://github.com/simstudioai/sim/pull/6821/files#diff-6904abbc735321d6460e87e0f3504d139a7c23626c23cc1b2e5ba684c4855d42
apps/sim/connectors/onedrive/onedrive.tshttps://github.com/simstudioai/sim/pull/6821/files#diff-60e130e26008d9eb6e671ee3195a984336ebd186565dced245cd820268143da1
View file https://github.com/simstudioai/sim/blob/d3dfab063e5a47306744efc9cec21b57a24b1f64/apps/sim/connectors/onedrive/onedrive.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/6821/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/6821/files#diff-60e130e26008d9eb6e671ee3195a984336ebd186565dced245cd820268143da1
https://github.com/simstudioai/sim/pull/6821/files#diff-60e130e26008d9eb6e671ee3195a984336ebd186565dced245cd820268143da1
https://github.com/simstudioai/sim/pull/6821/files#diff-60e130e26008d9eb6e671ee3195a984336ebd186565dced245cd820268143da1
https://github.com/simstudioai/sim/pull/6821/files#diff-60e130e26008d9eb6e671ee3195a984336ebd186565dced245cd820268143da1
https://github.com/simstudioai/sim/pull/6821/files#diff-60e130e26008d9eb6e671ee3195a984336ebd186565dced245cd820268143da1
https://github.com/simstudioai/sim/pull/6821/files#diff-60e130e26008d9eb6e671ee3195a984336ebd186565dced245cd820268143da1
apps/sim/connectors/sharepoint/sharepoint.test.tshttps://github.com/simstudioai/sim/pull/6821/files#diff-fd83e8cefcb2e1781421448d77ccd616be9a128f6a135da4784ea2e3bf2e89f7
View file https://github.com/simstudioai/sim/blob/d3dfab063e5a47306744efc9cec21b57a24b1f64/apps/sim/connectors/sharepoint/sharepoint.test.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/6821/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/6821/files#diff-fd83e8cefcb2e1781421448d77ccd616be9a128f6a135da4784ea2e3bf2e89f7
https://github.com/simstudioai/sim/pull/6821/files#diff-fd83e8cefcb2e1781421448d77ccd616be9a128f6a135da4784ea2e3bf2e89f7
https://github.com/simstudioai/sim/pull/6821/files#diff-fd83e8cefcb2e1781421448d77ccd616be9a128f6a135da4784ea2e3bf2e89f7
https://github.com/simstudioai/sim/pull/6821/files#diff-fd83e8cefcb2e1781421448d77ccd616be9a128f6a135da4784ea2e3bf2e89f7
apps/sim/connectors/sharepoint/sharepoint.tshttps://github.com/simstudioai/sim/pull/6821/files#diff-a39a21bb15e6c27b68de3dc7899e31ba8e4ee2fa37e7c34ce1c93a23ab25c625
View file https://github.com/simstudioai/sim/blob/d3dfab063e5a47306744efc9cec21b57a24b1f64/apps/sim/connectors/sharepoint/sharepoint.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/6821/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/6821/files#diff-a39a21bb15e6c27b68de3dc7899e31ba8e4ee2fa37e7c34ce1c93a23ab25c625
https://github.com/simstudioai/sim/pull/6821/files#diff-a39a21bb15e6c27b68de3dc7899e31ba8e4ee2fa37e7c34ce1c93a23ab25c625
https://github.com/simstudioai/sim/pull/6821/files#diff-a39a21bb15e6c27b68de3dc7899e31ba8e4ee2fa37e7c34ce1c93a23ab25c625
https://github.com/simstudioai/sim/pull/6821/files#diff-a39a21bb15e6c27b68de3dc7899e31ba8e4ee2fa37e7c34ce1c93a23ab25c625
https://github.com/simstudioai/sim/pull/6821/files#diff-a39a21bb15e6c27b68de3dc7899e31ba8e4ee2fa37e7c34ce1c93a23ab25c625
https://github.com/simstudioai/sim/pull/6821/files#diff-a39a21bb15e6c27b68de3dc7899e31ba8e4ee2fa37e7c34ce1c93a23ab25c625
https://github.com/simstudioai/sim/pull/6821/files#diff-a39a21bb15e6c27b68de3dc7899e31ba8e4ee2fa37e7c34ce1c93a23ab25c625
apps/sim/connectors/types.tshttps://github.com/simstudioai/sim/pull/6821/files#diff-f2cc63aee0c2986b6b2d30205f9f261e33b282d0cd330baf9f8bb0a1cd307178
View file https://github.com/simstudioai/sim/blob/d3dfab063e5a47306744efc9cec21b57a24b1f64/apps/sim/connectors/types.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/6821/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/6821/files#diff-f2cc63aee0c2986b6b2d30205f9f261e33b282d0cd330baf9f8bb0a1cd307178
https://github.com/simstudioai/sim/pull/6821/files#diff-f2cc63aee0c2986b6b2d30205f9f261e33b282d0cd330baf9f8bb0a1cd307178
Please reload this pagehttps://github.com/simstudioai/sim/pull/6821/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/6821/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.