Title: feat(connectors): hand source files to the document pipeline instead of extracting them by waleedlatif1 · Pull Request #6821 · simstudioai/sim · GitHub
Open Graph Title: feat(connectors): hand source files to the document pipeline instead of extracting them by waleedlatif1 · Pull Request #6821 · simstudioai/sim
X Title: feat(connectors): hand source files to the document pipeline instead of extracting them by waleedlatif1 · Pull Request #6821 · simstudioai/sim
Description: Connector PDFs never reached OCR. This makes connectors hand their source files to the shared document pipeline instead of extracting text themselves, so a synced file is parsed exactly the way an upload of that same file is. Why they bypassed OCR parseDocument gates OCR on mimeType === 'application/pdf'. Connector documents were stored as text/plain, so the branch could never fire. That mimeType was not wrong — it truthfully described what was stored. The problem was what we chose to store. The connector downloaded the PDF, extracted text with the local parser, and discarded the original before the pipeline that knows about OCR ever saw it. The result was two parsing implementations, with connectors on the weaker one: upload connector (before) PDF Mistral OCR local unpdf scanned PDF readable unindexable Office formats shared parsers a second copy in connectors/utils.ts What changed ExternalDocument gains an optional sourceFile carrier (bytes + fileName + mimeType as one object, so they cannot disagree). SharePoint and OneDrive populate it for any format the knowledge base can parse; the sync engine stores those bytes under the file's own name and type. Formats that are already text stay on the existing path — HTML still reduces to plain text, everything else is a UTF-8 decode — so nothing already indexed changes representation. Only the formats that were being parsed twice move. Two details that matter for correctness: MIME is derived from the extension, not from the source's declaration, so a provider that omits or mislabels it cannot strand a PDF on the non-OCR path. Re-sync now rewrites mimeType. The update path previously never set it, so a document first stored as connector-extracted text would have kept declaring text/plain forever and never routed to OCR even after this change. This removes the duplicate path rather than leaving both: extractConnectorText is text-only now, and ConnectorTextExtractionError / extractionFailedSkipReason are deleted rather than left unused. The guard that had to move with it DocParser and PptxParser never throw. On a legacy OLE binary or a deck with no text they return a placeholder sentence or scraped archive bytes, flagged degraded. That guard lived in the connector; since parsing moved, it moves too — parseWithFileParser now treats degraded exactly like empty output and fails the document with the same actionable message, naming the modern container for legacy formats (Re-save it as DOCX). Without this the change would have regressed into indexing placeholder text. Constraints checked rather than assumed Memory: estimateOpSizeBytes already reads the true source size from listing metadata, so batching has always reserved against the real file and merely over-reserved while only text was stored. chunkOpsByByteBudget explicitly lets a single oversized file form its own chunk, so the per-file cap exceeding the in-flight budget is not a blocker. Change detection: contentHash is connector-provided and independent of representation, so nothing re-syncs spuriously. Self-hosted: OCR degrades to defaultProvider: 'local' when unconfigured, so those deployments get today's behavior; deployments that pay for OCR now actually get it for connector files. Trade-offs Storage grows — raw files instead of extracted text. OCR is an external per-PDF call (1,000-page cap, batched) that does not appear to flow through the metered model-cost path, so the spend is worth confirming before this reaches large libraries. Existing connector documents need a re-sync to convert; they are not migrated by this change. Testing vitest run connectors/ lib/knowledge/ lib/uploads/ lib/file-parsers/ app/api/knowledge/ — 1,988 passed (129 files) New unreadable-document.test.ts pins the relocated degraded guard; verified it fails without the guard (2 of 4 red) Connector tests rewritten for the new contract: a PDF is declared application/pdf so OCR can route it, an Office file is delivered as its source file, and a text file is still extracted in-connector tsgo --noEmit clean apart from two pre-existing errors on staging (mssql, @sim/testing) bun run check:audits — 29/29
Open Graph Description: Connector PDFs never reached OCR. This makes connectors hand their source files to the shared document pipeline instead of extracting text themselves, so a synced file is parsed exactly the way an ...
X Description: Connector PDFs never reached OCR. This makes connectors hand their source files to the shared document pipeline instead of extracting text themselves, so a synced file is parsed exactly the way an ...
Opengraph URL: https://github.com/simstudioai/sim/pull/6821
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:18123bbd-403a-fdb9-9e69-deee430ce9cc |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | 95DE:1BD5CA:EA7DB4:1417615:6AFB5F15 |
| html-safe-nonce | 1b4dd23e8049cbe6fff9d65c302098f5bdb5eb83c876d48cca1147abd881978e |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5NURFOjFCRDVDQTpFQTdEQjQ6MTQxNzYxNTo2QUZCNUYxNSIsInZpc2l0b3JfaWQiOiI2NjA4MzY4MDYyMTc1MDc2MTE3IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | a390a64bf03092cc1628abdf8a8d6b8c2f46192a33d56e1f5135014b6aa879ba |
| hovercard-subject-tag | pull_request:4308518740 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/6821/files |
| twitter:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| og:image:alt | Connector PDFs never reached OCR. This makes connectors hand their source files to the shared document pipeline instead of extracting text themselves, so a synced file is parsed exactly the way an ... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | ea33d13c96d0ff66bd8fa906b193d2213536b78cb06b0c50a4632c58b1418f94 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/simstudioai/sim git https://github.com/simstudioai/sim.git |
| octolytics-dimension-user_id | 199344406 |
| octolytics-dimension-user_login | simstudioai |
| octolytics-dimension-repository_id | 912559512 |
| octolytics-dimension-repository_nwo | simstudioai/sim |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 912559512 |
| octolytics-dimension-repository_network_root_nwo | simstudioai/sim |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 59ef97164c52cf59bd10558ddec9c4f482763988 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width