Title: perf(files): stream workspace archives instead of buffering them by waleedlatif1 · Pull Request #5993 · simstudioai/sim · GitHub
Open Graph Title: perf(files): stream workspace archives instead of buffering them by waleedlatif1 · Pull Request #5993 · simstudioai/sim
X Title: perf(files): stream workspace archives instead of buffering them by waleedlatif1 · Pull Request #5993 · simstudioai/sim
Description: Stacked on #5986 — review that first. Base retargets to staging once it merges. Summary The zip route materialized every selected file before writing a byte, so peak memory tracked the size of the selection. Ordinary files are now appended as lazy streams — each opens its storage read only when the archiver reaches it, so one entry is resident at a time instead of the whole archive Generated documents still resolve to buffers first. They are the only entries whose bytes decide anything, and every status this route returns comes from them — once the first byte is written the status code is committed, so those decisions have to happen before the archive starts This removes the memory objection on #5986 at its root rather than tuning around it: the per-entry allowance and byte budget now govern documents only, which are small by nature Why these dependencies archiver (35.6M weekly downloads, 8.0.0 released this May) — its entry queue processes appends sequentially. Raw zip-stream, which archiver wraps, has no queue management lazystream (31.3M weekly) — defers each storage read until the archiver pulls it. Already a direct dependency of archiver, so declaring it just makes an import we use explicit. Handing the archiver one open stream per entry would hold more connections than the storage client pools, most sitting idle until their turn Reuse nodeReadableToWebStream moves out of input-validation.server.ts into lib/core/utils/node-stream.ts and is shared. Its existing comment is the reason it exists: Readable.toWeb throws an unhandled ERR_INVALID_STATE when a consumer cancels while the source is still flowing — exactly what a cancelled download does. It also copies chunks off pooled buffers, honours backpressure, and settles a destroy that never emits error. Trade worth reviewing A storage read that fails mid-archive now truncates the response instead of returning 500, because the status is already sent. Documents cannot hit this (they resolve up front); it applies to ordinary files. app/api/table/[tableId]/export already behaves this way. Type of Change Performance Testing Ten route tests, including one asserting that no storage read happens until the archive is pulled — the property the whole change rests on — plus nested paths across both entry kinds, the pending-artifact 409, per-entry vs shared-budget size attribution, and the declared-size pre-check rejecting before any read. 1112 tests pass across lib/uploads, lib/core/security, lib/core/utils, and the touched routes. Lint, check:utils, typecheck and check:api-validation clean. Not live-tested in a browser yet. Checklist Code follows project style guidelines Self-reviewed my changes Tests added/updated and passing No new warnings introduced I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)
Open Graph Description: Stacked on #5986 — review that first. Base retargets to staging once it merges. Summary The zip route materialized every selected file before writing a byte, so peak memory tracked the size of the...
X Description: Stacked on #5986 — review that first. Base retargets to staging once it merges. Summary The zip route materialized every selected file before writing a byte, so peak memory tracked the size of the...
Opengraph URL: https://github.com/simstudioai/sim/pull/5993
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:08fe3398-92b9-e7e4-1f0f-0a1dd379d75f |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | D850:5A71D:B63763:F96627:6AFB517D |
| html-safe-nonce | 56f448a71074804ade94f54565e55cf54624dd6c7d64eb96521f86cf3299765b |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJEODUwOjVBNzFEOkI2Mzc2MzpGOTY2Mjc6NkFGQjUxN0QiLCJ2aXNpdG9yX2lkIjoiMzkyNTIxNzIzMTE4NDI4NjA3NyIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | d7b3d4a404add01fef552932057f996a620c44f2ef9b6cc299ed82a6885d241c |
| hovercard-subject-tag | pull_request:4147127123 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/5993/files |
| twitter:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| og:image:alt | Stacked on #5986 — review that first. Base retargets to staging once it merges. Summary The zip route materialized every selected file before writing a byte, so peak memory tracked the size of the... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 7122bb414b542dba14d71a3f0b9f6fd2abce715b9f5912d082800563e6c806d1 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/simstudioai/sim git https://github.com/simstudioai/sim.git |
| octolytics-dimension-user_id | 199344406 |
| octolytics-dimension-user_login | simstudioai |
| octolytics-dimension-repository_id | 912559512 |
| octolytics-dimension-repository_nwo | simstudioai/sim |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 912559512 |
| octolytics-dimension-repository_network_root_nwo | simstudioai/sim |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | cbf66c024bb6d10e6955baa7413c54cf13a201bb |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width