Title: perf(tables): read a sorted view's order once per drain and project selected columns in SQL by waleedlatif1 · Pull Request #8915 · simstudioai/sim · GitHub
Open Graph Title: perf(tables): read a sorted view's order once per drain and project selected columns in SQL by waleedlatif1 · Pull Request #8915 · simstudioai/sim
X Title: perf(tables): read a sorted view's order once per drain and project selected columns in SQL by waleedlatif1 · Pull Request #8915 · simstudioai/sim
Description: Summary Custom-sort drains read the order once: sorted reads advanced by OFFSET, so every batch recomputed the sort key over the whole table (the key comes from the row's jsonb, so each pass decompresses every row). The drain now runs one id-only pass with the same ORDER BY/OFFSET, then fetches rows by primary key in the same byte-budgeted batches, preserving order. Unbounded drains start with a small id pass and extend as needed. Column selection is projected in SQL: when a read selects a subset of schema columns, data is narrowed in SQL (one read of the value, column ids bound as parameters, non-object data maps to {} as before) instead of shipping whole rows and projecting in Node. Batches are sized from the bytes actually fetched. Selecting every column keeps the existing Node-side path. Output is unchanged: rows, key order, null/missing semantics, hasMore/cursor/offset behavior, byte budget and TABLE_QUERY_RESULT_TOO_LARGE thresholds. Type of Change Performance improvement Testing Old vs new fetchRowsBounded compared over 4,410 cases (sorts incl. ties/nulls/missing keys, filters, column selections with and without SQL projection, limits, offsets, budgets, second id pass, non-object data): identical results; full sort passes reduced ~4×, never more than before Replica EXPLAIN (ANALYZE, BUFFERS, SERIALIZE) on a wide table: 2-column selection ~3× less server time and ~48× fewer bytes; sorted drain one full pass instead of one per batch. Trade-off: a selection that omits only columns no row has written pays ~10 ms more server time for the same bytes Integration cases: sorted drain across batch boundaries, from an offset and through a second id pass; projection exactness (JSON null kept, missing key omitted, caller column order) Unit tests for bounded fetch sizing and filter threading pass; biome on changed files Checklist Code follows project style guidelines Self-reviewed my changes Tests added/updated and passing (new tests pass the test-audit authoring gate) No new warnings introduced I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)
Open Graph Description: Summary Custom-sort drains read the order once: sorted reads advanced by OFFSET, so every batch recomputed the sort key over the whole table (the key comes from the row's jsonb, so each pass d...
X Description: Summary Custom-sort drains read the order once: sorted reads advanced by OFFSET, so every batch recomputed the sort key over the whole table (the key comes from the row's jsonb, so each pa...
Opengraph URL: https://github.com/simstudioai/sim/pull/8915
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:5d28b8a0-365e-ee14-6f16-6d28b1c793c3 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | B42E:1E90AB:35BB5A:4B92CB:6AFB714A |
| html-safe-nonce | 0a8b8345bf5d1510449c747eaf3f656360faf1d7bf2b3c2ed05ff39f78f21ec2 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCNDJFOjFFOTBBQjozNUJCNUE6NEI5MkNCOjZBRkI3MTRBIiwidmlzaXRvcl9pZCI6IjIxNzYyNDY4OTExNTczNTI3NzgiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | 23b84383fd808caf1056ac5980ed918525b9df99ae92c073e2cdd006794d4e2b |
| hovercard-subject-tag | pull_request:4825747810 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/8915/files |
| twitter:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| og:image:alt | Summary Custom-sort drains read the order once: sorted reads advanced by OFFSET, so every batch recomputed the sort key over the whole table (the key comes from the row's jsonb, so each pass d... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 0e29e7974fdda2feeaf5c891efd4b23b0734b23480232b62c7bdef13f40b5b48 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/simstudioai/sim git https://github.com/simstudioai/sim.git |
| octolytics-dimension-user_id | 199344406 |
| octolytics-dimension-user_login | simstudioai |
| octolytics-dimension-repository_id | 912559512 |
| octolytics-dimension-repository_nwo | simstudioai/sim |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 912559512 |
| octolytics-dimension-repository_network_root_nwo | simstudioai/sim |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 2e8872b770cc4d5d0cc1f2026afab747a268e2ce |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width