Title: Large Arrow results are fully materialized before use in three places (Flight, Trino, Spark) · Issue #6863 · feast-dev/feast · GitHub
Open Graph Title: Large Arrow results are fully materialized before use in three places (Flight, Trino, Spark) · Issue #6863 · feast-dev/feast
X Title: Large Arrow results are fully materialized before use in three places (Flight, Trino, Spark) · Issue #6863 · feast-dev/feast
Description: Is your feature request related to a problem? Please describe. Yes, in three independent places — Feast fully materializes an Arrow result as one giant, unchunked in-memory object before anything downstream can act on it, and none of the...
Open Graph Description: Is your feature request related to a problem? Please describe. Yes, in three independent places — Feast fully materializes an Arrow result as one giant, unchunked in-memory object before anything d...
X Description: Is your feature request related to a problem? Please describe. Yes, in three independent places — Feast fully materializes an Arrow result as one giant, unchunked in-memory object before anything d...
Opengraph URL: https://github.com/feast-dev/feast/issues/6863
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Large Arrow results are fully materialized before use in three places (Flight, Trino, Spark)","articleBody":"**Is your feature request related to a problem? Please describe.**\n\nYes, in three independent places — Feast fully materializes an Arrow result as one giant,\nunchunked in-memory object before anything downstream can act on it, and none of them scale:\n\n1. [`OfflineServer.do_get()`](https://github.com/feast-dev/feast/blob/master/sdk/python/feast/offline_server.py)\n wraps every result straight into `fl.RecordBatchStream(table)`, where `table` comes from each\n offline store's `_to_arrow_internal` — one giant, unchunked `RecordBatch`. Arrow's Flight/IPC\n writer refuses to serialize any single `RecordBatch` above 2GiB, so a job that completed\n successfully server-side crashes the moment the client tries to read the response:\n ```\n pyarrow.lib.ArrowInvalid: Cannot send record batches exceeding 2GiB yet\n ```\n\n2. [`TrinoRetrievalJob._to_arrow_internal()`](https://github.com/feast-dev/feast/blob/master/sdk/python/feast/infra/offline_stores/contrib/trino_offline_store/trino.py)\n does `cursor.fetchall()` → pandas `DataFrame` → `pyarrow.Table.from_pandas()` — two full\n in-memory copies of the result before a single row reaches the caller, and exactly what\n produces the oversized batch in (1.).\n\n3. [`SparkReadNode.execute()`](https://github.com/feast-dev/feast/blob/master/sdk/python/feast/infra/compute_engines/spark/nodes.py)\n converts the retrieved Arrow `Table` to Spark via `arrow_table.to_pandas()` then\n `createDataFrame(...)` — `to_pandas()` on a Flight/IPC-sourced, multi-chunk `Table` always\n makes a full pandas copy (pyarrow can't zero-copy multi-chunk/nullable columns, and\n `self_destruct` can't free the source either), doubling peak driver memory for every\n materialization job and driving real driver OOMs on large batch sources.\n\n**Describe the solution you'd like**\n\n- `offline_server.py` — before wrapping in `RecordBatchStream`, re-slice the outgoing table by\n row count (`to_batches(max_chunksize=N)`) and re-check each resulting batch's actual encoded\n byte size, bisecting further by row count for any batch still over a safety threshold.\n- Trino offline store — an opt-in streaming retrieval path:\n ```python\n # trino_queries.py\n query.start() # run the query, return column metadata\n query.iterate_pages(batch_size) # page via cursor.fetchmany()\n\n # trino.py\n TrinoRetrievalJob.to_arrow_reader() # -> pyarrow.RecordBatchReader, built from those pages\n ```\n `do_get()` prefers this reader over `to_arrow()` whenever a job carries no on-demand feature\n views.\n- `compute_engines/spark/nodes.py` — on Spark 4.0+, pass the `pyarrow.Table` directly to\n `createDataFrame()` (native ingestion via `_create_from_arrow_table`) instead of\n `.to_pandas()`, falling back to the pandas round-trip on older PySpark.\n\n**Describe alternatives you've considered**\n\n- (1.): just raising the offline server's chunking limits — doesn't address the memory doubling\n in (2.), only the point where it becomes fatal.\n- (2.): making `to_arrow()` itself lazy for every offline store — too large a behavior change for\n callers (on-demand transforms, dataset validation) that need the full `Table`; scoping to an\n opt-in method on Trino avoids that.\n- (3.): no real alternative once Spark 4.0+ supports native ingestion — the pandas round-trip is\n pure overhead.\n\n\nAdditional context\n\n\n**Test plan**\n\n- [ ] A synthetic result whose single `RecordBatch` exceeds 2GiB serializes over Arrow Flight\n without raising `ArrowInvalid`.\n- [ ] `TrinoRetrievalJob.to_arrow_reader()` never calls `cursor.fetchall()`, and its output\n matches `to_arrow()` for a query with `row(...)`/`map(...)`/`array(...)` columns and NULLs.\n- [ ] `SparkReadNode.execute()` uses native `pyarrow.Table` ingestion on Spark 4.0+, and output\n rows/schema match the `.to_pandas()` path for a `Table` with nulls and multiple chunks.","author":{"url":"https://github.com/dbbvitor","@type":"Person","name":"dbbvitor"},"datePublished":"2026-09-24T15:22:12Z","url":"https://github.com/feast-dev/feast/issues/6863"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:d5cf5789-ed03-0e97-c728-accf28f7c027 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | 9D76:2979B2:FAC529:1550EFB:6AFE980F |
| html-safe-nonce | 32ed93617ec3721e33fe235953a1c6505fa7e1fa5d5378023cad8099290b552a |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5RDc2OjI5NzlCMjpGQUM1Mjk6MTU1MEVGQjo2QUZFOTgwRiIsInZpc2l0b3JfaWQiOiI5MzcwODA4NjY5OTQwMzQ3MDMiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | c3e805f90c559e4f468191a25817af92f01ec97c4c08f32c8ff5521ff1462e2e |
| hovercard-subject-tag | issue:5570581574 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/feast-dev/feast/6863/issue_layout |
| twitter:image | https://opengraph.githubassets.com/e164944e7a17d0a8860371694da7031ff8d17b673b04c1b6f27eab6cf17063c4/feast-dev/feast/issues/6863 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/e164944e7a17d0a8860371694da7031ff8d17b673b04c1b6f27eab6cf17063c4/feast-dev/feast/issues/6863 |
| og:image:alt | Is your feature request related to a problem? Please describe. Yes, in three independent places — Feast fully materializes an Arrow result as one giant, unchunked in-memory object before anything d... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | dbbvitor |
| hostname | github.com |
| expected-hostname | github.com |
| None | b3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86 |
| turbo-cache-control | no-preview |
| go-import | github.com/feast-dev/feast git https://github.com/feast-dev/feast.git |
| octolytics-dimension-user_id | 57027613 |
| octolytics-dimension-user_login | feast-dev |
| octolytics-dimension-repository_id | 161133770 |
| octolytics-dimension-repository_nwo | feast-dev/feast |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 161133770 |
| octolytics-dimension-repository_network_root_nwo | feast-dev/feast |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 4a10310af51a1bfffbedc4fd81cb582acf0cb0ff |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width