Title: SparkOfflineStore.offline_write_batch ignores table_format and writes with file_format · Issue #6955 · feast-dev/feast · GitHub
Open Graph Title: SparkOfflineStore.offline_write_batch ignores table_format and writes with file_format · Issue #6955 · feast-dev/feast
X Title: SparkOfflineStore.offline_write_batch ignores table_format and writes with file_format · Issue #6955 · feast-dev/feast
Description: Problem SparkSource carries two independent format axes, and says so: # contrib/spark_offline_store/spark_source.py:45 file_format: Optional[str] = None, table_format: Optional[TableFormat] = None, ... # :66 table_format: The table metad...
Open Graph Description: Problem SparkSource carries two independent format axes, and says so: # contrib/spark_offline_store/spark_source.py:45 file_format: Optional[str] = None, table_format: Optional[TableFormat] = None,...
X Description: Problem SparkSource carries two independent format axes, and says so: # contrib/spark_offline_store/spark_source.py:45 file_format: Optional[str] = None, table_format: Optional[TableFormat] = None,...
Opengraph URL: https://github.com/feast-dev/feast/issues/6955
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"SparkOfflineStore.offline_write_batch ignores table_format and writes with file_format","articleBody":"## Problem\n\n`SparkSource` carries two independent format axes, and says so:\n\n```python\n# contrib/spark_offline_store/spark_source.py:45\nfile_format: Optional[str] = None,\ntable_format: Optional[TableFormat] = None,\n...\n# :66\n table_format: The table metadata format (iceberg, delta, hudi, etc.).\n Optional and separate from file_format.\n```\n\nThey are separate fields in the proto too:\n\n```protobuf\n// DataSource.proto\nmessage SparkOptions {\n ...\n string file_format = 4;\n TableFormat table_format = 6;\n}\n```\n\n`offline_write_batch` only consults `file_format`. Its path branch uses it twice — once to read the existing dataset for its schema, once to write:\n\n```python\n# contrib/spark_offline_store/spark.py:398\ndf_existing = spark_session.read.format(\n feature_view.batch_source.file_format # :400\n).load(feature_view.batch_source.path)\n...\ndf_batch.write.format(feature_view.batch_source.file_format).mode( # :406\n \"append\"\n).save(feature_view.batch_source.path)\n```\n\nSo a `SparkSource` addressed by `path` whose format is carried on `table_format` has no correct write. With `table_format` set and `file_format` left unset — which is what the two-axis split invites — both calls receive `None`.\n\nThe catalog-addressed case does not reach a wrong writer, but has no writer at all:\n\n```python\n# :414\nraise NotImplementedError(\n \"offline_write_batch not implemented for batch sources specified by a table\"\n)\n```\n\n## Not specific to one table format\n\nThis affects anything expressed through `table_format` on a `SparkSource`: Iceberg, Delta, Hudi, and `LanceFormat` from #6925. `read` is unaffected, since the Spark reader path does consult `table_format`, so the two directions currently disagree about which axis is authoritative.\n\n`IcebergSource` is unaffected in practice, but only because it is a separate data source class with its own branch at the top of the same function:\n\n```python\n# :369\nif isinstance(feature_view.batch_source, IcebergSource):\n _iceberg_offline_write_batch(config, feature_view, table)\n return\n```\n\nThat branch is the precedent I would follow, but it keys off the data source *class*, so it does nothing for a plain `SparkSource` that merely has a `table_format`.\n\n## Why I am raising it rather than sending a patch\n\nI do not think the fix is obvious, and the choice is a design decision rather than a bug fix:\n\n1. **Dispatch on `table_format` when present, `file_format` otherwise.** Smallest change and matches what the reader already does. Needs a decision on what a source with *both* set means — today that is accepted silently.\n2. **Give each table format a dedicated branch**, as `IcebergSource` has. Explicit, but grows the function per format and still leaves plain `SparkSource` + `table_format` unhandled unless the branch keys off the format rather than the class.\n3. **Reject `table_format` on the write path outright** until a format opts in, which at least replaces a wrong writer with an error.\n\nThere is also a prior question: if `table_format` is authoritative for writes, should `offline_write_batch` keep going through `read` + `_cast_data_frame` + `write.format(...).save(path)` at all, or use `writeTo(...)` for catalog-managed formats? That would also give the catalog-addressed case at `:414` a path, which the current structure cannot.\n\nHappy to implement whichever direction maintainers prefer. I ran into this looking at what a Spark write path for Lance would need, so I have a concrete case to test against, but I would rather not guess at the intended semantics of the two axes.\n\n## Reproduction\n\nA `SparkSource(path=..., table_format=LanceFormat())` with no `file_format` reaches `spark_session.read.format(None)` at `:400`. Any `table_format` reproduces it the same way; the format itself is not what matters.\n","author":{"url":"https://github.com/haoxu0","@type":"Person","name":"haoxu0"},"datePublished":"2026-10-06T07:28:21Z","url":"https://github.com/feast-dev/feast/issues/6955"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:6a1342fa-808c-85e2-57d5-2ef386b82aa2 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | AFC8:2D746B:2C6CE4:3BD775:6AFE981B |
| html-safe-nonce | 30051fe84edbef82b737a79af5e0a5349299fb339670bf2dec933dd7be8ac33e |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBRkM4OjJENzQ2QjoyQzZDRTQ6M0JENzc1OjZBRkU5ODFCIiwidmlzaXRvcl9pZCI6IjIxMDI1NDM4MzkzMTM5NTg5MzkiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | e45e30505c8f3f06dec18272b14ea516ac8d7c3ed379901af16774c9563a75f1 |
| hovercard-subject-tag | issue:5724677105 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/feast-dev/feast/6955/issue_layout |
| twitter:image | https://opengraph.githubassets.com/9513e2a297f3c6860a6e13ce53af0d3fb002c952c0cac0f62eecf4c864539b2d/feast-dev/feast/issues/6955 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/9513e2a297f3c6860a6e13ce53af0d3fb002c952c0cac0f62eecf4c864539b2d/feast-dev/feast/issues/6955 |
| og:image:alt | Problem SparkSource carries two independent format axes, and says so: # contrib/spark_offline_store/spark_source.py:45 file_format: Optional[str] = None, table_format: Optional[TableFormat] = None,... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | haoxu0 |
| hostname | github.com |
| expected-hostname | github.com |
| None | b3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86 |
| turbo-cache-control | no-preview |
| go-import | github.com/feast-dev/feast git https://github.com/feast-dev/feast.git |
| octolytics-dimension-user_id | 57027613 |
| octolytics-dimension-user_login | feast-dev |
| octolytics-dimension-repository_id | 161133770 |
| octolytics-dimension-repository_nwo | feast-dev/feast |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 161133770 |
| octolytics-dimension-repository_network_root_nwo | feast-dev/feast |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 4a10310af51a1bfffbedc4fd81cb582acf0cb0ff |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width