René's URL Explorer Experiment


Title: SparkOfflineStore.offline_write_batch ignores table_format and writes with file_format · Issue #6955 · feast-dev/feast · GitHub

Open Graph Title: SparkOfflineStore.offline_write_batch ignores table_format and writes with file_format · Issue #6955 · feast-dev/feast

X Title: SparkOfflineStore.offline_write_batch ignores table_format and writes with file_format · Issue #6955 · feast-dev/feast

Description: Problem SparkSource carries two independent format axes, and says so: # contrib/spark_offline_store/spark_source.py:45 file_format: Optional[str] = None, table_format: Optional[TableFormat] = None, ... # :66 table_format: The table metad...

Open Graph Description: Problem SparkSource carries two independent format axes, and says so: # contrib/spark_offline_store/spark_source.py:45 file_format: Optional[str] = None, table_format: Optional[TableFormat] = None,...

X Description: Problem SparkSource carries two independent format axes, and says so: # contrib/spark_offline_store/spark_source.py:45 file_format: Optional[str] = None, table_format: Optional[TableFormat] = None,...

Opengraph URL: https://github.com/feast-dev/feast/issues/6955

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"SparkOfflineStore.offline_write_batch ignores table_format and writes with file_format","articleBody":"## Problem\n\n`SparkSource` carries two independent format axes, and says so:\n\n```python\n# contrib/spark_offline_store/spark_source.py:45\nfile_format: Optional[str] = None,\ntable_format: Optional[TableFormat] = None,\n...\n# :66\n    table_format: The table metadata format (iceberg, delta, hudi, etc.).\n        Optional and separate from file_format.\n```\n\nThey are separate fields in the proto too:\n\n```protobuf\n// DataSource.proto\nmessage SparkOptions {\n  ...\n  string file_format = 4;\n  TableFormat table_format = 6;\n}\n```\n\n`offline_write_batch` only consults `file_format`. Its path branch uses it twice — once to read the existing dataset for its schema, once to write:\n\n```python\n# contrib/spark_offline_store/spark.py:398\ndf_existing = spark_session.read.format(\n    feature_view.batch_source.file_format     # :400\n).load(feature_view.batch_source.path)\n...\ndf_batch.write.format(feature_view.batch_source.file_format).mode(   # :406\n    \"append\"\n).save(feature_view.batch_source.path)\n```\n\nSo a `SparkSource` addressed by `path` whose format is carried on `table_format` has no correct write. With `table_format` set and `file_format` left unset — which is what the two-axis split invites — both calls receive `None`.\n\nThe catalog-addressed case does not reach a wrong writer, but has no writer at all:\n\n```python\n# :414\nraise NotImplementedError(\n    \"offline_write_batch not implemented for batch sources specified by a table\"\n)\n```\n\n## Not specific to one table format\n\nThis affects anything expressed through `table_format` on a `SparkSource`: Iceberg, Delta, Hudi, and `LanceFormat` from #6925. `read` is unaffected, since the Spark reader path does consult `table_format`, so the two directions currently disagree about which axis is authoritative.\n\n`IcebergSource` is unaffected in practice, but only because it is a separate data source class with its own branch at the top of the same function:\n\n```python\n# :369\nif isinstance(feature_view.batch_source, IcebergSource):\n    _iceberg_offline_write_batch(config, feature_view, table)\n    return\n```\n\nThat branch is the precedent I would follow, but it keys off the data source *class*, so it does nothing for a plain `SparkSource` that merely has a `table_format`.\n\n## Why I am raising it rather than sending a patch\n\nI do not think the fix is obvious, and the choice is a design decision rather than a bug fix:\n\n1. **Dispatch on `table_format` when present, `file_format` otherwise.** Smallest change and matches what the reader already does. Needs a decision on what a source with *both* set means — today that is accepted silently.\n2. **Give each table format a dedicated branch**, as `IcebergSource` has. Explicit, but grows the function per format and still leaves plain `SparkSource` + `table_format` unhandled unless the branch keys off the format rather than the class.\n3. **Reject `table_format` on the write path outright** until a format opts in, which at least replaces a wrong writer with an error.\n\nThere is also a prior question: if `table_format` is authoritative for writes, should `offline_write_batch` keep going through `read` + `_cast_data_frame` + `write.format(...).save(path)` at all, or use `writeTo(...)` for catalog-managed formats? That would also give the catalog-addressed case at `:414` a path, which the current structure cannot.\n\nHappy to implement whichever direction maintainers prefer. I ran into this looking at what a Spark write path for Lance would need, so I have a concrete case to test against, but I would rather not guess at the intended semantics of the two axes.\n\n## Reproduction\n\nA `SparkSource(path=..., table_format=LanceFormat())` with no `file_format` reaches `spark_session.read.format(None)` at `:400`. Any `table_format` reproduces it the same way; the format itself is not what matters.\n","author":{"url":"https://github.com/haoxu0","@type":"Person","name":"haoxu0"},"datePublished":"2026-10-06T07:28:21Z","url":"https://github.com/feast-dev/feast/issues/6955"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:6a1342fa-808c-85e2-57d5-2ef386b82aa2
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-idAFC8:2D746B:2C6CE4:3BD775:6AFE981B
html-safe-nonce30051fe84edbef82b737a79af5e0a5349299fb339670bf2dec933dd7be8ac33e
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBRkM4OjJENzQ2QjoyQzZDRTQ6M0JENzc1OjZBRkU5ODFCIiwidmlzaXRvcl9pZCI6IjIxMDI1NDM4MzkzMTM5NTg5MzkiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmace45e30505c8f3f06dec18272b14ea516ac8d7c3ed379901af16774c9563a75f1
hovercard-subject-tagissue:5724677105
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/feast-dev/feast/6955/issue_layout
twitter:imagehttps://opengraph.githubassets.com/9513e2a297f3c6860a6e13ce53af0d3fb002c952c0cac0f62eecf4c864539b2d/feast-dev/feast/issues/6955
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/9513e2a297f3c6860a6e13ce53af0d3fb002c952c0cac0f62eecf4c864539b2d/feast-dev/feast/issues/6955
og:image:altProblem SparkSource carries two independent format axes, and says so: # contrib/spark_offline_store/spark_source.py:45 file_format: Optional[str] = None, table_format: Optional[TableFormat] = None,...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernamehaoxu0
hostnamegithub.com
expected-hostnamegithub.com
Noneb3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86
turbo-cache-controlno-preview
go-importgithub.com/feast-dev/feast git https://github.com/feast-dev/feast.git
octolytics-dimension-user_id57027613
octolytics-dimension-user_loginfeast-dev
octolytics-dimension-repository_id161133770
octolytics-dimension-repository_nwofeast-dev/feast
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id161133770
octolytics-dimension-repository_network_root_nwofeast-dev/feast
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release4a10310af51a1bfffbedc4fd81cb582acf0cb0ff
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/feast-dev/feast/issues/6955#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ffeast-dev%2Ffeast%2Fissues%2F6955
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ffeast-dev%2Ffeast%2Fissues%2F6955
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=feast-dev%2Ffeast
Reloadhttps://github.com/feast-dev/feast/issues/6955
Reloadhttps://github.com/feast-dev/feast/issues/6955
Reloadhttps://github.com/feast-dev/feast/issues/6955
feast-devhttps://github.com/feast-dev
feasthttps://github.com/feast-dev/feast
Codehttps://github.com/feast-dev/feast
Issues223 (223)https://github.com/feast-dev/feast/issues
Pull requests143 (143)https://github.com/feast-dev/feast/pulls
Discussionshttps://github.com/feast-dev/feast/discussions
Actionshttps://github.com/feast-dev/feast/actions
Security and quality1 (1)https://github.com/feast-dev/feast/security
Insightshttps://github.com/feast-dev/feast/insights
https://github.com/feast-dev/feast
New issuehttps://github.com/login?return_to=https://github.com/feast-dev/feast/issues/6955
New issuehttps://github.com/login?return_to=https://github.com/feast-dev/feast/issues/6955
#7055https://github.com/feast-dev/feast/pull/7055
SparkOfflineStore.offline_write_batch ignores table_format and writes with file_formathttps://github.com/feast-dev/feast/issues/6955#top
#7055https://github.com/feast-dev/feast/pull/7055
https://github.com/haoxu0
haoxu0https://github.com/haoxu0
on Oct 6, 2026https://github.com/feast-dev/feast/issues/6955#issue-5724677105
#6925https://github.com/feast-dev/feast/pull/6925
https://github.com/Mrun25
Mrun25https://github.com/Mrun25
on Oct 20, 2026https://github.com/feast-dev/feast/issues/6955#issuecomment-6294301660
https://github.com/Mrun25
Mrun25https://github.com/Mrun25
on Oct 21, 2026https://github.com/feast-dev/feast/issues/6955#issuecomment-6308154354
#7055https://github.com/feast-dev/feast/pull/7055
ntkatholehttps://github.com/ntkathole
completedhttps://github.com/feast-dev/feast/issues?q=is%3Aissue%20state%3Aclosed%20archived%3Afalse%20reason%3Acompleted
#7055https://github.com/feast-dev/feast/pull/7055
on Nov 16, 2026https://github.com/feast-dev/feast/issues/6955#event-36242015133
Sign up for freehttps://github.com/signup?return_to=https://github.com/feast-dev/feast/issues/6955
Sign in to commenthttps://github.com/login?return_to=https://github.com/feast-dev/feast/issues/6955
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.