René's URL Explorer Experiment


Title: Support for DeltaLake, Iceberg, Hudi as an offline source covered by Spark, Ray and Dask engines · Issue #2406 · feast-dev/feast · GitHub

Open Graph Title: Support for DeltaLake, Iceberg, Hudi as an offline source covered by Spark, Ray and Dask engines · Issue #2406 · feast-dev/feast

X Title: Support for DeltaLake, Iceberg, Hudi as an offline source covered by Spark, Ray and Dask engines · Issue #2406 · feast-dev/feast

Description: Is your feature request related to a problem? Please describe. Feast currently can run on spark, dask and ray. Using this engines the support for deltalake, apache iceberg and apache hudi (which was requested by the community) can be add...

Open Graph Description: Is your feature request related to a problem? Please describe. Feast currently can run on spark, dask and ray. Using this engines the support for deltalake, apache iceberg and apache hudi (which wa...

X Description: Is your feature request related to a problem? Please describe. Feast currently can run on spark, dask and ray. Using this engines the support for deltalake, apache iceberg and apache hudi (which wa...

Opengraph URL: https://github.com/feast-dev/feast/issues/2406

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Support for DeltaLake, Iceberg, Hudi as an offline source covered by Spark, Ray and Dask engines","articleBody":"**Is your feature request related to a problem? Please describe.**\r\nFeast currently can run on [spark](https://github.com/feast-dev/feast/tree/master/sdk/python/feast/infra/offline_stores/contrib/spark_offline_store), [dask and ray](https://github.com/feast-dev/feast/blob/master/sdk/python/feast/infra/offline_stores/file.py). Using this engines the support for [deltalake](https://delta.io/), [apache iceberg](https://iceberg.apache.org/) and [apache hudi](https://hudi.apache.org/) (which was requested by the community) can be added. This feature will be very helpful for the teams which base on `Spark` but also for integrations with `Flink`, `Spark Streaming` eg. when historical features are saved in the data lake formats.  \r\n\r\n**Describe the solution you'd like**\r\n![EnginesSources drawio](https://user-images.githubusercontent.com/14150080/158275190-3ff14de2-375e-4f27-adaf-819930119032.png)\r\n\r\nThe solution assumes adding new data lake sources:\r\n* `DeltaDataSource`\r\n* `IcebergDataSource`\r\n* `HudiDataSource`\r\n\r\nand also support for CSV files.\r\n\r\nThe data lake sources support will be covered by the Spark engine (which is already in contrib) for users which use Feast on Spark (or Databricks) but also for Dask and Ray. \r\n\r\nAdditional assumptions:\r\n1) Data sources can be mixed eg. you can use `DeltaDataSource`, `CSV` and `IcebergDataSource` to fetch historical features.\r\n2) The engine change won't require changes in the code (only `feature_store.yaml` configuration) thus user can test on the laptop using Dask (without any cluster setup) and then deploy to Spark cluster (or Dask and Ray clusters)\r\n3) The implementation should enable to simply add new DataSources like [Apache Arrow Flight](https://arrow.apache.org/blog/2022/02/16/introducing-arrow-flight-sql/) (if the python api will be added) and simply mix them with other data sources in the future.\r\n\r\n## Delta Lake\r\nThe support for Delta Lake for Feast on Spark is already proposed and tested on Databricks and local spark (https://github.com/qooba/feast-pyspark). The solution is based on the plain `pyspark` rather than on `Spark SQL` and `Jinja` thus it is to decide which implementation will be more desirable and maintainable.\r\n\r\nThe support for Delta Lake for Feast on Dask (Ray) can be implemented using delta python interface:\r\n* https://github.com/delta-io/delta-rs\r\n* https://github.com/rajagurunath/dask_deltatable\r\n\r\n## Apache Iceberg\r\n\r\nThe Apache Iceberg is covered by the Spark engine but also by the [python api](https://iceberg.apache.org/docs/latest/python-feature-support/) which can be used to add Dask/Ray implementation.\r\n\r\n## Apache Hudi\r\n\r\nThe Apache Hudi is covered by the Spark. Currently there is no python api (as far as I know).\r\n\r\n## CSV \r\n\r\nThe support for `csv` files will be dedicated for the data scientists which would like to conduct ad-hoc experiments. \r\n\r\n**Describe alternatives you've considered**\r\nN/A\r\n\r\n**Additional context**\r\nN/A\r\n","author":{"url":"https://github.com/qooba","@type":"Person","name":"qooba"},"datePublished":"2022-03-14T23:57:26Z","url":"https://github.com/feast-dev/feast/issues/2406"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:ca52b327-636f-3c35-e09e-8d64eb2cb7d0
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-idC302:22BB81:ED74E4:1387A11:6AFE9524
html-safe-nonce9b9039d62cbdc75b22dc3b56bf33e069f394d77b6124b651399e2287def2e42c
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDMzAyOjIyQkI4MTpFRDc0RTQ6MTM4N0ExMTo2QUZFOTUyNCIsInZpc2l0b3JfaWQiOiIzNTQ0ODY1NDMyOTQwNzQ2MDIwIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0=
visitor-hmacef781f14266911d5c4f07fe95ecfe237ee1a8690e275236764529aea3e4ea684
hovercard-subject-tagissue:1169032116
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/feast-dev/feast/2406/issue_layout
twitter:imagehttps://opengraph.githubassets.com/d6af64a70e791dacc730055e248f7fee0d9eb4712089005726e9c4125861fe21/feast-dev/feast/issues/2406
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/d6af64a70e791dacc730055e248f7fee0d9eb4712089005726e9c4125861fe21/feast-dev/feast/issues/2406
og:image:altIs your feature request related to a problem? Please describe. Feast currently can run on spark, dask and ray. Using this engines the support for deltalake, apache iceberg and apache hudi (which wa...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernameqooba
hostnamegithub.com
expected-hostnamegithub.com
Noneb3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86
turbo-cache-controlno-preview
go-importgithub.com/feast-dev/feast git https://github.com/feast-dev/feast.git
octolytics-dimension-user_id57027613
octolytics-dimension-user_loginfeast-dev
octolytics-dimension-repository_id161133770
octolytics-dimension-repository_nwofeast-dev/feast
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id161133770
octolytics-dimension-repository_network_root_nwofeast-dev/feast
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release4a10310af51a1bfffbedc4fd81cb582acf0cb0ff
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/feast-dev/feast/issues/2406#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ffeast-dev%2Ffeast%2Fissues%2F2406
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ffeast-dev%2Ffeast%2Fissues%2F2406
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=feast-dev%2Ffeast
Reloadhttps://github.com/feast-dev/feast/issues/2406
Reloadhttps://github.com/feast-dev/feast/issues/2406
Reloadhttps://github.com/feast-dev/feast/issues/2406
feast-devhttps://github.com/feast-dev
feasthttps://github.com/feast-dev/feast
Codehttps://github.com/feast-dev/feast
Issues223 (223)https://github.com/feast-dev/feast/issues
Pull requests143 (143)https://github.com/feast-dev/feast/pulls
Discussionshttps://github.com/feast-dev/feast/discussions
Actionshttps://github.com/feast-dev/feast/actions
Security and quality1 (1)https://github.com/feast-dev/feast/security
Insightshttps://github.com/feast-dev/feast/insights
https://github.com/feast-dev/feast
New issuehttps://github.com/login?return_to=https://github.com/feast-dev/feast/issues/2406
New issuehttps://github.com/login?return_to=https://github.com/feast-dev/feast/issues/2406
Support for DeltaLake, Iceberg, Hudi as an offline source covered by Spark, Ray and Dask engineshttps://github.com/feast-dev/feast/issues/2406#top
kind/featureNew feature or requesthttps://github.com/feast-dev/feast/issues?q=state%3Aopen%20label%3A%22kind%2Ffeature%22
https://github.com/qooba
qoobahttps://github.com/qooba
on Mar 14, 2022https://github.com/feast-dev/feast/issues/2406#issue-1169032116
sparkhttps://github.com/feast-dev/feast/tree/master/sdk/python/feast/infra/offline_stores/contrib/spark_offline_store
dask and rayhttps://github.com/feast-dev/feast/blob/master/sdk/python/feast/infra/offline_stores/file.py
deltalakehttps://delta.io/
apache iceberghttps://iceberg.apache.org/
apache hudihttps://hudi.apache.org/
https://user-images.githubusercontent.com/14150080/158275190-3ff14de2-375e-4f27-adaf-819930119032.png
Apache Arrow Flighthttps://arrow.apache.org/blog/2022/02/16/introducing-arrow-flight-sql/
https://github.com/qooba/feast-pysparkhttps://github.com/qooba/feast-pyspark
https://github.com/delta-io/delta-rshttps://github.com/delta-io/delta-rs
https://github.com/rajagurunath/dask_deltatablehttps://github.com/rajagurunath/dask_deltatable
python apihttps://iceberg.apache.org/docs/latest/python-feature-support/
qoobahttps://github.com/qooba
kind/featureNew feature or requesthttps://github.com/feast-dev/feast/issues?q=state%3Aopen%20label%3A%22kind%2Ffeature%22
on Mar 14, 2022https://github.com/feast-dev/feast/issues/2406#event-6239407951
https://github.com/tianlinzx
tianlinzxhttps://github.com/tianlinzx
on Apr 20, 2022https://github.com/feast-dev/feast/issues/2406#issuecomment-1104525026
adchiahttps://github.com/adchia
Feast Roadmaphttps://github.com/orgs/feast-dev/projects/2
on Apr 21, 2022https://github.com/feast-dev/feast/issues/2406#event-15232806158
https://github.com/qooba
qoobahttps://github.com/qooba
on Apr 27, 2022https://github.com/feast-dev/feast/issues/2406#issuecomment-1111580153
https://github.com/qooba/yummyhttps://github.com/qooba/yummy
qoobahttps://github.com/qooba
completedhttps://github.com/feast-dev/feast/issues?q=is%3Aissue%20state%3Aclosed%20archived%3Afalse%20reason%3Acompleted
on Apr 27, 2022https://github.com/feast-dev/feast/issues/2406#event-6510774593
qoobahttps://github.com/qooba
Feast Roadmaphttps://github.com/orgs/feast-dev/projects/2
on Apr 27, 2022https://github.com/feast-dev/feast/issues/2406
https://github.com/LeonardAukea
LeonardAukeahttps://github.com/LeonardAukea
on May 19, 2022https://github.com/feast-dev/feast/issues/2406#issuecomment-1131384722
https://github.com/qooba/yummyhttps://github.com/qooba/yummy
https://github.com/qooba
qoobahttps://github.com/qooba
on May 19, 2022https://github.com/feast-dev/feast/issues/2406#issuecomment-1132198778
https://github.com/qooba/yummyhttps://github.com/qooba/yummy
https://github.com/qooba
qoobahttps://github.com/qooba
on Jul 25, 2022https://github.com/feast-dev/feast/issues/2406#issuecomment-1194702636
https://github.com/qooba/yummyhttps://github.com/qooba/yummy
@LeonardAukeahttps://github.com/LeonardAukea
https://www.youtube.com/watch?v=kv0iWuSf4jwhttps://www.youtube.com/watch?v=kv0iWuSf4jw
https://github.com/andywalner
andywalnerhttps://github.com/andywalner
on Mar 19, 2025https://github.com/feast-dev/feast/issues/2406#issuecomment-2737617612
https://hudi.apache.org/docs/python-rust-quick-start-guide/https://hudi.apache.org/docs/python-rust-quick-start-guide/
https://github.com/franciscojavierarceo
franciscojavierarceohttps://github.com/franciscojavierarceo
on May 15, 2025https://github.com/feast-dev/feast/issues/2406#issuecomment-2884735579
@qoobahttps://github.com/qooba
franciscojavierarceohttps://github.com/franciscojavierarceo
on May 15, 2025https://github.com/feast-dev/feast/issues/2406#event-17677469739
https://github.com/haoxu0
haoxu0https://github.com/haoxu0
on Jun 13, 2025https://github.com/feast-dev/feast/issues/2406#issuecomment-2971898419
Sign up for freehttps://github.com/signup?return_to=https://github.com/feast-dev/feast/issues/2406
Sign in to commenthttps://github.com/login?return_to=https://github.com/feast-dev/feast/issues/2406
kind/featureNew feature or requesthttps://github.com/feast-dev/feast/issues?q=state%3Aopen%20label%3A%22kind%2Ffeature%22
Feast Roadmaphttps://github.com/orgs/feast-dev/projects/2
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.