Title: Support for DeltaLake, Iceberg, Hudi as an offline source covered by Spark, Ray and Dask engines · Issue #2406 · feast-dev/feast · GitHub
Open Graph Title: Support for DeltaLake, Iceberg, Hudi as an offline source covered by Spark, Ray and Dask engines · Issue #2406 · feast-dev/feast
X Title: Support for DeltaLake, Iceberg, Hudi as an offline source covered by Spark, Ray and Dask engines · Issue #2406 · feast-dev/feast
Description: Is your feature request related to a problem? Please describe. Feast currently can run on spark, dask and ray. Using this engines the support for deltalake, apache iceberg and apache hudi (which was requested by the community) can be add...
Open Graph Description: Is your feature request related to a problem? Please describe. Feast currently can run on spark, dask and ray. Using this engines the support for deltalake, apache iceberg and apache hudi (which wa...
X Description: Is your feature request related to a problem? Please describe. Feast currently can run on spark, dask and ray. Using this engines the support for deltalake, apache iceberg and apache hudi (which wa...
Opengraph URL: https://github.com/feast-dev/feast/issues/2406
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Support for DeltaLake, Iceberg, Hudi as an offline source covered by Spark, Ray and Dask engines","articleBody":"**Is your feature request related to a problem? Please describe.**\r\nFeast currently can run on [spark](https://github.com/feast-dev/feast/tree/master/sdk/python/feast/infra/offline_stores/contrib/spark_offline_store), [dask and ray](https://github.com/feast-dev/feast/blob/master/sdk/python/feast/infra/offline_stores/file.py). Using this engines the support for [deltalake](https://delta.io/), [apache iceberg](https://iceberg.apache.org/) and [apache hudi](https://hudi.apache.org/) (which was requested by the community) can be added. This feature will be very helpful for the teams which base on `Spark` but also for integrations with `Flink`, `Spark Streaming` eg. when historical features are saved in the data lake formats. \r\n\r\n**Describe the solution you'd like**\r\n\r\n\r\nThe solution assumes adding new data lake sources:\r\n* `DeltaDataSource`\r\n* `IcebergDataSource`\r\n* `HudiDataSource`\r\n\r\nand also support for CSV files.\r\n\r\nThe data lake sources support will be covered by the Spark engine (which is already in contrib) for users which use Feast on Spark (or Databricks) but also for Dask and Ray. \r\n\r\nAdditional assumptions:\r\n1) Data sources can be mixed eg. you can use `DeltaDataSource`, `CSV` and `IcebergDataSource` to fetch historical features.\r\n2) The engine change won't require changes in the code (only `feature_store.yaml` configuration) thus user can test on the laptop using Dask (without any cluster setup) and then deploy to Spark cluster (or Dask and Ray clusters)\r\n3) The implementation should enable to simply add new DataSources like [Apache Arrow Flight](https://arrow.apache.org/blog/2022/02/16/introducing-arrow-flight-sql/) (if the python api will be added) and simply mix them with other data sources in the future.\r\n\r\n## Delta Lake\r\nThe support for Delta Lake for Feast on Spark is already proposed and tested on Databricks and local spark (https://github.com/qooba/feast-pyspark). The solution is based on the plain `pyspark` rather than on `Spark SQL` and `Jinja` thus it is to decide which implementation will be more desirable and maintainable.\r\n\r\nThe support for Delta Lake for Feast on Dask (Ray) can be implemented using delta python interface:\r\n* https://github.com/delta-io/delta-rs\r\n* https://github.com/rajagurunath/dask_deltatable\r\n\r\n## Apache Iceberg\r\n\r\nThe Apache Iceberg is covered by the Spark engine but also by the [python api](https://iceberg.apache.org/docs/latest/python-feature-support/) which can be used to add Dask/Ray implementation.\r\n\r\n## Apache Hudi\r\n\r\nThe Apache Hudi is covered by the Spark. Currently there is no python api (as far as I know).\r\n\r\n## CSV \r\n\r\nThe support for `csv` files will be dedicated for the data scientists which would like to conduct ad-hoc experiments. \r\n\r\n**Describe alternatives you've considered**\r\nN/A\r\n\r\n**Additional context**\r\nN/A\r\n","author":{"url":"https://github.com/qooba","@type":"Person","name":"qooba"},"datePublished":"2022-03-14T23:57:26Z","url":"https://github.com/feast-dev/feast/issues/2406"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:ca52b327-636f-3c35-e09e-8d64eb2cb7d0 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | C302:22BB81:ED74E4:1387A11:6AFE9524 |
| html-safe-nonce | 9b9039d62cbdc75b22dc3b56bf33e069f394d77b6124b651399e2287def2e42c |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDMzAyOjIyQkI4MTpFRDc0RTQ6MTM4N0ExMTo2QUZFOTUyNCIsInZpc2l0b3JfaWQiOiIzNTQ0ODY1NDMyOTQwNzQ2MDIwIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | ef781f14266911d5c4f07fe95ecfe237ee1a8690e275236764529aea3e4ea684 |
| hovercard-subject-tag | issue:1169032116 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/feast-dev/feast/2406/issue_layout |
| twitter:image | https://opengraph.githubassets.com/d6af64a70e791dacc730055e248f7fee0d9eb4712089005726e9c4125861fe21/feast-dev/feast/issues/2406 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/d6af64a70e791dacc730055e248f7fee0d9eb4712089005726e9c4125861fe21/feast-dev/feast/issues/2406 |
| og:image:alt | Is your feature request related to a problem? Please describe. Feast currently can run on spark, dask and ray. Using this engines the support for deltalake, apache iceberg and apache hudi (which wa... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | qooba |
| hostname | github.com |
| expected-hostname | github.com |
| None | b3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86 |
| turbo-cache-control | no-preview |
| go-import | github.com/feast-dev/feast git https://github.com/feast-dev/feast.git |
| octolytics-dimension-user_id | 57027613 |
| octolytics-dimension-user_login | feast-dev |
| octolytics-dimension-repository_id | 161133770 |
| octolytics-dimension-repository_nwo | feast-dev/feast |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 161133770 |
| octolytics-dimension-repository_network_root_nwo | feast-dev/feast |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 4a10310af51a1bfffbedc4fd81cb582acf0cb0ff |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width