René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Make materialization more scalable + performant","articleBody":"This issue discusses common issues users face when materializing features to the online store in Feast.\r\n\r\n## User problems\r\nGenerally, users with large datasets can face issues on reliably loading data into the online store to meet their online needs.\r\n\r\n### 1. Materialization in the default provider is not scalable\r\nAs per https://github.com/feast-dev/feast/issues/2071, \r\n\r\n\u003e Currently, the materialization process loads all the data from the Offline Store to an Arrow table, then converts all the data to Protobuf, then writes all the data to the Online Store. This process requires holding the entire dataset in memory which is not practical.\r\n\r\n### 2. Materialization can be slow\r\nFor users that aren't working with a small number of feature views or large number of unique entities, Feast's python based materialization works fine. However, this does not hold true for many users.\r\n\r\nThe default provider is slow to materialize data. Users can report multiple hours to do incremental materialization, or worse materialization never completes.\r\n\r\nUsers have had to build custom providers to solve this (e.g. by kicking off Dataflow or Spark jobs to more quickly materialize large amounts of data)\r\n\r\n### 3. Materialization not always reliable\r\nThere are several datastore specific issues such as https://github.com/feast-dev/feast/issues/2027 and https://github.com/feast-dev/feast/issues/2323, where batch write transactions can time out:\r\n\r\n```\r\nFile \"/usr/local/lib/python3.7/dist-packages/google/api_core/grpc_helpers.py\", line 69, in error_remapped_callable\r\nsix.raise_from(exceptions.from_grpc_error(exc), exc)\r\nFile \"\", line 3, in raise_from\r\ngoogle.api_core.exceptions.InvalidArgument: 400 The referenced transaction has expired or is no longer valid.\r\n```\r\n\r\nIn datastore, there are also contention errors (https://github.com/feast-dev/feast/issues/1575):\r\n\r\n```\r\nMaterializing 1 feature views from 2021-04-29 21:19:00-07:00 to 2021-04-29 21:19:05-07:00 into the datastore online store.\r\nmy_fv:\r\n  8%|████▋                                                    | 2720/33120 [00:07\u003c01:18, 388.10it/s]\r\n\r\ngoogle.api_core.exceptions.Aborted: 409 too much contention on these datastore entities. please try again. entity groups:\r\n```\r\n\r\n\r\n\r\n## Describe the solution you'd like\r\nA clear and concise description of what you want to happen.\r\n\r\nThere are multiple ways of addressing this. Some ideas\r\n- Materialize feature views in parallel instead of sequentially (e.g. https://github.com/feast-dev/feast/issues/2421)\r\n- Speed up via PyPy or Cython to optimize slowest python parts.\r\n- Speed up with online store specific bulk loading options (e.g. https://redis.io/docs/reference/patterns/bulk-loading/ or https://cloud.google.com/datastore/docs/reference/admin/rpc/google.datastore.admin.v1#google.datastore.admin.v1.DatastoreAdmin.ImportEntities)\r\n- Materialize via Spark \r\n- Materialize with serverless functions e.g. AWS Lambda\r\n- Spin up separate tasks to materialize each feature view separately\r\n- Spin up separate tasks to materialize smaller intervals of data\r\n- Fix Datastore materialization: https://github.com/feast-dev/feast/issues/1575","author":{"url":"https://github.com/adchia","@type":"Person","name":"adchia"},"datePublished":"2022-04-21T21:25:32.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/2594/feast/issues/2594"}