René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Bytewax materializer materializes entire dataset on every pod","articleBody":"## Expected Behavior \r\n\r\nEach pod should materialize a separate batch of data\r\n\r\n## Current Behavior\r\n\r\nEvery pod redundantly materializes the entire dataset, causing a massive overspend on processing time and database writes\r\n\r\n## Steps to reproduce\r\n\r\nObserved with snowflake offline store and dynamo db online store.  Materialize a feature with enough records for the snowflake s3 integration to output multiple files.  View the logs of each pod and add up the total records materialized.  Observe that every pod has materialized the entire dataset\r\n\r\n### Specifications\r\n\r\n- Version: 0.31\r\n- Platform: fedora linux\r\n- Subsystem: bytewax batch_engine\r\n\r\n## Possible Solution\r\n\r\nUse JOB_COMPLETION_INDEX in each pods dataflow to select a single file to process\r\n","author":{"url":"https://github.com/james-crabtree-sp","@type":"Person","name":"james-crabtree-sp"},"datePublished":"2023-10-09T19:18:29.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/3786/feast/issues/3786"}