René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Feature Request: Add configurable batching for offline store writes in Feature Server push API","articleBody":"## Is your feature request related to a problem? Please describe.\n\nCurrently, when using the Feature Server's push API with streaming ingestion, offline store writes are not automatically batched. Users must either:\n1. Manually batch calls to `write_to_offline_store()` in their application code\n2. Set up a separate stream processing pipeline (e.g., Spark with Kafka) to handle batching\n\nFor users receiving high-frequency feature updates (e.g., 100 requests/second) who want to write to both online and offline stores, this creates unnecessary complexity and can lead to inefficient writes to offline stores like BigQuery or Parquet.\n\n## Describe the solution you'd like\n\nAdd configurable automatic batching for offline store writes in the Feature Server, similar to how the Spark-Kafka processor handles batching via the `processingTime` parameter.\n\n**Proposed configuration options:**\n- `offline_write_batch_size`: Number of records to buffer before flushing to offline store\n- `offline_write_batch_interval`: Time interval (seconds) to wait before flushing to offline store\n- Flush on whichever threshold is reached first\n\n**Example configuration in `feature_store.yaml`:**\n```yaml\nonline_store:\n  type: redis\n  ...\noffline_store:\n  type: bigquery\n  ...\nfeature_server:\n  offline_write_batching:\n    enabled: true\n    batch_size: 1000\n    batch_interval_seconds: 30\n```\n\n## Describe alternatives you've considered\n\n1. **Current approach with stream processor**: Users can set up a Spark Structured Streaming job that reads from Kafka and uses `processingTime` to control batching. This works but requires additional infrastructure and complexity.\n\n2. **Manual batching in application code**: Users can implement their own buffering logic before calling `write_to_offline_store()`, but this duplicates effort across different applications.\n\n3. **Use only online store**: Skip offline store writes entirely during streaming and rely on periodic materialization from batch sources, but this loses the real-time historical logging capability.\n\n## Additional context\n\nThis feature request came from a community discussion about streaming ingestion patterns. When using `PushMode.ONLINE_AND_OFFLINE` with high-frequency updates, the lack of automatic batching can result in:\n- Excessive API calls to offline stores (one per feature update)\n- Higher costs for cloud data warehouses like BigQuery\n- Increased latency for write operations\n\n**Current implementation reference:**\nThe Spark-Kafka processor already implements batching at `sdk/python/feast/infra/contrib/spark_kafka_processor.py:134-173` using `foreachBatch`. Similar batching logic could be added to the Feature Server's push endpoint.\n\n**Benefits:**\n- Simplified architecture for users who want real-time serving + historical logging\n- Reduced costs for offline store writes\n- Better alignment with best practices for data warehouse ingestion\n- No breaking changes (feature would be opt-in via configuration)\n\nRelated code:\n- Feature Server: `sdk/python/feast/feature_store.py:1786-1828` (`write_to_offline_store`)\n- Spark batching example: `sdk/python/feast/infra/contrib/spark_kafka_processor.py:134-173`\n\ncc @feast-dev/maintainers","author":{"url":"https://github.com/devin-ai-integration[bot]","@type":"Person","name":"devin-ai-integration[bot]"},"datePublished":"2025-10-20T13:24:26.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":3},"url":"https://github.com/5683/feast/issues/5683"}