René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Wire ComputeEngine.get_historical_features() into the standard retrieval path to replace per-store BFV transformation duplication","articleBody":"**Background**\n\nFeast currently has two separate mechanisms for applying BatchFeatureView (BFV) transformations during historical feature retrieval:\n\n1. **Compute Engine DAG path** (SparkComputeEngine.get_historical_features(), RayComputeEngine.get_historical_features()) — uses FeatureBuilder + ExecutionPlan with proper node sequencing: source → transform → filter → dedup. This path already handles BFV transformations correctly for both materialization and retrieval.\n\n2. **Offline Store path** (passthrough_provider.get_historical_features() → offline_store.get_historical_features()) — the standard path invoked by FeatureStore.get_historical_features(). This path bypasses the compute engine entirely.\n\nBecause of this split, each offline store that wants BFV compute-on-read during historical retrieval must re-implement transformation detection and application logic independently:\n\n- RayOfflineStore.get_historical_features() implements it via resolve_feature_view_source_with_fallback() + map_batches()\n- [PR #6357](https://github.com/feast-dev/feast/pull/6357) adds a parallel implementation for SparkOfflineStore\n\nThis means transformation logic is duplicated across stores rather than living in the shared compute engine layer.\n\n**Root Cause**\n\npassthrough_provider.get_historical_features() always delegates to self.offline_store, never to self.batch_engine:\n\n```\n# sdk/python/feast/infra/passthrough_provider.py\ndef get_historical_features(self, config, feature_views, ...):\n    job = self.offline_store.get_historical_features(...)  # batch_engine never called\n    return job\n```\n\nNote the contrast with materialization, which correctly routes through the engine:\n\n`jobs = self.batch_engine.materialize(registry, task)`\n\n**Blocking Interface Mismatch**\n\nThe compute engine cannot be directly substituted today because of a fundamental impedance mismatch:\n\n- passthrough_provider.get_historical_features() receives feature_views: List[FeatureView] — multiple views for a single multi-view PIT join\n- HistoricalRetrievalTask (the compute engine's input) has feature_view: Union[BatchFeatureView, StreamFeatureView] — a single view\n\n```\n# sdk/python/feast/infra/common/retrieval_task.py\n@dataclass\nclass HistoricalRetrievalTask:\n    feature_view: Union[BatchFeatureView, StreamFeatureView]  # singular\n    entity_df: Union[pd.DataFrame, str]\n    ...\n```\n\nThe compute engine is designed for per-view DAG execution. The PIT join logic that combines multiple views into one entity-aligned training dataset lives in each offline store's SQL template, not in the compute engine.\n\n**Proposed Solution**\n\nThis is a multi-step refactor:\n\n**Step 1** — Extend HistoricalRetrievalTask to support multiple feature views:\n\n```\n@dataclass\nclass HistoricalRetrievalTask:\n    feature_views: List[Union[BatchFeatureView, FeatureView]]  # plural\n    entity_df: Union[pd.DataFrame, str]\n    feature_refs: List[str]\n```\n\n**Step 2** — Move PIT join coordination into ComputeEngine.get_historical_features():\n\nThe base ComputeEngine should orchestrate per-view DAG execution across all requested feature views and join the results back to entity_df — making the PIT join logic engine-owned rather than SQL-template-owned.\n\n**Step 3** — Wire the provider to call batch_engine for historical retrieval:\n\n```\n# passthrough_provider.py\ndef get_historical_features(self, config, feature_views, feature_refs, entity_df, ...):\n    task = HistoricalRetrievalTask(\n        feature_views=feature_views,\n        feature_refs=feature_refs,\n        entity_df=entity_df,\n        ...\n    )\n    return self.batch_engine.get_historical_features(registry, task)\n```\n\n**Step 4** — Retire per-store transformation logic:\n\nOnce the compute engine owns this path, the custom BFV transformation code in RayOfflineStore and SparkOfflineStore can be removed. All stores get transformation support for free through the engine's FeatureBuilder.\n","author":{"url":"https://github.com/ntkathole","@type":"Person","name":"ntkathole"},"datePublished":"2026-05-02T06:09:24.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/6359/feast/issues/6359"}