René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"BFV Compute-on-Read for get_historical_features() in SparkOfflineStore","articleBody":"**Is your feature request related to a problem? Please describe.**\nWhen using @batch_feature_view with TransformationMode.PYTHON and the Spark offline store, calling store.get_historical_features() fails with UNRESOLVED_COLUMN errors. The PIT join SQL template reads directly from the raw batch_source and expects output columns (e.g., aggregated features) to exist in the source data. The BFV's Python transformation is never invoked during offline reads — only during feast materialize.\nThis forces users to either:\n- Maintain a separate ETL pipeline that pre-computes the same features the BFV defines\n- Use plain FeatureView pointing at pre-computed data, duplicating transformation logic\nThis breaks the \"define once, use everywhere\" promise of the feature store.\n\n**Describe the solution you'd like**\nIn SparkOfflineStore.get_historical_features(), before building the PIT join SQL, detect BatchFeatureView instances with a UDF. For each:\n1. Read the raw source into a Spark DataFrame\n2. Invoke the BFV's udf() function (same as SparkTransformationNode.execute() does during materialization)\n3. Register the transformed DataFrame as a Spark temp view\n4. Replace the table_subquery in the FeatureViewQueryContext with the temp view name\nThis makes the entire pipeline distributed Spark: raw read -\u003e transformation -\u003e PIT join -\u003e training data. No code duplication required.\n\n**Describe alternatives you've considered**\n\n- Pre-compute features via an external Spark job and use plain FeatureView (works but duplicates logic)\n- Set offline=True on BFVs and rely on materialized offline parquet (requires running feast materialize before training, adds operational complexity)\n- Use on_demand_feature_view for transformations (doesn't support Spark-native aggregations like groupBy)\n\n**Additional context**\nAdd any other context or screenshots about the feature request here.\n","author":{"url":"https://github.com/abhijeet-dhumal","@type":"Person","name":"abhijeet-dhumal"},"datePublished":"2026-04-29T12:37:06.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/6345/feast/issues/6345"}