Title: feat(spark): Add compute-on-read support for BatchFeatureView in get_… by SIDDHESH1564 · Pull Request #6357 · feast-dev/feast · GitHub
Open Graph Title: feat(spark): Add compute-on-read support for BatchFeatureView in get_… by SIDDHESH1564 · Pull Request #6357 · feast-dev/feast
X Title: feat(spark): Add compute-on-read support for BatchFeatureView in get_… by SIDDHESH1564 · Pull Request #6357 · feast-dev/feast
Description: What this PR does / why we need it: When using @batch_feature_view with TransformationMode.PYTHON in the Spark offline store, get_historical_features() fails with UNRESOLVED_COLUMN errors. This occurs because the PIT join SQL reads directly from the raw batch_source and expects transformed feature columns (e.g., aggregated outputs) to already exist in the source data. However, BFV transformations are only executed during feast materialize, not during offline retrieval. This PR introduces compute-on-read support for BatchFeatureView in SparkOfflineStore. Before generating the PIT join SQL, BFVs with a UDF are detected and their transformations are applied: Read the raw source into a Spark DataFrame Invoke the BFV's feature_transformation.udf() (same function used during materialization) Register the transformed DataFrame as a Spark temporary view Replace the table_subquery in the query context with the temp view name This enables reuse of BFV definitions during offline training without requiring pre-materialization or external ETL pipelines. The entire pipeline remains fully distributed in Spark. Which issue(s) this PR fixes: Fixes #6345 Checks I've made sure the tests are passing. My commits are signed off (git commit -s) My PR title follows conventional commits format Testing Strategy Unit tests Integration tests Manual tests Testing is not required for this change Added 7 unit tests covering: BFV with UDF → table_subquery replaced with temp view UDF invoked with source DataFrame Transformed DataFrame registered as temp view Plain FeatureView passes through unchanged BFV without UDF passes through unchanged Mixed BFV + plain FeatureView scenarios All non-transformation context fields preserved Misc Changes: sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark.py: Added BatchFeatureView import Added _apply_bfv_transformations() helper function Integrated call into get_historical_features() between query context construction and PIT join SQL generation sdk/python/tests/unit/infra/offline_stores/contrib/spark_offline_store/test_spark_bfv_compute_on_read.py (new): 7 unit tests for compute-on-read behavior
Open Graph Description: What this PR does / why we need it: When using @batch_feature_view with TransformationMode.PYTHON in the Spark offline store, get_historical_features() fails with UNRESOLVED_COLUMN errors. This occ...
X Description: What this PR does / why we need it: When using @batch_feature_view with TransformationMode.PYTHON in the Spark offline store, get_historical_features() fails with UNRESOLVED_COLUMN errors. This occ...
Opengraph URL: https://github.com/feast-dev/feast/pull/6357
X: @github
Domain: github.com
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:9271e972-05a4-cb45-0da6-e10d4b91ce5e |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | E368:B5AA6:1EA4DDD:29C4580:6A4F2B56 |
| html-safe-nonce | 25d7218c96feb74b720d3c7dccdec5d6b847dce8c557ae0c6987f42a6278c660 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJFMzY4OkI1QUE2OjFFQTREREQ6MjlDNDU4MDo2QTRGMkI1NiIsInZpc2l0b3JfaWQiOiI1Mzg0NTI3NDgyMDM5Nzc0MDM4IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | 32261464d4cd9fcf4d363633cbcff7a2ef055e7720fd0554d96357e2eccd17b4 |
| hovercard-subject-tag | pull_request:3615640496 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/feast-dev/feast/pull/6357/files |
| twitter:image | https://avatars.githubusercontent.com/u/91747482?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/91747482?s=400&v=4 |
| og:image:alt | What this PR does / why we need it: When using @batch_feature_view with TransformationMode.PYTHON in the Spark offline store, get_historical_features() fails with UNRESOLVED_COLUMN errors. This occ... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | b92d11c0aa4a77d54ef4af1078b6a15fb5a70a215b30c4ecf28889d5a8e656d9 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/feast-dev/feast git https://github.com/feast-dev/feast.git |
| octolytics-dimension-user_id | 57027613 |
| octolytics-dimension-user_login | feast-dev |
| octolytics-dimension-repository_id | 161133770 |
| octolytics-dimension-repository_nwo | feast-dev/feast |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 161133770 |
| octolytics-dimension-repository_network_root_nwo | feast-dev/feast |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 2b8f23afb982271f1b22258a94aede67a6b77760 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width