Title: fix: Make vector length validation reachable, schema-driven and vectorized by haoxu0 · Pull Request #6909 · feast-dev/feast · GitHub
Open Graph Title: fix: Make vector length validation reachable, schema-driven and vectorized by haoxu0 · Pull Request #6909 · feast-dev/feast
X Title: fix: Make vector length validation reachable, schema-driven and vectorized by haoxu0 · Pull Request #6909 · feast-dev/feast
Description: Fixes three of the five items in #6907. 1. Validation was unreachable from the batch path _validate_vector_features had a single call site, inside _get_feature_view_and_df_for_online_write. So: entry point before after write_to_online_store validated validated materialize not validated validated materialize_incremental not validated validated Embedding workloads are overwhelmingly batch, so the path actually carrying vectors at volume was the unvalidated one. Rather than patch each engine, this uses the one place they all funnel through: utils._convert_arrow_to_proto. The local, spark, ray, flink, kubernetes and aws_lambda engines plus PassthroughProvider all call it, so a single Arrow-level check in _convert_arrow_fv_to_proto covers all of them. The new _validate_vector_field_lengths is O(1) for fixed_size_list (compares type.list_size) and one vectorized pass for list/large_list, so it is cheap enough to leave on unconditionally. Null rows are tolerated. A non-list column carrying a declared vector_length is an error. One known gap: the snowflake compute engine builds its rows directly rather than via _convert_arrow_to_proto, so it is not covered. Happy to extend if preferred, but it needs a different seam. 2. The check assumed the vector was the first feature if feature_view.features and feature_view.features[0].vector_index: Declaring any field ahead of the vector field silently disabled validation entirely. Both validators now resolve the field with _get_feature_view_vector_field_metadata(), which scans feature_view.schema, raises on more than one vector field, and is already used in three other places in feature_store.py. There are regression tests for this on both paths. 3. iterrows() did not scale Replaced with a single pass over the column. Measured on 200k rows: before (iterrows): 1.508s after: 0.025s (~60x) The row index and both distinct failure messages are preserved. Deliberately not included #6907 items 3 and 5 are behaviour decisions rather than clear-cut fixes, so they are left for discussion there: requiring vector_length when vector_index=True (today vector_length=0 is the default and means "skip") whether Field.__eq__ should compare vector_index and vector_search_metric, which are currently commented out Testing New sdk/python/tests/unit/test_vector_length_validation.py, 16 tests across both validators: fixed-size and variable-size lists, first offending row reporting, vector field not first, unset vector_length, null rows, missing column, non-list type, RecordBatch input, and numpy vectors. No behaviour change for feature views that do not set vector_length, since 0 still short-circuits. Verified no regressions against master by running the relevant suites on both sides: test_utils.py, test_vector_store.py, test_vector_store_utils.py, test_doc_embedder.py, test_precomputed_feature_vectors.py, plus the new file — 121 passed online_store/test_online_retrieval.py and test_on_demand_python_transformation.py — identical results on both sides across 3 runs each (one pre-existing failure from Torch not being installed locally) mypy feast/utils.py feast/feature_store.py — same 2 pre-existing errors before and after ruff check and ruff format --check clean
Open Graph Description: Fixes three of the five items in #6907. 1. Validation was unreachable from the batch path _validate_vector_features had a single call site, inside _get_feature_view_and_df_for_online_write. So: ...
X Description: Fixes three of the five items in #6907. 1. Validation was unreachable from the batch path _validate_vector_features had a single call site, inside _get_feature_view_and_df_for_online_write. So: ...
Opengraph URL: https://github.com/feast-dev/feast/pull/6909
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:d5206279-19bb-e40f-d272-90350267ea03 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | B47E:3C4BAF:2B754A:3C096F:6AFE9837 |
| html-safe-nonce | 97ba222598f52d4c5b9cf19d4db199053f9ef19b11ea821a63e36c9f0817eb41 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCNDdFOjNDNEJBRjoyQjc1NEE6M0MwOTZGOjZBRkU5ODM3IiwidmlzaXRvcl9pZCI6IjQxMzcxNzgxMTQ5NjQ3NTY1MzUiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | 8985408982405cb00506344afa6d71d17e4ecdf52bef06abda4fd940682a2fc1 |
| hovercard-subject-tag | pull_request:4694108108 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/feast-dev/feast/pull/6909/files |
| twitter:image | https://avatars.githubusercontent.com/u/20422210?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/20422210?s=400&v=4 |
| og:image:alt | Fixes three of the five items in #6907. 1. Validation was unreachable from the batch path _validate_vector_features had a single call site, inside _get_feature_view_and_df_for_online_write. So: ... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | b3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/feast-dev/feast git https://github.com/feast-dev/feast.git |
| octolytics-dimension-user_id | 57027613 |
| octolytics-dimension-user_login | feast-dev |
| octolytics-dimension-repository_id | 161133770 |
| octolytics-dimension-repository_nwo | feast-dev/feast |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 161133770 |
| octolytics-dimension-repository_network_root_nwo | feast-dev/feast |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 4a10310af51a1bfffbedc4fd81cb582acf0cb0ff |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width