Title: fix: Record which storage class wrote a custom_storage payload by haoxu0 · Pull Request #6988 · feast-dev/feast · GitHub
Open Graph Title: fix: Record which storage class wrote a custom_storage payload by haoxu0 · Pull Request #6988 · feast-dev/feast
X Title: fix: Record which storage class wrote a custom_storage payload by haoxu0 · Pull Request #6988 · feast-dev/feast
Description: What this PR does / why we need it: Three SavedDatasetStorage subclasses declare _proto_attr_name = "custom_storage": $ git grep -n '_proto_attr_name = "custom_storage"' sdk/python/feast/infra/offline_stores/contrib/clickhouse_offline_store/clickhouse_source.py:161 sdk/python/feast/infra/offline_stores/contrib/couchbase_offline_store/couchbase_source.py:318 sdk/python/feast/infra/offline_stores/contrib/postgres_offline_store/postgres_source.py:221 _StorageRegistry keys a flat dict on that name and the last definition wins (saved_dataset.py:26,31), so SavedDatasetStorage.from_proto resolved custom_storage to whichever module was imported last rather than to the class that wrote the payload. Writing a Postgres saved dataset and reading it back, with Couchbase also installed: wrote : SavedDatasetPostgreSQLStorage table_ref=public.my_saved_dataset oneof kind : custom_storage read back : raised KeyError: 'database' Reordering the three imports changes which class wins: postgres -> clickhouse -> couchbase -> SavedDatasetCouchbaseColumnarStorage couchbase -> postgres -> clickhouse -> SavedDatasetClickhouseStorage clickhouse -> couchbase -> postgres -> SavedDatasetPostgreSQLStorage Content cannot be used to disambiguate, because PostgreSQL and Clickhouse serialize identical configurations: postgres {name, query, table} clickhouse {name, query, table} couchbase {name, query, database, scope, collection} SavedDatasetStorage now records its concrete class path in the CustomSourceOptions.configuration JSON and resolves that on read through the existing get_saved_dataset_storage_class_from_path. The proto documents that field as owned by the implementer: Serialized configuration information for the data source. The implementer of the custom data source is responsible for serializing and deserializing data from bytes and every claimant already reads it by explicit key, so the extra key is ignored by older readers. This mirrors how DataSource.from_proto already handles CUSTOM_SOURCE, via data_source_class_type (data_source.py:343), without a proto change. Which issue(s) this PR fixes: Fixes #6946 Behavior this does not fix Payloads written before this key existed still cannot be disambiguated: the bytes of the competing classes are identical and the information was never stored, so no reader can recover it. Those now raise naming the competing classes rather than silently mis-dispatching. A field with a single claimant is unaffected and keeps reading untagged payloads, so existing single-extra installs do not change behavior. Checks I've made sure the tests are passing. My commits are signed off (git commit -s) My PR title follows conventional commits format Testing Strategy Unit tests Integration tests Manual tests Testing is not required for this change sdk/python/tests/unit/test_saved_dataset_storage_dispatch.py, 8 tests: the three real in-tree classes round-trip to themselves (parametrized, importorskip per extra) two synthetic claimants sharing the field both round-trip, including the one the flat registry does not point at the written proto records the concrete class and preserves the class's own payload an untagged payload with several claimants names them all an unresolvable recorded class path falls back instead of raising an import error a field with one claimant is unchanged The synthetic classes are defined in a real module inserted into sys.modules so the recorded path resolves, and a fixture restores the process-global registry afterwards. Verified against master: ruff check and ruff format --check clean, and mypy feast reports the same 24 pre-existing errors as origin/master with none added. Full tests/unit run differs from master by 9 failures that all reproduce on a clean origin/master when those modules are run in isolation (milvus, chronon HTTP retry, metrics httpd).
Open Graph Description: What this PR does / why we need it: Three SavedDatasetStorage subclasses declare _proto_attr_name = "custom_storage": $ git grep -n '_proto_attr_name = "custom_storage"'...
X Description: What this PR does / why we need it: Three SavedDatasetStorage subclasses declare _proto_attr_name = "custom_storage": $ git grep -n '_proto_attr_name = "custom_st...
Opengraph URL: https://github.com/feast-dev/feast/pull/6988
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:9f91769e-b78c-a42a-8b23-46541ba10124 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | 8774:79CF9:32AF1A:435A11:6AFE98AB |
| html-safe-nonce | b09a1c3cdf44b213c2b5afb76fbdb4de988cf2eb3d22f6bb8375bdb7bf36d06b |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI4Nzc0Ojc5Q0Y5OjMyQUYxQTo0MzVBMTE6NkFGRTk4QUIiLCJ2aXNpdG9yX2lkIjoiNzkwNzQ3MTMyMTMyMjEzMzY3NSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | e5ee993c631bb2dcd22c586974833043b00394a09fdfa35a2f4319cb7d030ddb |
| hovercard-subject-tag | pull_request:4804785269 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/feast-dev/feast/pull/6988/files |
| twitter:image | https://avatars.githubusercontent.com/u/20422210?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/20422210?s=400&v=4 |
| og:image:alt | What this PR does / why we need it: Three SavedDatasetStorage subclasses declare _proto_attr_name = "custom_storage": $ git grep -n '_proto_attr_name = "custom_storage"'... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | b3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/feast-dev/feast git https://github.com/feast-dev/feast.git |
| octolytics-dimension-user_id | 57027613 |
| octolytics-dimension-user_login | feast-dev |
| octolytics-dimension-repository_id | 161133770 |
| octolytics-dimension-repository_nwo | feast-dev/feast |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 161133770 |
| octolytics-dimension-repository_network_root_nwo | feast-dev/feast |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 4a10310af51a1bfffbedc4fd81cb582acf0cb0ff |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width