Title: SavedDatasetStorage registry: custom_storage is claimed by three classes, so from_proto dispatch depends on import order · Issue #6946 · feast-dev/feast · GitHub
Open Graph Title: SavedDatasetStorage registry: custom_storage is claimed by three classes, so from_proto dispatch depends on import order · Issue #6946 · feast-dev/feast
X Title: SavedDatasetStorage registry: custom_storage is claimed by three classes, so from_proto dispatch depends on import order · Issue #6946 · feast-dev/feast
Description: Problem _StorageRegistry keys storage classes by _proto_attr_name in a plain dict: # sdk/python/feast/saved_dataset.py class _StorageRegistry(type): classes_by_proto_attr_name: Dict[str, Type["SavedDatasetStorage"]] = {} def __new__(cls,...
Open Graph Description: Problem _StorageRegistry keys storage classes by _proto_attr_name in a plain dict: # sdk/python/feast/saved_dataset.py class _StorageRegistry(type): classes_by_proto_attr_name: Dict[str, Type["Save...
X Description: Problem _StorageRegistry keys storage classes by _proto_attr_name in a plain dict: # sdk/python/feast/saved_dataset.py class _StorageRegistry(type): classes_by_proto_attr_name: Dict[str, Type["...
Opengraph URL: https://github.com/feast-dev/feast/issues/6946
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"SavedDatasetStorage registry: custom_storage is claimed by three classes, so from_proto dispatch depends on import order","articleBody":"## Problem\n\n`_StorageRegistry` keys storage classes by `_proto_attr_name` in a plain dict:\n\n```python\n# sdk/python/feast/saved_dataset.py\nclass _StorageRegistry(type):\n classes_by_proto_attr_name: Dict[str, Type[\"SavedDatasetStorage\"]] = {}\n\n def __new__(cls, name, bases, dct):\n kls = type.__new__(cls, name, bases, dct)\n if dct.get(\"_proto_attr_name\"):\n cls.classes_by_proto_attr_name[dct[\"_proto_attr_name\"]] = kls\n return kls\n```\n\nThree classes claim the same key:\n\n| file | line | class |\n|---|---|---|\n| `contrib/couchbase_offline_store/couchbase_source.py` | 318 | `SavedDatasetCouchbaseStorage` |\n| `contrib/clickhouse_offline_store/clickhouse_source.py` | 161 | `SavedDatasetClickhouseStorage` |\n| `contrib/postgres_offline_store/postgres_source.py` | 221 | `SavedDatasetPostgreSQLStorage` |\n\nSince the assignment is unconditional, the last class whose module is imported wins, and `SavedDatasetStorage.from_proto` dispatches every `custom_storage` proto to that one:\n\n```python\n@staticmethod\ndef from_proto(storage_proto: SavedDatasetStorageProto) -> \"SavedDatasetStorage\":\n proto_attr_name = cast(str, storage_proto.WhichOneof(\"kind\"))\n return _StorageRegistry.classes_by_proto_attr_name[proto_attr_name].from_proto(\n storage_proto\n )\n```\n\nSo a saved dataset written against Postgres can be read back as a Clickhouse or Couchbase storage, depending only on which offline store modules happened to be imported first.\n\nThe other eight `_proto_attr_name` values are each claimed exactly once, so `custom_storage` is the only colliding key.\n\n## Scope of what I verified\n\nRead from source: the three claims, the dict assignment, and the `from_proto` dispatch.\n\nI could not demonstrate the overwrite end to end, because only `postgres` of the three has its optional dependency installable in my environment — importing the other two fails on `clickhouse_connect` / couchbase. With postgres alone:\n\n```\nimported SavedDatasetPostgreSQLStorage -> custom_storage now = SavedDatasetPostgreSQLStorage\n```\n\nIn practice the collision is latent rather than active, since a deployment rarely installs more than one of these three. That is presumably why it has not surfaced.\n\n## Why I am raising it\n\n`DataSource.CustomSourceOptions` is the only extensible slot in the `SavedDatasetStorage` oneof:\n\n```protobuf\nmessage SavedDatasetStorage {\n oneof kind {\n DataSource.FileOptions file_storage = 4;\n DataSource.BigQueryOptions bigquery_storage = 5;\n DataSource.RedshiftOptions redshift_storage = 6;\n DataSource.SnowflakeOptions snowflake_storage = 7;\n DataSource.TrinoOptions trino_storage = 8;\n DataSource.SparkOptions spark_storage = 9;\n DataSource.CustomSourceOptions custom_storage = 10;\n DataSource.AthenaOptions athena_storage = 11;\n }\n}\n```\n\nSo any new `CUSTOM_SOURCE`-based data source that wants saved-dataset support has to either add a fourth claim to a key that already does not round-trip reliably, or add a new proto field. I ran into this adding a `SavedDatasetStorage` for a Lance source (#6945) and chose to defer rather than make the collision worse.\n\nNote that `DataSource` itself does not have this problem: it carries `data_source_class_type` alongside `custom_options` and resolves the concrete class from that string, which is exactly the discriminator `SavedDatasetStorage` lacks.\n\n## Possible directions\n\n1. Give `SavedDatasetStorage` the discriminator `DataSource` already has, resolving the class from a stored class path instead of from the oneof field name, and keep the registry for backwards compatibility.\n2. Give each custom storage its own proto field, which does not scale but is explicit.\n3. Make the registry reject a duplicate claim, which would turn this into an import-time error and surface it immediately. That alone is breaking for the three existing classes, so it would need (1) or (2) first.\n\nHappy to send a PR for whichever direction maintainers prefer. I would lean to (1) for symmetry with `DataSource`.\n","author":{"url":"https://github.com/haoxu0","@type":"Person","name":"haoxu0"},"datePublished":"2026-10-05T20:31:36Z","url":"https://github.com/feast-dev/feast/issues/6946"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:c288a65e-0fef-ccf1-25a5-47b54cfa67f0 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | D90E:F21AC:2AEE1B:3BA4CA:6AFE9814 |
| html-safe-nonce | b8e436885b440ee458dc48c7fb4fd63856f576f9c9c35b25bf770718721829e0 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJEOTBFOkYyMUFDOjJBRUUxQjozQkE0Q0E6NkFGRTk4MTQiLCJ2aXNpdG9yX2lkIjoiNTQyMDk3NTEwMjgwNjEwNDA4NCIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 436d853c61fb23828dbe7c0dd907fd0cd75541a470566a59b7982f4c005b5e48 |
| hovercard-subject-tag | issue:5718521941 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/feast-dev/feast/6946/issue_layout |
| twitter:image | https://opengraph.githubassets.com/7d7947ac3f850145f5fd79d70c6ecf64b7f516928eaafdcd05ae791d66ca1f93/feast-dev/feast/issues/6946 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/7d7947ac3f850145f5fd79d70c6ecf64b7f516928eaafdcd05ae791d66ca1f93/feast-dev/feast/issues/6946 |
| og:image:alt | Problem _StorageRegistry keys storage classes by _proto_attr_name in a plain dict: # sdk/python/feast/saved_dataset.py class _StorageRegistry(type): classes_by_proto_attr_name: Dict[str, Type["Save... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | haoxu0 |
| hostname | github.com |
| expected-hostname | github.com |
| None | b3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86 |
| turbo-cache-control | no-preview |
| go-import | github.com/feast-dev/feast git https://github.com/feast-dev/feast.git |
| octolytics-dimension-user_id | 57027613 |
| octolytics-dimension-user_login | feast-dev |
| octolytics-dimension-repository_id | 161133770 |
| octolytics-dimension-repository_nwo | feast-dev/feast |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 161133770 |
| octolytics-dimension-repository_network_root_nwo | feast-dev/feast |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 4a10310af51a1bfffbedc4fd81cb582acf0cb0ff |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width