René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Duplicate data source in feature registry","articleBody":"## Expected Behavior \r\nRegistry protobuf should not have duplicate data source definitions.\r\n\r\n## Current Behavior\r\nEvery time we run `feast apply`, we noticed that a data source source is appended to the registry protobuf with the exact same details.\r\n\r\n```python\r\nfrom feast.protos.feast.core.Registry_pb2 import Registry as RegistryProto\r\n\r\nregistry_proto = RegistryProto()\r\n\r\nwith open('registry.db', 'rb') as f: data = f.read()\r\n\r\nprint(registry_proto.FromString(data))\r\n````\r\n\r\nThis is what the protobuf looks like after running `feast apply` 3 times in a row without making any changes to the definition.\r\n\r\n```\r\nentities {\r\n  spec {\r\n    name: \"__dummy\"\r\n    value_type: STRING\r\n    join_key: \"__dummy_id\"\r\n    project: \"myproject\"\r\n  }\r\n  meta {\r\n    created_timestamp {\r\n      seconds: 1650485581\r\n      nanos: 37674000\r\n    }\r\n    last_updated_timestamp {\r\n      seconds: 1650485581\r\n      nanos: 37723000\r\n    }\r\n  }\r\n}\r\nregistry_schema_version: \"1\"\r\nversion_id: \"743b0630-373b-405f-861d-3de738962d7c\"\r\nlast_updated {\r\n  seconds: 1650485581\r\n  nanos: 37831000\r\n}\r\nfeature_views {\r\n  spec {\r\n    name: \"IRIS\"\r\n    project: \"myproject\"\r\n    entities: \"__dummy\"\r\n    features {\r\n      name: \"PETAL_LENGTH\"\r\n      value_type: FLOAT\r\n    }\r\n    features {\r\n      name: \"PETAL_WIDTH\"\r\n      value_type: FLOAT\r\n    }\r\n    features {\r\n      name: \"SEPAL_LENGTH\"\r\n      value_type: FLOAT\r\n    }\r\n    features {\r\n      name: \"SEPAL_WIDTH\"\r\n      value_type: FLOAT\r\n    }\r\n    features {\r\n      name: \"SPECIES\"\r\n      value_type: INT64\r\n    }\r\n    ttl {\r\n      seconds: 31449600\r\n    }\r\n    batch_source {\r\n      type: BATCH_SNOWFLAKE\r\n      timestamp_field: \"EVENT_TIMESTAMP\"\r\n      created_timestamp_column: \"CREATE_TIMESTAMP\"\r\n      data_source_class_type: \"feast.infra.offline_stores.snowflake_source.SnowflakeSource\"\r\n      snowflake_options {\r\n        table: \"IRIS\"\r\n        schema: \"MY_SCHEMA\"\r\n        database: \"MY_DATABASE\"\r\n      }\r\n    }\r\n    online: true\r\n    description: \"A sample feature view containing the Iris dataset.\"\r\n  }\r\n  meta {\r\n    created_timestamp {\r\n      seconds: 1650485581\r\n      nanos: 37333000\r\n    }\r\n    last_updated_timestamp {\r\n      seconds: 1650485581\r\n      nanos: 37333000\r\n    }\r\n  }\r\n}\r\ndata_sources {\r\n  type: BATCH_SNOWFLAKE\r\n  timestamp_field: \"EVENT_TIMESTAMP\"\r\n  created_timestamp_column: \"CREATE_TIMESTAMP\"\r\n  data_source_class_type: \"feast.infra.offline_stores.snowflake_source.SnowflakeSource\"\r\n  snowflake_options {\r\n    table: \"IRIS\"\r\n    schema: \"MY_SCHEMA\"\r\n    database: \"MY_DATABASE\"\r\n  }\r\n  project: \"myproject\"\r\n}\r\ndata_sources {\r\n  type: BATCH_SNOWFLAKE\r\n  timestamp_field: \"EVENT_TIMESTAMP\"\r\n  created_timestamp_column: \"CREATE_TIMESTAMP\"\r\n  data_source_class_type: \"feast.infra.offline_stores.snowflake_source.SnowflakeSource\"\r\n  snowflake_options {\r\n    table: \"IRIS\"\r\n    schema: \"MY_SCHEMA\"\r\n    database: \"MY_DATABASE\"\r\n  }\r\n  project: \"myproject\"\r\n}\r\ndata_sources {\r\n  type: BATCH_SNOWFLAKE\r\n  timestamp_field: \"EVENT_TIMESTAMP\"\r\n  created_timestamp_column: \"CREATE_TIMESTAMP\"\r\n  data_source_class_type: \"feast.infra.offline_stores.snowflake_source.SnowflakeSource\"\r\n  snowflake_options {\r\n    table: \"IRIS\"\r\n    schema: \"MY_SCHEMA\"\r\n    database: \"MY_DATABASE\"\r\n  }\r\n  project: \"myproject\"\r\n}\r\n```\r\n## Steps to reproduce\r\n\r\nHere's the `feature_store.yaml`:\r\n```yaml\r\nproject: myproject\r\nregistry: s3://mybucket/registry.db\r\nprovider: local\r\noffline_store:\r\n  type: snowflake.offline\r\n  account: myaccount\r\n  user: myuser\r\n  password: mypasssword\r\n  role: myrole\r\n  warehouse: mywarehouse\r\n  database: mydatabase\r\nonline_store:\r\n  type: redis\r\n  connection_string: myredisconnectionstring\r\n```\r\n\r\nAnd here's the feature definition `feature_definitions/iris.py`:\r\n```python\r\nfrom datetime import timedelta\r\nfrom pathlib import Path\r\n\r\nfrom feast import Feature, FeatureView, SnowflakeSource, ValueType\r\n\r\nname = Path(__file__).stem\r\n\r\niris_source = SnowflakeSource(\r\n    schema=\"MY_SCHEMA\",\r\n    database=\"MY_DATABASE\",\r\n    table=\"IRIS\",\r\n    timestamp_field=\"EVENT_TIMESTAMP\",\r\n    created_timestamp_column=\"CREATE_TIMESTAMP\"\r\n)\r\n\r\niris = FeatureView(\r\n    name=name.upper(),\r\n    entities=[],\r\n    ttl=timedelta(weeks=52),\r\n    schema=[\r\n        Feature(name=\"PETAL_LENGTHS\", dtype=ValueType.FLOAT),\r\n        Feature(name=\"PETAL_WIDTH\", dtype=ValueType.FLOAT),\r\n        Feature(name=\"SEPAL_LENGTH\", dtype=ValueType.FLOAT),\r\n        Feature(name=\"SEPAL_WIDTH\", dtype=ValueType.FLOAT),\r\n        Feature(name=\"SPECIES\", dtype=ValueType.INT64),\r\n    ],\r\n    online=True,\r\n    source=iris_source,\r\n    description=\"A sample feature view containing the Iris dataset.\"\r\n)\r\n```\r\n\r\n### Specifications\r\n\r\n- Version: `0.2.0`\r\n- Platform: AWS EC2 (Linux2 AMI)\r\n- Subsystem:\r\n\r\n## Possible Solution\r\n","author":{"url":"https://github.com/tkch3n","@type":"Person","name":"tkch3n"},"datePublished":"2022-04-20T20:50:31.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":4},"url":"https://github.com/2581/feast/issues/2581"}