René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Corrupted registry breaks usage of feature store","articleBody":"## Expected Behavior \r\nConnecting to the feast registry and pulling features from a connected registry should work without an issue.\r\nAll feature views should be callable.\r\n\r\n## Current Behavior\r\nThe registry entry for one of four feature views seems to be corrupted. Opening a new connection succeeds for the first three features. Upon calling the fourth feature, the instantiation fails with the following error. This errors occurs regardless of the location from which we run `store = FeatureStore(config=repo_config)`.\r\n\r\n```Traceback (most recent call last):\r\n  File \"/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/bin/feast\", line 8, in \u003cmodule\u003e\r\n    sys.exit(cli())\r\n             ^^^^^\r\n  File \"/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/click/core.py\", line 1130, in __call__\r\n    return self.main(*args, **kwargs)\r\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/click/core.py\", line 1055, in main\r\n    rv = self.invoke(ctx)\r\n         ^^^^^^^^^^^^^^^^\r\n  File \"/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/click/core.py\", line 1657, in invoke\r\n    return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/click/core.py\", line [140](https://git.takeaway.com/scoober/ml-platform/poc/feast-poc/-/jobs/11952145#L140)4, in invoke\r\n    return ctx.invoke(self.callback, **ctx.params)\r\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/click/core.py\", line 760, in invoke\r\n    return __callback(*args, **kwargs)\r\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/click/decorators.py\", line 26, in new_func\r\n    return f(get_current_context(), *args, **kwargs)\r\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/feast/cli.py\", line 618, in materialize_incremental_command\r\n    store = FeatureStore(repo_path=str(repo), fs_yaml_file=fs_yaml_file)\r\n            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/feast/usage.py\", line 362, in wrapper\r\n    raise exc.with_traceback(traceback)\r\n  File \"/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/feast/usage.py\", line 348, in wrapper\r\n    return func(*args, **kwargs)\r\n           ^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/feast/feature_store.py\", line [160](https://git.takeaway.com/scoober/ml-platform/poc/feast-poc/-/jobs/11952145#L160), in __init__\r\n    self._registry = SqlRegistry(registry_config, self.config.project, None)\r\n                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/feast/infra/registry/sql.py\", line 201, in __init__\r\n    self.cached_registry_proto = self.proto()\r\n                                 ^^^^^^^^^^^^\r\n  File \"/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/feast/infra/registry/sql.py\", line 842, in proto\r\n    objs: List[Any] = lister(project)  # type: ignore\r\n                      ^^^^^^^^^^^^^^^\r\n  File \"/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/feast/infra/registry/sql.py\", line 582, in list_feature_views\r\n    return self._list_objects(\r\n           ^^^^^^^^^^^^^^^^^^^\r\n  File \"/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/feast/infra/registry/sql.py\", line 1006, in _list_objects\r\n    return [\r\n           ^\r\n  File \"/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/feast/infra/registry/sql.py\", line 1008, in \u003clistcomp\u003e\r\n    proto_class.FromString(row[proto_field_name])\r\ngoogle.protobuf.message.DecodeError: Error parsing message\r\n```\r\nExisting connections fail to pull any feature view from the feature store with:\r\n```\r\nTraceback (most recent call last):\r\n  File \\\"/usr/local/lib/python3.10/site-packages/mlserver/parallel/worker.py\\\", line 136, in _process_request    return_value = await method(\r\n  File \\\"/app/./customer-eta/eta/customer_eta.py\\\", line 90, in predict\r\n    _input = self._preprocessor.preprocess(payload)\r\n  File \\\"/app/./customer-eta/eta/steps/preprocessing.py\\\", line 28, in preprocess\r\n    feature_data = self.feature_store.get_online_features(\r\n  File \\\"/usr/local/lib/python3.10/site-packages/feast/usage.py\\\", line 288, in wrapper\r\n    return func(*args, **kwargs)\r\n  File \\\"/usr/local/lib/python3.10/site-packages/feast/feature_store.py\\\", line 1583, in get_online_features\r\n    return self._get_online_features(\r\n  File \\\"/usr/local/lib/python3.10/site-packages/feast/feature_store.py\\\", line 1605, in _get_online_features\r\n    _feature_refs = self._get_features(features, allow_cache=True)\r\n  File \\\"/usr/local/lib/python3.10/site-packages/feast/feature_store.py\\\", line 524, in _get_features\r\n    feature_service_from_registry = self.get_feature_service(\r\n  File \\\"/usr/local/lib/python3.10/site-packages/feast/usage.py\\\", line 299, in wrapper\r\n    raise exc.with_traceback(traceback)\r\n  File \\\"/usr/local/lib/python3.10/site-packages/feast/usage.py\\\", line 288, in wrapper\r\n    return func(*args, **kwargs)\r\n  File \\\"/usr/local/lib/python3.10/site-packages/feast/feature_store.py\\\", line 386, in get_feature_service\r\n    return self._registry.get_feature_service(name, self.project, allow_cache)\r\n  File \\\"/usr/local/lib/python3.10/site-packages/feast/infra/registry/sql.py\\\", line 383, in get_feature_service\r\n    self._refresh_cached_registry_if_necessary()\r\n  File \\\"/usr/local/lib/python3.10/site-packages/feast/infra/registry/sql.py\\\", line 259, in _refresh_cached_registry_if_necessary\r\n    self.refresh()\r\n  File \\\"/usr/local/lib/python3.10/site-packages/feast/infra/registry/sql.py\\\", line 238, in refresh\r\n    self.cached_registry_proto = self.proto()\r\n  File \\\"/usr/local/lib/python3.10/site-packages/feast/infra/registry/sql.py\\\", line 828, in proto\r\n    projects = self._get_all_projects()\r\n  File \\\"/usr/local/lib/python3.10/site-packages/feast/infra/registry/sql.py\\\", line 1074, in _get_all_projects\r\n    rows = conn.execute(stmt).all()\r\n  File \\\"/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py\\\", line 1385, in execute\r\n    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)\r\n  File \\\"/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/elements.py\\\", line 334, in _execute_on_connection\r\n    return connection._execute_clauseelement(\r\n  File \\\"/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py\\\", line 1577, in _execute_clauseelement\r\n    ret = self._execute_context(\r\n  File \\\"/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py\\\", line 1948, in _execute_context\r\n    self._handle_dbapi_exception(\r\n  File \\\"/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py\\\", line 2129, in _handle_dbapi_exception\r\n    util.raise_(\r\n  File \\\"/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py\\\", line 211, in raise_\r\n    raise exception\r\n  File \\\"/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py\\\", line 1905, in _execute_context\r\n    self.dialect.do_execute(\r\n  File \\\"/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py\\\", line 736, in do_execute\r\n    cursor.execute(statement, parameters)\r\n  File \\\"/usr/local/lib/python3.10/site-packages/pymysql/cursors.py\\\", line 153, in execute\r\n    result = self._query(query)\r\n  File \\\"/usr/local/lib/python3.10/site-packages/pymysql/cursors.py\\\", line 322, in _query\r\n    conn.query(q)\r\n  File \\\"/usr/local/lib/python3.10/site-packages/pymysql/connections.py\\\", line 557, in query\r\n    self._execute_command(COMMAND.COM_QUERY, sql)\r\n  File \\\"/usr/local/lib/python3.10/site-packages/pymysql/connections.py\\\", line 861, in _execute_command\r\n    self._write_bytes(packet)\r\n  File \\\"/usr/local/lib/python3.10/site-packages/pymysql/connections.py\\\", line 806, in _write_bytes\r\n    raise err.OperationalError(\r\nsqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2006, \\\"MySQL server has gone away (BrokenPipeError(32, 'Broken pipe'))\\\")\r\n[SQL: SELECT entities.entity_name, entities.project_id, entities.last_updated_timestamp, entities.entity_proto \r\nFROM entities]\r\n(Background on this error at: https://sqlalche.me/e/14/e3q8)\", \"level\": \"ERROR\"\r\n```\r\n\r\n## Steps to reproduce\r\nWe haven't been able to reproduce the error yet. We have a CI job running `feast materialize-incremental` every 5 mins, pushing the same demo data. That job started failing at some point without any changes in the data, dependencies or setup in general.\r\n\r\nAuthentication and manual querying of the registry and databases works fine.\r\n\r\n### Specifications\r\n\r\n- Version: feast == 0.31.1, python == 3.10.10\r\n- Platform: AWS, registry: RDS, online store: dynamodb, offline store: redshift\r\n- Subsystem:\r\n\r\n## Possible Solution\r\nWe dumped the feature store and recreated it running `feast apply`. This is fine for now since we are in an experimental setup. Once we move towards using feast in production, we would like to have an understanding of what is happening here and how to resolve this without dumping the whole feature store.\r\n","author":{"url":"https://github.com/LeonKolyang","@type":"Person","name":"LeonKolyang"},"datePublished":"2023-07-31T12:52:56.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/3703/feast/issues/3703"}