René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Unable to `feast apply` when using SQL Registry","articleBody":"## Expected Behavior \r\nUsing a SQL registry, we should be able to run `feast apply` more than once for a given project, so that feast can identify the diff between the existing registry and project.\r\n\r\n## Current Behavior\r\nWhen using a SQL registry, running `feast apply` for the first time works. However, when running it again, it returns the following error:\r\n```\r\nTraceback (most recent call last):\r\n  File \"/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/bin/feast\", line 8, in \u003cmodule\u003e\r\n    sys.exit(cli())\r\n  File \"/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/click/core.py\", line 1137, in __call__\r\n    return self.main(*args, **kwargs)\r\n  File \"/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/click/core.py\", line 1062, in main\r\n    rv = self.invoke(ctx)\r\n  File \"/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/click/core.py\", line 1668, in invoke\r\n    return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n  File \"/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/click/core.py\", line 1404, in invoke\r\n    return ctx.invoke(self.callback, **ctx.params)\r\n  File \"/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/click/core.py\", line 763, in invoke\r\n    return __callback(*args, **kwargs)\r\n  File \"/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/click/decorators.py\", line 26, in new_func\r\n    return f(get_current_context(), *args, **kwargs)\r\n  File \"/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/cli.py\", line 492, in apply_total_command\r\n    apply_total(repo_config, repo, skip_source_validation)\r\n  File \"/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/usage.py\", line 274, in wrapper\r\n    return func(*args, **kwargs)\r\n  File \"/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/repo_operations.py\", line 305, in apply_total\r\n    apply_total_with_repo_instance(\r\n  File \"/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/repo_operations.py\", line 265, in apply_total_with_repo_instance\r\n    registry_diff, infra_diff, new_infra = store.plan(repo)\r\n  File \"/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/usage.py\", line 285, in wrapper\r\n    raise exc.with_traceback(traceback)\r\n  File \"/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/usage.py\", line 274, in wrapper\r\n    return func(*args, **kwargs)\r\n  File \"/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/feature_store.py\", line 708, in plan\r\n    current_infra_proto = self._registry.proto().infra.__deepcopy__()\r\n  File \"/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/infra/registry_stores/sql.py\", line 666, in proto\r\n    r.infra.CopyFrom(self.get_infra(project).to_proto())\r\n  File \"/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/infra/registry_stores/sql.py\", line 558, in get_infra\r\n    return self._get_object(\r\n  File \"/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/infra/registry_stores/sql.py\", line 774, in _get_object\r\n    raise not_found_exception(name, project)\r\nTypeError: 'NoneType' object is not callable\r\n```\r\n\r\nThis error is raised because of what looks like a bug in `SqlRegistry.get_infra()`. Indeed, `SqlRegistry.get_infra()` calls `SqlRegistry._get_object()` but passing `None` for the `not_found_exception` argument.\r\n[feast/infra/registry_stores/sql.py\r\n](https://github.com/feast-dev/feast/blob/31be8e8965444ce52c16542bf3983093514d0abb/sdk/python/feast/infra/registry_stores/sql.py#L562)\r\n```python\r\n    ...\r\n    def get_infra(self, project: str, allow_cache: bool = False) -\u003e Infra:\r\n        return self._get_object(\r\n            managed_infra,\r\n            \"infra_obj\",\r\n            project,\r\n            InfraProto,\r\n            Infra,\r\n            \"infra_name\",\r\n            \"infra_proto\",\r\n            \u003e\u003e\u003e\u003e None, \u003c\u003c\u003c\u003c\r\n        )\r\n    ...\r\n    def _get_object(\r\n        self,\r\n        table,\r\n        name,\r\n        project,\r\n        proto_class,\r\n        python_class,\r\n        id_field_name,\r\n        proto_field_name,\r\n        \u003e\u003e\u003e not_found_exception \u003c\u003c\u003c,\r\n    ):\r\n        self._maybe_init_project_metadata(project)\r\n\r\n        with self.engine.connect() as conn:\r\n            stmt = select(table).where(\r\n                getattr(table.c, id_field_name) == name, table.c.project_id == project\r\n            )\r\n            row = conn.execute(stmt).first()\r\n            if row:\r\n                _proto = proto_class.FromString(row[proto_field_name])\r\n                return python_class.from_proto(_proto)\r\n        \u003e\u003e\u003e raise not_found_exception(name, project) \u003c\u003c\u003c\r\n```\r\nSince no rows are returned for the `managed_infra` table, `_get_object` attempts to raise an error with the `not_found_exception` callable provided, but since `None` was provided, it is unable to raise the error correctly.\r\n\r\nThis seems to be hiding the actual error, which is that no rows are returned from the `managed_infra` table, which I can confirm is indeed empty in the SQL backend. I've provided my `feature_store.yaml` configuration below -- is the expectation that something should be created in the `managed_infra` table?\r\n\r\n## Steps to reproduce\r\n`feature_store.yaml`\r\n```yaml\r\nproject: feature_store\r\nprovider: local\r\nregistry:\r\n    registry_type: sql\r\n    path: postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DATABASE}\r\noffline_store:\r\n    type: snowflake.offline\r\n    account: ${SNOWFLAKE_ACCOUNT}\r\n    user: ${SNOWFLAKE_USER}\r\n    password: ${SNOWFLAKE_PASSWORD}\r\n    role: ${SNOWFLAKE_ROLE}\r\n    warehouse: ${SNOWFLAKE_WAREHOUSE}\r\n    database: ${SNOWFLAKE_DATABASE}\r\n    schema: ${SNOWFLAKE_SCHEMA}\r\nonline_store:\r\n    type: redis\r\n    redis_type: redis\r\n    connection_string: \"${REDIS_HOST}:${REDIS_PORT},password=${REDIS_PASSWORD}\"\r\n```\r\n\r\nRun the following commands:\r\n```\r\nfeast apply # works\r\nfeast apply # raises the error above\r\n```\r\n\r\n### Specifications\r\n\r\n- Version: 0.22.1\r\n- Platform: macOS\r\n- Subsystem: 12.4\r\n\r\n## Possible Solution\r\n\r\n","author":{"url":"https://github.com/chasleslr","@type":"Person","name":"chasleslr"},"datePublished":"2022-08-02T18:04:14.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":4},"url":"https://github.com/3000/feast/issues/3000"}