René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Error raised when planning/applying a change that inserts/removes a push source","articleBody":"## Expected Behavior \r\n\r\nThe command `feast apply` (or `plan`) is never supposed to raise an error. In particular, one may want to insert a push source into an existing repo, between a batch source and a Feature View: i.e. from (arrows here represent dependency in the feature repo definitions)\r\n```\r\nbatch_source \u003c-- feature_view\r\n```\r\nto\r\n```\r\nbatch_source \u003c-- push_source \u003c-- feature_view\r\n```\r\n\r\n## Current Behavior\r\n\r\nAn apply/plan invocation that leads Feast to compare a repo without the push source and one with the push source results in an error\r\nbeing thrown within the PushSource class, specifically in its `__eq__` method, because the \"other\" operand is None.\r\n\r\n## Steps to reproduce\r\n\r\nInstall Feast 0.24.1 and use the quickstart drivers example:\r\n```\r\nfeast init repo_name\r\ncd repo_name/feature_repo\r\nfeast apply\r\n```\r\n\r\nNow _alter the `example_repo.py`, changing the definition of `driver_stats_fresh_fv` so that its `source` parameter reads `driver_stats_source` instead of `driver_stats_push_source`: it is now\r\n```\r\ndriver_stats_fresh_fv = FeatureView(\r\n    name=\"driver_hourly_stats_fresh\",\r\n    entities=[driver],\r\n    ttl=timedelta(days=1),\r\n    schema=[\r\n        Field(name=\"conv_rate\", dtype=Float32),\r\n        Field(name=\"acc_rate\", dtype=Float32),\r\n        Field(name=\"avg_daily_trips\", dtype=Int64),\r\n    ],\r\n    online=True,\r\n    # THE FOLLOWING LINE EDITED instead of `driver_stats_push_source`:\r\n    source=driver_stats_source,  # Changed from above\r\n    tags={\"team\": \"driver_performance\"},\r\n)\r\n```\r\n\r\nFinally, try `feast apply` or `feast plan`, and the error is thrown. Stack:\r\n\r\n\r\n```\r\n  File \"/home/stefano/.virtualenvs/feast-pushsource-issue-39/bin/feast\", line 8, in \u003cmodule\u003e\r\n    sys.exit(cli())\r\n  File \"/home/stefano/.virtualenvs/feast-pushsource-issue-39/lib/python3.9/site-packages/click/core.py\", line 1130, in __call__\r\n    return self.main(*args, **kwargs)\r\n  File \"/home/stefano/.virtualenvs/feast-pushsource-issue-39/lib/python3.9/site-packages/click/core.py\", line 1055, in main\r\n    rv = self.invoke(ctx)\r\n  File \"/home/stefano/.virtualenvs/feast-pushsource-issue-39/lib/python3.9/site-packages/click/core.py\", line 1657, in invoke\r\n    return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n  File \"/home/stefano/.virtualenvs/feast-pushsource-issue-39/lib/python3.9/site-packages/click/core.py\", line 1404, in invoke\r\n    return ctx.invoke(self.callback, **ctx.params)\r\n  File \"/home/stefano/.virtualenvs/feast-pushsource-issue-39/lib/python3.9/site-packages/click/core.py\", line 760, in invoke\r\n    return __callback(*args, **kwargs)\r\n  File \"/home/stefano/.virtualenvs/feast-pushsource-issue-39/lib/python3.9/site-packages/click/decorators.py\", line 26, in new_func\r\n    return f(get_current_context(), *args, **kwargs)\r\n  File \"/home/stefano/.virtualenvs/feast-pushsource-issue-39/lib/python3.9/site-packages/feast/cli.py\", line 519, in apply_total_command\r\n    apply_total(repo_config, repo, skip_source_validation)\r\n  File \"/home/stefano/.virtualenvs/feast-pushsource-issue-39/lib/python3.9/site-packages/feast/usage.py\", line 283, in wrapper\r\n    return func(*args, **kwargs)\r\n  File \"/home/stefano/.virtualenvs/feast-pushsource-issue-39/lib/python3.9/site-packages/feast/repo_operations.py\", line 335, in apply_total\r\n    apply_total_with_repo_instance(\r\n  File \"/home/stefano/.virtualenvs/feast-pushsource-issue-39/lib/python3.9/site-packages/feast/repo_operations.py\", line 296, in apply_total_with_repo_instance\r\n    registry_diff, infra_diff, new_infra = store.plan(repo)\r\n  File \"/home/stefano/.virtualenvs/feast-pushsource-issue-39/lib/python3.9/site-packages/feast/usage.py\", line 294, in wrapper\r\n    raise exc.with_traceback(traceback)\r\n  File \"/home/stefano/.virtualenvs/feast-pushsource-issue-39/lib/python3.9/site-packages/feast/usage.py\", line 283, in wrapper\r\n    return func(*args, **kwargs)\r\n  File \"/home/stefano/.virtualenvs/feast-pushsource-issue-39/lib/python3.9/site-packages/feast/feature_store.py\", line 734, in plan\r\n    registry_diff = diff_between(\r\n  File \"/home/stefano/.virtualenvs/feast-pushsource-issue-39/lib/python3.9/site-packages/feast/diff/registry_diff.py\", line 289, in diff_between\r\n    diff_registry_objects(current_obj, e, object_type)\r\n  File \"/home/stefano/.virtualenvs/feast-pushsource-issue-39/lib/python3.9/site-packages/feast/diff/registry_diff.py\", line 142, in diff_registry_objects\r\n    if current != new:\r\n  File \"/home/stefano/.virtualenvs/feast-pushsource-issue-39/lib/python3.9/site-packages/feast/feature_view.py\", line 249, in __eq__\r\n    or self.stream_source != other.stream_source\r\n  File \"/home/stefano/.virtualenvs/feast-pushsource-issue-39/lib/python3.9/site-packages/feast/data_source.py\", line 763, in __eq__\r\n    raise TypeError(\"Comparisons should only involve PushSource class objects.\")\r\nTypeError: Comparisons should only involve PushSource class objects.\r\n```\r\n\r\n### Specifications\r\n\r\n- Version: Feast v0.24.1\r\n- Platform: Linux (ubuntu 20) with Python 3.9\r\n- Subsystem:\r\n\r\n## Possible Solution\r\n\r\nWhen comparing \"existing\" and \"desired\" repo structure, the push sources are\r\ncompared (old vs new) to check if there are changes.\r\nBut for `PushSource` (as well as for `KafkaSource`, `RequestSource` and `KinesisSource` for that matter)\r\nthe comparison breaks with the error above in their `__eq__` method: the first thing\r\nhappening in the method is \"raise error, unless `isinstance(other, PushSource)`\".\r\n\r\nThis is at odds with the superclass `DataSource`, whose `__eq__` starts with: `if other is None: return False`. If subclasses had this,\r\nno errors would be raised and the feature repo's structure could have been modified (in particular, inserting/removing\r\na push source between a batch source and a feature view, at will).\r\n\r\n","author":{"url":"https://github.com/hemidactylus","@type":"Person","name":"hemidactylus"},"datePublished":"2022-09-12T20:54:37.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":5},"url":"https://github.com/3207/feast/issues/3207"}