René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Unify Feature Transformations and Feature Views","articleBody":"# Problem\n\nIn Feast, the current On Demand Feature View executes feature transformations at read time. However, this behavior is not immediately obvious from the name \"On Demand Feature View.\" \n\nThe terms \"On Demand\" don't explicitly convey when the transformation occurs, leading to potential confusion for users trying to understand the timing and execution flow of feature transformations. Additionally, there's inconsistency in how different types of feature views (batch, streaming, on-demand) are declared and used, which can make the codebase less intuitive and harder to maintain.\n\n# Proposed Solution\nI propose we refactor and rename the way transformations are defined in Feast to make the execution timing explicit. Introducing a `@transform` decorator with a type parameter that accepts a `FeastTransformation` enum can achieve this clarity. The enum would define the transformation types:\n\n```python\nfrom enum import Enum\n\nclass FeastTransformation(Enum):\n    NONE = 0         # No transformations happening using Feast\n    ON_READ = 1      # Equivalent to the current On Demand Feature View\n    ON_WRITE = 2     # New: Transformations executed at write time\n    BATCH = 3        # New: Batch processing transformations\n    STREAMING = 4    # Migrates the existing stream_feature_view decorator\n\n@transform(\n    type=FeastTransformation.ON_WRITE,\n    schema=[...],\n    entity=[...],\n    sources=[...],\n    mode='python',\n)\ndef my_feature_view(inputs: List[Dict[str, float]]) -\u003e List[Dict[str, float]]:\n    # Transformation logic here\n    return outputs\n```\n\nBy explicitly declaring when the transformation should occur `{'ON_READ', 'ON_WRITE', 'BATCH', 'STREAMING'}`, the code becomes more readable, maintainable, and, more importantly, we will be able to remove a lot of duplicate, confusing code that obfuscates what is actually happening (particularly during the execution of `FeatureStore.apply()`).\n\n**This approach unifies the declaration of feature views and transformations, making it easier for users to understand and for developers to extend functionality in the future.*\n\n# Alternatives\n\n* Enhancing Documentation: Improving the existing documentation to better explain the behavior of On Demand Feature View without changing the code structure. \n  - This doesn't solve the inherent ambiguity in the naming.\n\n* Renaming Existing Decorators: Simply renaming on_demand_feature_view to something like on_read_feature_view to make the execution timing clearer. \n  - While this improves clarity, it doesn't address the inconsistency in how different feature views are declared.\n* Using Configuration Parameters: Adding parameters to existing decorators to specify execution timing. \n  - This could make the decorators more complex and doesn't provide a unified approach.\n\n# Additional Context\n**Consistency and Extensibility**: This change promotes a consistent method of declaring feature transformations, regardless of when they execute. It also sets a foundation for adding more transformation types in the future.\n\n**Clarity for New Users**: Making the execution timing explicit in the code helps new users understand the flow of data and transformations in Feast, reducing the learning curve.\n\n**Impact on Existing Codebase**: Careful consideration and planning would be needed to migrate existing code to the new structure without breaking functionality. Deprecation warnings and migration guides could facilitate this transition.\n\n**Community Feedback**: Engaging with the community to gather feedback on this proposed change could provide additional insights and help refine the solution.\n\n**Migration**: We can make the code backwards compatible and even provide some helper scripts to migrate to the new syntax but I think this is the right long term proposal that settles the several discussions we've had.\n\nhttps://github.com/feast-dev/feast/issues/4376\nhttps://github.com/feast-dev/feast/issues/4277\nhttps://github.com/feast-dev/feast/issues/3639\n\n\n[Document here](https://docs.google.com/document/d/1KXCXcsXq1bUvbSpfhnUjDSsu4HpuUZ5XiZoQyltCkvo/edit?tab=t.0#heading=h.g6j9jyjkf5sm)","author":{"url":"https://github.com/franciscojavierarceo","@type":"Person","name":"franciscojavierarceo"},"datePublished":"2024-09-28T09:44:53.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":22},"url":"https://github.com/4584/feast/issues/4584"}