René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Stream transformations","articleBody":"**Note:** Users can push streaming features to the online store via [push sources](https://docs.feast.dev/reference/data-sources/push)\n**Note:** [RFC](https://docs.google.com/document/d/1UzEyETHUaGpn0ap4G82DHluiCj7zEbrQLkJJkKSv4e8/edit#) can be found here.\n**See also**: [Feast Slack: #feast-feature-transformations](https://tectonfeast.slack.com/archives/C03A2CH7GAG)\n\n## Requirements\n### Problem 1\nData scientists cannot meaningfully author event-backed features that will be fresh at serving time. \n\nOften, right now data scientists will define a feature, and then work with engineers to build that streaming pipeline to push to the online store. There is a tight coupling and engineering bottleneck to reach production. \n\n### Problem 2\nUsers cannot guarantee consistent feature transformations across batch + streaming contexts. If they author a SQL transformation in batch, and a PySpark transformation for streaming, they may have inconsistencies.\n- This problem is also related to versioning transformation logic within Feast. If both batch + stream transformation logic is present, it is easier to ensure they are consistent. An ML platform team can also build a custom feature view that only requires one transformation to be defined, but map to different semantics in batch vs streaming.\n\n### Problem 3\nUsers cannot version transformation logic and tie it to downstream models (a lineage problem). It's useful to see what models may be impacted by changing some stream transformation / aggregation logic, and perhaps allow for A/B experiments using different transformations.\n\n## Potential solution\n### Types of transformations\nWe should support defining both:\n- stateless transformations (e.g. row level transforms / mapping)\n- stateful transformations (e.g. windowed transformations)\n\n### Data scientist developer experience\nIt should be easy for a data scientist to correctly author stream transformations (including aggregations). \n- One way is to introduce a higher level DSL for aggregation logic. \n- Another way is to allow streaming some sample data in, applying the transformation, and seeing the input vs output.\n\n### Example feature view\n```python\n@stream_feature_view(\n    name=\"driver_stats\",\n    entities=[driver],\n    mode=\"spark_sql\",\n    aggregations=[\n        Aggregation(\n            column='distance',\n            function='mean', \n            time_windows=[timedelta(days=7)]   \n         )\n    ],\n    aggregation_interval=timedelta(days=1),\n    source=driver_stats_stream_source,\n)\ndef driver_stats(driver_stats_stream_source):\n    return f'''\n        SELECT\n            driver_id,\n            distance as most_recent_trip_distance,\n            duration as most_recent_trip_duration,\n            event_timestamp,\n        FROM\n            {driver_stats_stream_source}\n        '''\n```\n## Alternatives considered\n- Kicking off long running jobs that ingests events, transforms (Spark, Flink, Bytewax, etc), and writes to the online stores.","author":{"url":"https://github.com/adchia","@type":"Person","name":"adchia"},"datePublished":"2022-04-21T22:48:25.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/2597/feast/issues/2597"}