René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Add support for OpenLineage","articleBody":"**Is your feature request related to a problem? Please describe.**\nFeast currently manages ML features, their definitions, and their retrieval for training and serving models, but does **not emit or capture standardized lineage events** according to the OpenLineage spec. This means:\n\n* There is **limited visibility into how Feast feature pipelines transform and move data** across sources, feature views, and serving layers.\n* Teams using Feast cannot easily integrate with lineage and observability tools that rely on OpenLineage for tracing dataset dependencies, run metadata, and pipeline audit history.\n* Without lineage events, it’s difficult to tie feature generation to downstream model training and production inference in a standardized ecosystem of metadata tools.\n\n\u003e OpenLineage is an open framework and specification for capturing lineage metadata about jobs, runs, and datasets — enabling consistent lineage collection and analysis across tools and pipelines. ([[openlineage.io](https://openlineage.io/)][1])\n\n**Describe the solution you'd like**\n---\n\n### 💡 **Describe the solution you’d like**\n\nAdd first-class support for **OpenLineage event emission** within Feast to enable standardized lineage capture for feature generation, ingestion, and retrieval.\n\n#### Key components of the solution\n\n1. **OpenLineage client integration**\n\n   * Include an OpenLineage client library (Python) that can construct and emit lineage events for Feast pipelines.\n   * Events should cover key Feast activities such as:\n\n     * Feature view materialization runs (batch/stream)\n     * Historical feature retrieval for training dataset generation\n     * Online feature retrievals (as optional lineage events)\n   * Construct core OpenLineage entities (job/run/dataset) using Feast concept metadata.\n\n2. **Instrument Feast pipelines**\n\n   * Hook event emission into Feast orchestration workflows (e.g., `feast materialize`, SDK workflows).\n   * Capture dataset inputs/outputs:\n\n     * Offline store table/partition references used as inputs\n     * Feature view outputs materialized in offline/online stores\n\n3. **Custom OpenLineage facets (optional but recommended)**\n\n   * Add Feast-specific facets (e.g., `feast_feature_view_facet`) to enrich dataset and job metadata with feature definition details.\n   * Follow OpenLineage custom facet naming and schema guidelines. ([[openlineage.io](https://openlineage.io/docs/spec/facets/?utm_source=chatgpt.com)][2])\n\n4. **Backend support**\n\n   * Provide configuration for sending OpenLineage events to supported backends (HTTP, Kafka, Marquez, etc.).\n\n---\n**Describe alternatives you've considered**\n* **Ad hoc lineage export scripts** — Users can write custom tooling to snapshot pipelines and log lineage manually, but this is error prone and non-standard.\n* **External orchestration lineage only** — Rely on orchestrators like Airflow to produce lineage, but this fails to capture lineage inside Feast’s core feature logic and misses feature store semantics.\n* **Out-of-band dumps** — Export Feast registry/feature metadata offline, but does not capture temporal run lineage.\n\n---\n\n**Additional context**\n\n* **OpenLineage spec** provides a standard API for capturing lineage events (jobs/runs/datasets) and is extensible via facets. ([[openlineage.io](https://openlineage.io/docs/spec/facets/?utm_source=chatgpt.com)][2])\n* Capturing lineage benefits model governance, debugging, root-cause analysis, and data impact studies across ML systems.\n* Possible integration patterns:\n\n  * Add an optional `--openlineage` flag to commands like `feast materialize`.\n  * Emit lineage events in Feast SDK flows when users retrieve features programmatically.\n  * Expose configuration options in `feature_store.yaml` for lineage backend endpoints.\n\n---\n\n[1]: https://openlineage.io/ \"Home | OpenLineage\"\n[2]: https://openlineage.io/docs/spec/facets/?utm_source=chatgpt.com \"Facets \u0026 Extensibility\"\n\n","author":{"url":"https://github.com/franciscojavierarceo","@type":"Person","name":"franciscojavierarceo"},"datePublished":"2026-01-21T15:16:02.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/5882/feast/issues/5882"}