Title: Granular OpenLineage event emission on `feast apply` for precise FeatureView lineage · Issue #6852 · feast-dev/feast · GitHub
Open Graph Title: Granular OpenLineage event emission on `feast apply` for precise FeatureView lineage · Issue #6852 · feast-dev/feast
X Title: Granular OpenLineage event emission on `feast apply` for precise FeatureView lineage · Issue #6852 · feast-dev/feast
Description: Is your feature request related to a problem? Please describe. When openlineage.emit_on_apply: true is enabled in feature_store.yaml, running feast apply currently aggregates all defined data sources, entities, and feature views into a s...
Open Graph Description: Is your feature request related to a problem? Please describe. When openlineage.emit_on_apply: true is enabled in feature_store.yaml, running feast apply currently aggregates all defined data sourc...
X Description: Is your feature request related to a problem? Please describe. When openlineage.emit_on_apply: true is enabled in feature_store.yaml, running feast apply currently aggregates all defined data sourc...
Opengraph URL: https://github.com/feast-dev/feast/issues/6852
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Granular OpenLineage event emission on `feast apply` for precise FeatureView lineage","articleBody":"**Is your feature request related to a problem? Please describe.**\nWhen `openlineage.emit_on_apply: true` is enabled in `feature_store.yaml`, running `feast apply` currently aggregates all defined data sources, entities, and feature views into a single bulk definition job event (`feast.\u003cproject>.feature_views`).\n\nBecause OpenLineage lineage engines (including Feast's lineage consumer) infer dataset-to-dataset relationships from job inputs and outputs, placing all data sources as inputs and all feature views as outputs of the same job causes the lineage graph to form a full $\\(M \\times N\\)$ Cartesian product. Every data source appears connected to every feature view (as dashed `derived` edges), regardless of whether they are actually related in the feature store definitions.\n\nIn feature stores with multiple data sources and feature views, this makes the OpenLineage lineage graph cluttered and inaccurate for downstream consumers and the UI.\n\n**Describe the solution you'd like**\n\n`FeastOpenLineageEmitter.emit_apply()` could emit **more granular, per-FeatureView definition events** instead of a single monolithic job event. \nSpecifically:\n1. For each `FeatureView` / `StreamFeatureView` / `OnDemandFeatureView`, emit a distinct definition event where:\n - `inputs`: Only the data sources and entities declared for that specific feature view.\n - `outputs`: That specific feature view dataset.\n2. Maintain the existing pattern used for `FeatureService` definitions (which are already emitted individually per service).\nThis ensures that OpenLineage accurately captures the true 1:1 or N:1 dependency graph between sources and feature views both in Feast UI and in external lineage backends (e.g., Marquez, DataHub, Atlan).\n\n**Describe alternatives you've considered**\n\n1. **Relying on the UI's Feast Registry Lineage toggle**:\n - While the built-in \"Feast registry lineage\" view correctly parses definitions directly from the registry proto, it does not help external OpenLineage consumers/catalogs that rely on standard OpenLineage event streams emitted during CI/CD or `feast apply`.\n2. **Filtering or matching datasets in the backend processor**:\n - Having `OpenLineageProcessor` inspect facets to filter out unrelated edges is fragile. Emitting accurate event boundaries at the source (`FeastOpenLineageEmitter`) adheres more closely to OpenLineage design principles.\n\n\n**Additional context**\n\n### Technical Details\n- **Bulk emission in emitter**:\n In `sdk/python/feast/openlineage/emitter.py` within `emit_apply()`, all sources and entities are collected into `fv_inputs`, and all feature views are collected into `fv_outputs`:\n ```python\n # sdk/python/feast/openlineage/emitter.py\n result1 = self._client.emit_run_event(\n job_name=feature_views_job_name(project),\n run_id=str(uuid.uuid4()),\n event_type=RunState.COMPLETE,\n inputs=fv_inputs,\n outputs=fv_outputs,\n job_facets=job_facets,\n namespace=namespace,\n )\n ```\n- **Transitive edge derivation**:\n In `sdk/python/feast/openlineage/processor.py`, `_build_dataset_to_dataset_edges()` maps all job inputs to all job outputs:\n ```python\n # sdk/python/feast/openlineage/processor.py\n for inp in inputs:\n for out in outputs:\n self._store.upsert_lineage_edge(\n source_type=\"dataset\",\n source_name=inp[\"name\"],\n target_type=\"dataset\",\n target_name=out[\"name\"],\n edge_type=\"derived\",\n )\n ```\n\n- **Existing helper**:\n `feature_view_to_job()` and `emit_feature_view_lineage()` in `sdk/python/feast/openlineage/mappers.py` already implement per-view mapping logic, which can be reused or aligned with `emit_apply()`.\n\n","author":{"url":"https://github.com/mao-liu","@type":"Person","name":"mao-liu"},"datePublished":"2026-09-20T22:25:06Z","url":"https://github.com/feast-dev/feast/issues/6852"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:c31b9efc-7941-1b8b-b845-af11d124bc64 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | EAAE:352B6A:DFA84F:13191EA:6AFE9224 |
| html-safe-nonce | 4ed8116c9490998056cc8f286756d111e918b655980df6ae61cccdbbc3cf57d4 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJFQUFFOjM1MkI2QTpERkE4NEY6MTMxOTFFQTo2QUZFOTIyNCIsInZpc2l0b3JfaWQiOiIxODIxNDgyMjc0NjgwMzc3ODkyIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | 0031dd69800868b6bf03e9fd7a0418054297c2e5b3252ba4aaec9c4642d3dac3 |
| hovercard-subject-tag | issue:5520855244 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/feast-dev/feast/6852/issue_layout |
| twitter:image | https://opengraph.githubassets.com/aba537f2ac689cbb8511481fe6f89e2bb2b12c0ac5e8805dd66ce8d651ffadf2/feast-dev/feast/issues/6852 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/aba537f2ac689cbb8511481fe6f89e2bb2b12c0ac5e8805dd66ce8d651ffadf2/feast-dev/feast/issues/6852 |
| og:image:alt | Is your feature request related to a problem? Please describe. When openlineage.emit_on_apply: true is enabled in feature_store.yaml, running feast apply currently aggregates all defined data sourc... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | mao-liu |
| hostname | github.com |
| expected-hostname | github.com |
| None | b3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86 |
| turbo-cache-control | no-preview |
| go-import | github.com/feast-dev/feast git https://github.com/feast-dev/feast.git |
| octolytics-dimension-user_id | 57027613 |
| octolytics-dimension-user_login | feast-dev |
| octolytics-dimension-repository_id | 161133770 |
| octolytics-dimension-repository_nwo | feast-dev/feast |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 161133770 |
| octolytics-dimension-repository_network_root_nwo | feast-dev/feast |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 4a10310af51a1bfffbedc4fd81cb582acf0cb0ff |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width