Title: Add upstream FeatureView lineage support for PushSource · Issue #6839 · feast-dev/feast · GitHub
Open Graph Title: Add upstream FeatureView lineage support for PushSource · Issue #6839 · feast-dev/feast
X Title: Add upstream FeatureView lineage support for PushSource · Issue #6839 · feast-dev/feast
Description: Is your feature request related to a problem? Please describe. In many real-world production ML architectures, derived features are computed outside of Feast by external stream or batch computation engines. These external pipelines read ...
Open Graph Description: Is your feature request related to a problem? Please describe. In many real-world production ML architectures, derived features are computed outside of Feast by external stream or batch computation...
X Description: Is your feature request related to a problem? Please describe. In many real-world production ML architectures, derived features are computed outside of Feast by external stream or batch computation...
Opengraph URL: https://github.com/feast-dev/feast/issues/6839
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Add upstream FeatureView lineage support for PushSource","articleBody":"**Is your feature request related to a problem? Please describe.**\n\nIn many real-world production ML architectures, derived features are computed outside of Feast by external stream or batch computation engines. These external pipelines read upstream features from existing Feast `FeatureView`s and push the transformed features into Feast via a `PushSource` backed by an offline batch store.\n\nCurrently, Feast's static registry lineage only tracks:\n```\nPushSource --> Target FeatureView\n```\nBecause `PushSource` does not support declaring the upstream `FeatureView`s it was computed from, the relationship between the upstream feature views and the push source is lost in both the Feast UI Lineage graph and the registry metadata.\n\n**Describe the solution you'd like**\n\nWe would like `PushSource` to support declaring upstream `FeatureView` dependencies via a `source_views` (or `upstream_feature_views`) parameter:\n\n```python\n# 1. Upstream feature views\nuser_tx_fv = FeatureView(name=\"user_transaction_stats\", ...)\nuser_credit_fv = FeatureView(name=\"user_credit_profile\", ...)\n# 2. PushSource declaring upstream views and offline batch source\nrisk_push_source = PushSource(\n name=\"risk_calc_pipeline\",\n batch_source=FileSource(path=\"data/risk_offline.parquet\"),\n source_views=[user_tx_fv, user_credit_fv], # \u003c-- Upstream dependencies\n description=\"External Spark job computing risk scores from transaction & credit features\",\n)\n# 3. Target feature view receiving the pushed features\nuser_risk_fv = FeatureView(\n name=\"user_risk_target_fv\",\n entities=[user_entity],\n ttl=timedelta(days=30),\n source=risk_push_source,\n schema=[Field(name=\"risk_score\", dtype=Float32)],\n)\n```\n\nWith this metadata, Feast can render the complete directed graph in the Feast UI Lineage view:\n\n```\n[ user_transaction_stats ] (FeatureView) ──┐\n ├──→ [ risk_calc_pipeline ] (PushSource) ──→ [ user_risk_target_fv ] (FeatureView)\n[ user_credit_profile ] (FeatureView) ──┘\n```\n\nProposed Implementation Details\n\n1. Protobuf (protos/feast/core/DataSource.proto): Add upstream_feature_views to PushOptions:\n```\nmessage PushOptions {\n reserved 1;\n repeated string upstream_feature_views = 2;\n}\n```\n2. Python SDK (sdk/python/feast/data_source.py): Update PushSource constructor, `to_proto()`, and `from_proto()` to accept and serialize `source_views: Optional[List[Union[BaseFeatureView, str]]]`.\n3. Registry Lineage (sdk/python/feast/lineage/registry_lineage.py): Update `RegistryLineageGenerator._parse_direct_relationships()` to emit `EntityRelation(source=FeatureView, target=DataSource)` for each upstream feature view defined on a PushSource.\n4. UI Lineage Parser (ui/src/parsers/parseEntityRelationships.ts): Extract `upstreamFeatureViews` from `dataSources` to draw the corresponding edges in React Flow.\n5. OpenLineage Emitter (sdk/python/feast/openlineage/mappers.py): Ensure `emit_apply()` creates the corresponding input dataset connections for OpenLineage producers.\n\n**Describe alternatives you've considered**\n\n1. Using OnDemandFeatureView (ODFV): ODFV handles on-the-fly transformations at request/serving time, but is not intended for pre-computed, stored, push-ingested features requiring offline batch storage.\n2. Adding metadata in tags: Storing `tags={\"depends_on\": \"user_tx_fv,user_credit_fv\"}` records metadata, but is non-standard and does not render visual lineage relationships in the Feast UI graph.\n3. Emitting manual OpenLineage events via CI/CD: Works if an OpenLineage backend is configured, but does not solve native Feast static registry lineage or out-of-the-box UI visualization.\n\n**Additional context**\nI am happy to contribute this change and submit a PR covering the proto update, Python SDK, registry lineage generator, and UI relationship parser if the community agrees with this direction!","author":{"url":"https://github.com/mao-liu","@type":"Person","name":"mao-liu"},"datePublished":"2026-09-15T10:50:47Z","url":"https://github.com/feast-dev/feast/issues/6839"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:2837d401-244b-4384-83c4-088f16ab3794 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | AFDE:79CF9:1C1C7A:256CB1:6AFE9229 |
| html-safe-nonce | aa4dc3c341fcc42a60e15de5b2d87d85d0a508234f07fa0959385baf3d956e67 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBRkRFOjc5Q0Y5OjFDMUM3QToyNTZDQjE6NkFGRTkyMjkiLCJ2aXNpdG9yX2lkIjoiNDQ4OTIyNzI4MTMxODcxMTg0OSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | c506ac996488cf86880da4b014beb3771005e09c5a63fd95311c7c2bb790e83f |
| hovercard-subject-tag | issue:5460931111 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/feast-dev/feast/6839/issue_layout |
| twitter:image | https://opengraph.githubassets.com/e82cb4952a9dbc45d5f5ac8a30cc45a0dc42f74ea60e22233cd3bbe1575d6089/feast-dev/feast/issues/6839 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/e82cb4952a9dbc45d5f5ac8a30cc45a0dc42f74ea60e22233cd3bbe1575d6089/feast-dev/feast/issues/6839 |
| og:image:alt | Is your feature request related to a problem? Please describe. In many real-world production ML architectures, derived features are computed outside of Feast by external stream or batch computation... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | mao-liu |
| hostname | github.com |
| expected-hostname | github.com |
| None | b3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86 |
| turbo-cache-control | no-preview |
| go-import | github.com/feast-dev/feast git https://github.com/feast-dev/feast.git |
| octolytics-dimension-user_id | 57027613 |
| octolytics-dimension-user_login | feast-dev |
| octolytics-dimension-repository_id | 161133770 |
| octolytics-dimension-repository_nwo | feast-dev/feast |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 161133770 |
| octolytics-dimension-repository_network_root_nwo | feast-dev/feast |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 4a10310af51a1bfffbedc4fd81cb582acf0cb0ff |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width