Title: feat: Add PolarisSource and Polaris governance sync on top of IcebergSource · Issue #7005 · feast-dev/feast · GitHub
Open Graph Title: feat: Add PolarisSource and Polaris governance sync on top of IcebergSource · Issue #7005 · feast-dev/feast
X Title: feat: Add PolarisSource and Polaris governance sync on top of IcebergSource · Issue #7005 · feast-dev/feast
Description: Is your feature request related to a problem? Please describe. Follow-up to #6499. IcebergSource (catalog_type="rest") already works against any Iceberg REST catalog, and UnityCatalogSource + UnityCatalogProvider add Unity Catalog govern...
Open Graph Description: Is your feature request related to a problem? Please describe. Follow-up to #6499. IcebergSource (catalog_type="rest") already works against any Iceberg REST catalog, and UnityCatalogSource + Unity...
X Description: Is your feature request related to a problem? Please describe. Follow-up to #6499. IcebergSource (catalog_type="rest") already works against any Iceberg REST catalog, and UnityCatalogSour...
Opengraph URL: https://github.com/feast-dev/feast/issues/7005
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"feat: Add PolarisSource and Polaris governance sync on top of IcebergSource","articleBody":"**Is your feature request related to a problem? Please describe.**\n\nFollow-up to #6499. `IcebergSource` (catalog_type=\"rest\") already works against any Iceberg REST catalog, and `UnityCatalogSource` + `UnityCatalogProvider` add Unity Catalog governance on top. There is no equivalent for Apache Polaris, and a few gaps make plain `IcebergSource` awkward to use against Polaris today:\n\n1. **Auth.** Only a static bearer token via `token_env_var` is supported. Polaris uses OAuth2 client credentials with short-lived tokens, so long-running `feast materialize` / serving jobs break when the token expires. The Polaris `/v1/oauth/tokens` endpoint is also marked deprecated for removal in the spec, so external IdP tokens need to be supported too.\n2. **Defaults.** Users must hand-write `endpoint` (`\u003chost>/api/catalog`) and the token env var for every source.\n3. **Read path.** In `duckdb.py::_read_iceberg_catalog_source` the REST branch calls `pyiceberg.load_catalog` without `X-Iceberg-Access-Delegation`, so credential vending from Polaris is not used, and the table is read with a full `scan().to_arrow()` (no column projection / predicate pushdown).\n4. **Governance.** Nothing records which Polaris tables are backing Feast feature views (project, feature view, owner, tags), unlike UC.\n5. **Discovery.** No helper to enumerate Polaris namespaces/tables and build sources from them.\n\n**Describe the solution you'd like**\n\nMirror the UC layering, keeping Polaris-specific code out of the generic `IcebergSource`:\n\n- `PolarisSource(IcebergSource)`: defaults from `POLARIS_HOST` / `POLARIS_TOKEN` (endpoint `\u003chost>/api/catalog`, `warehouse` = Polaris catalog name), plus `register_as_feature_table` / `sync_lineage`-style flags, serialized via `custom_options`.\n- `PolarisClient`: governance calls that are available on Polaris, using the standard Iceberg `updateTable` with `set-properties` to write `feast.project`, `feast.feature_view`, `feast.owner`, `feast.tag.*`, `feast.is_feature_table`, `feast.deleted_at`. Lineage has no Polaris API, so it is out of scope (or an optional `feast.upstream` property).\n- `PolarisProvider(PassthroughProvider)`: on `feast apply` / delete, sync those properties for views whose batch source is a `PolarisSource` (same shape as `UnityCatalogProvider`; never drops the table).\n- OAuth2 client-credentials token acquisition and refresh for the REST client (usable by `IcebergSource` as well).\n- Route the DuckDB (and Spark) read path through credential vending. (Column projection / filter pushdown was split out into #7019.)\n- A small discovery helper that lists namespaces/tables and returns `PolarisSource` objects for the user to wire into feature views.\n\n**Describe alternatives you've considered**\n\n- Use plain `IcebergSource` against Polaris: works for reads with a static token, but no refresh, no vended credentials on the DuckDB path, and no governance sync.\n- Use Polaris as the Feast registry: not a fit. Polaris is a table catalog (Iceberg REST, Generic Tables, Policies), not a transactional object/blob store, so it does not provide the atomic-update semantics the registry needs.\n- Policy API for governance: possible later, but properties on the table are the minimal, standards-based first step.\n\n**Additional context**\n\n- Depends on the `iceberg_catalog` contrib package already on master (#6499).\n- Based on reading the Polaris REST specs and the existing `iceberg_catalog` code; not yet validated end to end against a running Polaris. Happy to put up a PR if the direction looks right.\n\n","author":{"url":"https://github.com/haoxu0","@type":"Person","name":"haoxu0"},"datePublished":"2026-10-13T18:06:29Z","url":"https://github.com/feast-dev/feast/issues/7005"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:3c316be8-2801-b631-4c34-4ee1287c4e0e |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | A6AA:1DD5B3:193893:22EF86:6AFE9223 |
| html-safe-nonce | e4eb2eee6467e82fc7d007bdabb1877fdf4477d77a9e8e8bd6b5842dc676e3d5 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBNkFBOjFERDVCMzoxOTM4OTM6MjJFRjg2OjZBRkU5MjIzIiwidmlzaXRvcl9pZCI6IjEwMTA0MDk3MzY5OTE4Mzg3NTUiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | 505c887bfd4504c5ab78b061da977ebbc38ca8809cf471bad9d24a5de4e4fede |
| hovercard-subject-tag | issue:5835291986 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/feast-dev/feast/7005/issue_layout |
| twitter:image | https://opengraph.githubassets.com/a016a4f6b43d30c21c3466bf00592e4c9dd4925ffda52e2722bdba4b63fe7e58/feast-dev/feast/issues/7005 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/a016a4f6b43d30c21c3466bf00592e4c9dd4925ffda52e2722bdba4b63fe7e58/feast-dev/feast/issues/7005 |
| og:image:alt | Is your feature request related to a problem? Please describe. Follow-up to #6499. IcebergSource (catalog_type="rest") already works against any Iceberg REST catalog, and UnityCatalogSource + Unity... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | haoxu0 |
| hostname | github.com |
| expected-hostname | github.com |
| None | b3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86 |
| turbo-cache-control | no-preview |
| go-import | github.com/feast-dev/feast git https://github.com/feast-dev/feast.git |
| octolytics-dimension-user_id | 57027613 |
| octolytics-dimension-user_login | feast-dev |
| octolytics-dimension-repository_id | 161133770 |
| octolytics-dimension-repository_nwo | feast-dev/feast |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 161133770 |
| octolytics-dimension-repository_network_root_nwo | feast-dev/feast |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 4a10310af51a1bfffbedc4fd81cb582acf0cb0ff |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width