René's URL Explorer Experiment


Title: feat: Add a non-JVM read path for Lance data sources by haoxu0 · Pull Request #6943 · feast-dev/feast · GitHub

Open Graph Title: feat: Add a non-JVM read path for Lance data sources by haoxu0 · Pull Request #6943 · feast-dev/feast

X Title: feat: Add a non-JVM read path for Lance data sources by haoxu0 · Pull Request #6943 · feast-dev/feast

Description: Completes the read half of #6899. LanceFormat landed in #6925 as a format descriptor; nothing read Lance until now. Lance already worked through SparkSource, which drives its reader generically from table_format.format_type.value and table_format.properties. What was missing is a path that needs no JVM — which is also the real test of whether the DataSource abstraction is engine-agnostic rather than Spark-agnostic in name only. Placement, and why not the two obvious alternatives A new LanceSource read by the existing DuckDB store — not a Lance offline store, and not an extension of FileSource. FileSource is the wrong host. Its format axis is already occupied by file_format (ParquetFormat/DeltaFormat), so adding table_format gives one source two overlapping format axes. It is also read by two stores with incompatible contracts: duckdb._read_data_source dispatches on type, while dask._read_datasource has no dispatch seam at all — it reads file_options.uri unconditionally as Parquet and hard-asserts isinstance(..., FileSource) in three places, so a third format would break silently under Dask. Decisively: Lance's catalog addressing (catalog/namespace/table) has no path to put in FileSource.path, so the catalog-based layer would not fit the class even if the path-based one did. A Lance offline store would be the wrong 120 lines. duckdb.py is not an engine — it is a binding that injects _read_data_source/_write_data_source callbacks into the real engine in ibis.py. A LanceOfflineStore would be a near-verbatim copy of it plus a repo_config.py entry, and would force users to choose Lance instead of Parquet rather than mixing them in one feature service. Reading a source as ibis.memtable(arrow_table) in the DuckDB store already has two precedents — IcebergSource and MlflowDatasetSource. Following them means ibis.py is untouched, so the point-in-time join, TTL handling, field mapping and ODFVs all work unchanged, and neither repo_config.py nor data_source.py needs an edit because CUSTOM_SOURCE + data_source_class_type is self-describing in the registry proto. Net: +47 lines to duckdb.py, one new source module. Both addressing modes work # path-based LanceSource(uri="s3://bucket/poi_embeddings.lance", timestamp_field="generated_at") # catalog-based, through namespace_client + table_id LanceSource( table="poi_embeddings", namespace_impl="rest", table_format=LanceFormat(catalog="my_catalog", namespace="features"), timestamp_field="generated_at", ) Pin semantics: a pin selects data, never shape Following what was argued on #5782 and #6925. Three things enforce it: get_table_column_names_and_types reads the pinned schema, so feast apply infers the schema reads will actually see. A pre-flight check in get_historical_features fails with a message naming the pin when a pinned version cannot satisfy the declared schema. It reads schemas only — one metadata round trip, no scan. Vector widths go through _validate_vector_field_lengths from #6909 rather than a second validator. This is exact for Lance because Lance stores vectors as fixed_size_list, where that check is a property of the Arrow type. Demonstrated rather than asserted — the dataset is overwritten with a corrected value for the same (entity, timestamp), and the pinned read still returns the original: latest ( latest version): raw=[0.77, 1.17] get_historical_features@12:30 -> 0.77 tag pin ( tag 'eval_2026_q1'): raw=[0.5, 0.9 ] get_historical_features@12:30 -> 0.50 ver pin ( version 1): raw=[0.5, 0.9 ] get_historical_features@12:30 -> 0.50 Testing 43 tests, all using the dir namespace implementation — it exercises the identical namespace_client + table_id code path as a remote catalog, so the suite needs no server. Point-in-time correctness is covered for both the path-based and catalog-based variants, with a 12:30 query selecting the 11:00 row and never the 13:00 one. Regression evidence, rebased onto current master. Failure names compared rather than counts, since the suite has flaky members: with this change : 40 failures master : 40 failures only with change : (none) only on master : (none) ruff check and ruff format --check clean on all four files. mypy --ignore-missing-imports clean. (Bare mypy reports pre-existing missing-stub noise for pyarrow/pandas/ibis/pyiceberg/deltalake; the repo has no mypy config section.) Known limitations, deliberately Read-only. _write_data_source is untouched, so a LanceSource is not yet a persist target and there is no SavedDatasetLanceStorage. materialize into an online store does work, via pull_latest_from_table_or_query. Happy to follow up — _write_data_source already dispatches on IcebergSource, so the seam exists. DuckDB only. Dask cannot read Lance, by the reasoning above. RayOfflineStore would be a natural second host since it already has a _resolve_source_dataset dispatch, but it is not in this PR. Only the dir impl is exercised. rest, glue, unity, polaris and the Hive/Iceberg impls need lance-namespace-impls plus a live catalog. The code path is identical, which is why dir is a meaningful test, but I have not run against a remote catalog. Eager materialisation. The read pulls the dataset into Arrow before ibis sees it, matching what IcebergSource and MlflowDatasetSource already do. Column projection is pushed down via to_arrow(columns=...), but the retrieval path does not pass a projection and no predicate pushdown reaches Lance. Vector-width validation is exact only for fixed_size_list. For a plain list/large_list column the pre-flight runs on a zero-row table and infers nothing. Lance writes fixed_size_list, so this is the right trade in practice. Making it exact in general needs the feature view threaded into the data_source_reader callback in ibis.py, whose signature is (DataSource, str) — a separate change that would benefit every source, not just this one. Related: #6899, #6925, #6909, #5782, #5652

Open Graph Description: Completes the read half of #6899. LanceFormat landed in #6925 as a format descriptor; nothing read Lance until now. Lance already worked through SparkSource, which drives its reader generically fro...

X Description: Completes the read half of #6899. LanceFormat landed in #6925 as a format descriptor; nothing read Lance until now. Lance already worked through SparkSource, which drives its reader generically fro...

Opengraph URL: https://github.com/feast-dev/feast/pull/6943

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:f14f87f0-f0c4-3789-1834-0f9170bd3dfc
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idD52E:22BB81:F913B9:1480686:6AFE988B
html-safe-nonce601fdc37fd3f2af1441e90972761a42a4887ae46ea017c19412cd3db8eb7c1a4
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJENTJFOjIyQkI4MTpGOTEzQjk6MTQ4MDY4Njo2QUZFOTg4QiIsInZpc2l0b3JfaWQiOiIyNzg4ODU0NTkwNzM1MjkyNTU1IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0=
visitor-hmaca5b587dde3da8f1a05ba427dc7ce204fd2021185dc8d201cab46ce1f18f5e9fa
hovercard-subject-tagpull_request:4749869634
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/feast-dev/feast/pull/6943/files
twitter:imagehttps://avatars.githubusercontent.com/u/20422210?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/20422210?s=400&v=4
og:image:altCompletes the read half of #6899. LanceFormat landed in #6925 as a format descriptor; nothing read Lance until now. Lance already worked through SparkSource, which drives its reader generically fro...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
Noneb3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/feast-dev/feast git https://github.com/feast-dev/feast.git
octolytics-dimension-user_id57027613
octolytics-dimension-user_loginfeast-dev
octolytics-dimension-repository_id161133770
octolytics-dimension-repository_nwofeast-dev/feast
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id161133770
octolytics-dimension-repository_network_root_nwofeast-dev/feast
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release4a10310af51a1bfffbedc4fd81cb582acf0cb0ff
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/feast-dev/feast/pull/6943/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ffeast-dev%2Ffeast%2Fpull%2F6943%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ffeast-dev%2Ffeast%2Fpull%2F6943%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=feast-dev%2Ffeast
Reloadhttps://github.com/feast-dev/feast/pull/6943/files
Reloadhttps://github.com/feast-dev/feast/pull/6943/files
Reloadhttps://github.com/feast-dev/feast/pull/6943/files
feast-devhttps://github.com/feast-dev
feasthttps://github.com/feast-dev/feast
Codehttps://github.com/feast-dev/feast
Issues223 (223)https://github.com/feast-dev/feast/issues
Pull requests143 (143)https://github.com/feast-dev/feast/pulls
Discussionshttps://github.com/feast-dev/feast/discussions
Actionshttps://github.com/feast-dev/feast/actions
Security and quality1 (1)https://github.com/feast-dev/feast/security
Insightshttps://github.com/feast-dev/feast/insights
Sign up for GitHub https://github.com/signup?return_to=%2Ffeast-dev%2Ffeast%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Ffeast-dev%2Ffeast%2Fissues%2Fnew%2Fchoose
ntkatholehttps://github.com/ntkathole
feast-dev:masterhttps://github.com/feast-dev/feast/tree/master
haoxu0:feat/lance-readhttps://github.com/haoxu0/feast/tree/feat/lance-read
Conversation 19 https://github.com/feast-dev/feast/pull/6943
Commits 2 https://github.com/feast-dev/feast/pull/6943/commits
Checks 23 https://github.com/feast-dev/feast/pull/6943/checks
Files changed https://github.com/feast-dev/feast/pull/6943/files
Please reload this pagehttps://github.com/feast-dev/feast/pull/6943/files
feat: Add a non-JVM read path for Lance data sources https://github.com/feast-dev/feast/pull/6943/files#top
Show all changes 2 commits https://github.com/feast-dev/feast/pull/6943/files
e7e2adf feat: Add a non-JVM read path for Lance data sources haoxu0 Oct 5, 2026 https://github.com/feast-dev/feast/pull/6943/commits/e7e2adf9415b76a9d6f744bcf9184ce6cd08671a
8b8d17c fix: Address Lance read review feedback haoxu0 Oct 7, 2026 https://github.com/feast-dev/feast/pull/6943/commits/8b8d17c6089e9ac82d3ee94c8198cec4e2f1b762
Clear filters https://github.com/feast-dev/feast/pull/6943/files
Please reload this pagehttps://github.com/feast-dev/feast/pull/6943/files
Please reload this pagehttps://github.com/feast-dev/feast/pull/6943/files
__init__.py https://github.com/feast-dev/feast/pull/6943/files#diff-8e96065d52b1053786e7b63026d05b9cd9206408c51f4a3d045c08648437fe52
lance_source.py https://github.com/feast-dev/feast/pull/6943/files#diff-de55de91f24b10043a4361da7e06d46f1eb87f3f9f9150e1cb9a0e544f39cfdf
duckdb.py https://github.com/feast-dev/feast/pull/6943/files#diff-ea2fa10bfef868a728514ef5f0890661d652c372ca286830d16361d99e93477d
test_lance_source.py https://github.com/feast-dev/feast/pull/6943/files#diff-779994a44e0f89d67f1a056146bc4fa50860b62f31e76873daf8bb3a781d6b07
https://github.com/feast-dev/feast/blob/master/CODEOWNERS#L10
sdk/python/feast/infra/data_sources/contrib/lance/__init__.pyhttps://github.com/feast-dev/feast/pull/6943/files#diff-8e96065d52b1053786e7b63026d05b9cd9206408c51f4a3d045c08648437fe52
View file https://github.com/haoxu0/feast/blob/8b8d17c6089e9ac82d3ee94c8198cec4e2f1b762/sdk/python/feast/infra/data_sources/contrib/lance/__init__.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/feast-dev/feast/pull/6943/{{ revealButtonHref }}
Please reload this pagehttps://github.com/feast-dev/feast/pull/6943/files
Please reload this pagehttps://github.com/feast-dev/feast/pull/6943/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.