René's URL Explorer Experiment


Title: feat: Add a non-JVM write path for Lance data sources by haoxu0 · Pull Request #6945 · feast-dev/feast · GitHub

Open Graph Title: feat: Add a non-JVM write path for Lance data sources by haoxu0 · Pull Request #6945 · feast-dev/feast

X Title: feat: Add a non-JVM write path for Lance data sources by haoxu0 · Pull Request #6945 · feast-dev/feast

Description: Rebased onto master now that #6943 (the read path) and #6944 (fixed_size_list decoding) have merged. The two commits that carried them are gone; what is left is the write path alone, and it applies cleanly. What this does #6943 made a LanceSource readable without Spark. This makes one writable, through the same two addressing modes — a uri, or a namespace_client + table_id resolved through a Lance namespace — so a catalog-addressed dataset is written to the location its namespace resolves rather than to one assembled locally. _write_data_source gains a LanceSource branch, following the IcebergSource branch already there, which makes the path reachable from DuckDBOfflineStore.offline_write_batch. The isinstance narrowing that the read path open-coded is now a shared _as_lance_source helper used by both, rather than a second copy of the optional-import guard. +94 lines in duckdb.py, +121 in lance_source.py. How far the namespace is involved Measured by recording the calls a write makes on the namespace client: operation namespace calls create declare_table append describe_table only open describe_table, namespace_id So a Lance namespace records that a table exists and where it lives, and does not track its versions — the version an append produces is never reported back to it. This is the reason the pin contract below has to be enforced locally: no catalog is going to reject a write on a pin's behalf. There is a test that records these calls so the behaviour is pinned rather than assumed. Three things settled before Lance is called A pinned source is refused. A Lance commit always produces a new version, so write_dataset has no version argument at all. A write through a source pinned to version 1 or to a tag would succeed and then be invisible through the very source that performed it. Measured: tag prod -> version 1 appended; latest ver 2 | tag prod still reads rows: 10 <-- write invisible to pinned source An absent dataset is created rather than appended to, because Lance has no create-or-append mode: CREATE on existing: REJECTED -> OSError Dataset already exists An existing dataset's vector widths are compared against the incoming data. This is the one case Lance itself permits silently: APPEND dim-mismatch: REJECTED -> `emb` should have type fixed_size_list:float:8 but type was fixed_size_list:float:16 OVERWRITE 8->32: fixed_size_list[32] ver 3 <-- accepted, schema silently rewritten v1 still readable: fixed_size_list[8] <-- history now self-inconsistent So a single overwrite can leave a dataset whose version history has mutually incomparable vectors, with every ANN index built on the old width invalidated. No legitimate schema evolution changes an embedding's dimension, so this is refused rather than carried out. The guard is narrow on purpose: only vector widths are policed. Other type changes stay Lance's business, because for those an overwrite is a defensible evolution — there is a test asserting an int64 → string overwrite still goes through. Declared vector_length Checked at the store entry point, not in the writer callback, which is handed a DataSource and so cannot see the feature view. It reuses _validate_vector_field_lengths from #6909 rather than adding a second implementation. Without it, creating a dataset at a width other than the declared one would succeed and fail only on the next read. Mode semantics entry point mode passed by offline_write_batch append offline_write_batch_ibis, 3 positional args persist overwrite ibis.py persist overwrite without allow_overwrite raises SavedDatasetLocationAlreadyExists, matching the FileSource branch. persist is not reachable for Lance yet: it needs a SavedDatasetStorage, and _DATA_SOURCE_TO_SAVED_DATASET_STORAGE currently maps only FileSource and SparkSource (IcebergSource has no entry either). I left that for a follow-up because the obvious slot is a problem on its own — _proto_attr_name = "custom_storage" is already claimed by three classes (couchbase, clickhouse, postgres) against a plain-dict registry, so SavedDatasetStorage.from_proto dispatches to whichever was imported last. I raised that as #6946 rather than add a fourth claimant. overwrite is still implemented and tested directly, since it is required for correctness once that lands. A Spark write path is also deliberately out of scope here. SparkOfflineStore.offline_write_batch consults only file_format and ignores table_format, which I raised as #6955; that needs a decision about which of the two axes is authoritative before a Lance branch there would mean anything. Verification 20 tests added; 65 pass in test_lance_source.py (45 from #6943 plus these). Catalog-based cases use the dir namespace implementation, which drives the identical namespace_client + table_id code path as a remote catalog, so no server is needed. Covered: create and append for both addressing modes; version-pin and tag-pin refusal; a refused write leaving the dataset at its original version and row count; the append-invisible-to-an-earlier-pin property that motivates the refusal; overwrite with and without allow_overwrite; vector-width change refused and width-preserving overwrite allowed; a non-vector type change left alone; a width guard ignoring columns absent from the write; declared-width enforcement at offline_write_batch; the namespace call sequence above; and a write-then-read round trip through _read_data_source. Full sdk/python/tests/unit: 40 failures on master and 53 on this branch, but the 13 names in the difference all reproduce on untouched master when that subset is run in isolation — they are pre-existing order- and environment-dependent cases in test_milvus_online_store, test_chronon_online_store and test_metrics, not regressions. Nothing in the difference touches Lance. ruff check sdk/python/ and ruff format --check clean. One caveat worth stating plainly: as Copilot noted on #6943, this test module is skipped entirely by its module-level importorskip, so these tests do not run in CI today. #6958 adds the lance extra and wires it into feast[ci], which is what will actually turn them on. Until it lands, the numbers above are local.

Open Graph Description: Rebased onto master now that #6943 (the read path) and #6944 (fixed_size_list decoding) have merged. The two commits that carried them are gone; what is left is the write path alone, and it applies...

X Description: Rebased onto master now that #6943 (the read path) and #6944 (fixed_size_list decoding) have merged. The two commits that carried them are gone; what is left is the write path alone, and it applies...

Opengraph URL: https://github.com/feast-dev/feast/pull/6945

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:3321b7b9-4621-bdf7-51ab-ca5b263558e9
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-id9296:729DB:EC0CA9:137370B:6AFE9878
html-safe-noncec0285cb0fa564c3cfb52bec51fff3400f464b30fa8cb351a2ad75e84bab3e720
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5Mjk2OjcyOURCOkVDMENBOToxMzczNzBCOjZBRkU5ODc4IiwidmlzaXRvcl9pZCI6IjU0NTQ3OTgyNDA4NDUwNDM4MzIiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmace3aa88b6ba13c7b70bf9d13cc30bd167fb35a0afc93910a615b0085b4b9989e2
hovercard-subject-tagpull_request:4750429857
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/feast-dev/feast/pull/6945/files
twitter:imagehttps://avatars.githubusercontent.com/u/20422210?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/20422210?s=400&v=4
og:image:altRebased onto master now that #6943 (the read path) and #6944 (fixed_size_list decoding) have merged. The two commits that carried them are gone; what is left is the write path alone, and it applies...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
Noneb3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/feast-dev/feast git https://github.com/feast-dev/feast.git
octolytics-dimension-user_id57027613
octolytics-dimension-user_loginfeast-dev
octolytics-dimension-repository_id161133770
octolytics-dimension-repository_nwofeast-dev/feast
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id161133770
octolytics-dimension-repository_network_root_nwofeast-dev/feast
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release4a10310af51a1bfffbedc4fd81cb582acf0cb0ff
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/feast-dev/feast/pull/6945/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ffeast-dev%2Ffeast%2Fpull%2F6945%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ffeast-dev%2Ffeast%2Fpull%2F6945%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=feast-dev%2Ffeast
Reloadhttps://github.com/feast-dev/feast/pull/6945/files
Reloadhttps://github.com/feast-dev/feast/pull/6945/files
Reloadhttps://github.com/feast-dev/feast/pull/6945/files
feast-devhttps://github.com/feast-dev
feasthttps://github.com/feast-dev/feast
Codehttps://github.com/feast-dev/feast
Issues223 (223)https://github.com/feast-dev/feast/issues
Pull requests143 (143)https://github.com/feast-dev/feast/pulls
Discussionshttps://github.com/feast-dev/feast/discussions
Actionshttps://github.com/feast-dev/feast/actions
Security and quality1 (1)https://github.com/feast-dev/feast/security
Insightshttps://github.com/feast-dev/feast/insights
Sign up for GitHub https://github.com/signup?return_to=%2Ffeast-dev%2Ffeast%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Ffeast-dev%2Ffeast%2Fissues%2Fnew%2Fchoose
ntkatholehttps://github.com/ntkathole
feast-dev:masterhttps://github.com/feast-dev/feast/tree/master
haoxu0:feat/lance-writehttps://github.com/haoxu0/feast/tree/feat/lance-write
Conversation 3 https://github.com/feast-dev/feast/pull/6945
Commits 3 https://github.com/feast-dev/feast/pull/6945/commits
Checks 23 https://github.com/feast-dev/feast/pull/6945/checks
Files changed https://github.com/feast-dev/feast/pull/6945/files
Please reload this pagehttps://github.com/feast-dev/feast/pull/6945/files
feat: Add a non-JVM write path for Lance data sources https://github.com/feast-dev/feast/pull/6945/files#top
Show all changes 3 commits https://github.com/feast-dev/feast/pull/6945/files
828e5a1 feat: Add a non-JVM write path for Lance data sources haoxu0 Oct 5, 2026 https://github.com/feast-dev/feast/pull/6945/commits/828e5a1ac7a30c05b6d8eda35a967791d99e0620
f96470b fix: Correct what a Lance namespace does during a write haoxu0 Oct 6, 2026 https://github.com/feast-dev/feast/pull/6945/commits/f96470b9ae413b3a1c69e1c2aa279c742432b30c
ee23843 fix: Accept an optional source in _as_lance_source haoxu0 Oct 9, 2026 https://github.com/feast-dev/feast/pull/6945/commits/ee238438abfbc6c3a71a792c37c6ec7633cb1848
Clear filters https://github.com/feast-dev/feast/pull/6945/files
Please reload this pagehttps://github.com/feast-dev/feast/pull/6945/files
Please reload this pagehttps://github.com/feast-dev/feast/pull/6945/files
lance_source.py https://github.com/feast-dev/feast/pull/6945/files#diff-de55de91f24b10043a4361da7e06d46f1eb87f3f9f9150e1cb9a0e544f39cfdf
duckdb.py https://github.com/feast-dev/feast/pull/6945/files#diff-ea2fa10bfef868a728514ef5f0890661d652c372ca286830d16361d99e93477d
test_lance_source.py https://github.com/feast-dev/feast/pull/6945/files#diff-779994a44e0f89d67f1a056146bc4fa50860b62f31e76873daf8bb3a781d6b07
https://github.com/feast-dev/feast/blob/master/CODEOWNERS#L10
sdk/python/feast/infra/data_sources/contrib/lance/lance_source.pyhttps://github.com/feast-dev/feast/pull/6945/files#diff-de55de91f24b10043a4361da7e06d46f1eb87f3f9f9150e1cb9a0e544f39cfdf
View file https://github.com/haoxu0/feast/blob/ee238438abfbc6c3a71a792c37c6ec7633cb1848/sdk/python/feast/infra/data_sources/contrib/lance/lance_source.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/feast-dev/feast/pull/6945/{{ revealButtonHref }}
https://github.com/feast-dev/feast/pull/6945/files#diff-de55de91f24b10043a4361da7e06d46f1eb87f3f9f9150e1cb9a0e544f39cfdf
https://github.com/feast-dev/feast/pull/6945/files#diff-de55de91f24b10043a4361da7e06d46f1eb87f3f9f9150e1cb9a0e544f39cfdf
https://github.com/feast-dev/feast/pull/6945/files#diff-de55de91f24b10043a4361da7e06d46f1eb87f3f9f9150e1cb9a0e544f39cfdf
https://github.com/feast-dev/feast/blob/master/CODEOWNERS#L10
sdk/python/feast/infra/offline_stores/duckdb.pyhttps://github.com/feast-dev/feast/pull/6945/files#diff-ea2fa10bfef868a728514ef5f0890661d652c372ca286830d16361d99e93477d
View file https://github.com/haoxu0/feast/blob/ee238438abfbc6c3a71a792c37c6ec7633cb1848/sdk/python/feast/infra/offline_stores/duckdb.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/feast-dev/feast/pull/6945/{{ revealButtonHref }}
https://github.com/feast-dev/feast/pull/6945/files#diff-ea2fa10bfef868a728514ef5f0890661d652c372ca286830d16361d99e93477d
https://github.com/feast-dev/feast/pull/6945/files#diff-ea2fa10bfef868a728514ef5f0890661d652c372ca286830d16361d99e93477d
https://github.com/feast-dev/feast/pull/6945/files#diff-ea2fa10bfef868a728514ef5f0890661d652c372ca286830d16361d99e93477d
https://github.com/feast-dev/feast/pull/6945/files#diff-ea2fa10bfef868a728514ef5f0890661d652c372ca286830d16361d99e93477d
ntkatholehttps://github.com/ntkathole
Oct 9, 2026https://github.com/feast-dev/feast/pull/6945/files#r4226251813
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
Please reload this pagehttps://github.com/feast-dev/feast/pull/6945/files
https://github.com/feast-dev/feast/pull/6945/files#diff-ea2fa10bfef868a728514ef5f0890661d652c372ca286830d16361d99e93477d
https://github.com/feast-dev/feast/pull/6945/files#diff-ea2fa10bfef868a728514ef5f0890661d652c372ca286830d16361d99e93477d
https://github.com/feast-dev/feast/pull/6945/files#diff-ea2fa10bfef868a728514ef5f0890661d652c372ca286830d16361d99e93477d
https://github.com/feast-dev/feast/pull/6945/files#diff-ea2fa10bfef868a728514ef5f0890661d652c372ca286830d16361d99e93477d
Please reload this pagehttps://github.com/feast-dev/feast/pull/6945/files
Please reload this pagehttps://github.com/feast-dev/feast/pull/6945/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.