René's URL Explorer Experiment


Title: feat: Add native complex types and scalar index tags to the Milvus online store by jiwidi · Pull Request #7120 · feast-dev/feast · GitHub

Open Graph Title: feat: Add native complex types and scalar index tags to the Milvus online store by jiwidi · Pull Request #7120 · feast-dev/feast

X Title: feat: Add native complex types and scalar index tags to the Milvus online store by jiwidi · Pull Request #7120 · feast-dev/feast

Description: What this PR does / why we need it: With enable_openai_compatible_store: true the Milvus online store already stores numeric features as native columns, so that filters compare numbers instead of text. Arrays, Json, Map and Struct values were still base64 text, so a filter could not reach them and Milvus could not validate them. This PR completes the native storage behind the same flag: Feast type With enable_openai_compatible_store: true Array of String, Bool, Int32, Int64, Float32, Float64 or UnixTimestamp, without vector_index ARRAY of the element type, with at most milvus.max_capacity elements (default and maximum 4096) Json and Array(Json) JSON Both columns are nullable unless the field has milvus.nullable: "false", so a missing value stays null. A Json value must be an object or an array: PyMilvus cannot store a top-level string, number or boolean in a JSON column, and JSON null becomes a null column value. Map and Struct stay serialized, because JSON cannot hold their nested bytes, timestamps or empty typed lists. The column types of an existing collection decide how a row is written and read, as #7025 does for numeric columns, so collections created before this change keep working. Filter keys can address a JSON path or an array element with Milvus syntax, so retrieve_online_documents_v2 and online_delete can select rows by them: ComparisonFilter(type="gt", key='attributes["price"]', value=100) ComparisonFilter(type="eq", key="tags[0]", value="sale") A filter on a scalar column scans the collection unless the column has an index. Field tags now declare scalar indexes, which Feast creates with the collection, in the same way the milvus.partition_key tag declares the partition key: Field(name="brand_id", dtype=String, tags={"milvus.index": "INVERTED"}), Field(name="price", dtype=Float64, tags={"milvus.index": "STL_SORT"}), Field( name="attributes", dtype=Json, tags={"milvus.json_indexes": '[{"json_path": "attributes[\\"price\\"]", "json_cast_type": "double"}]'}, ), milvus.index names any scalar index type the server supports, milvus.index_params passes its parameters and milvus.json_indexes lists JSON path indexes of a JSON column. For an existing collection without them, Feast logs a warning, as it does for a missing partition key. The Elasticsearch online store indexes every feature value by default, Qdrant creates payload indexes on the fields its filters use, and Postgres indexes its filter columns; Milvus filters hit the feature columns, and those have no index unless one is declared. The docs describe the migration: the flag only applies when Feast creates a collection, a feast teardown drops the collections of the whole project, and a staged cutover avoids it. Benchmark. One million rows on a local single-node Milvus 3.0.2 in Docker, through PyMilvus: a 1000-value brand text column, a price number, a tags array of two to four values and a doc JSON object, plus an 8-dimensional HNSW vector. Three collections hold the same rows: text columns as before this PR, native columns, and native columns with an INVERTED index on brand and price and a JSON path index on doc["a"]. Each cell is the p50 / p95 latency of 30 runs after 5 warm-ups, for a plain query with the filter and for a top_k=10 vector search with the filter, measured after every index reported all rows indexed. text columns native columns native columns with indexes insert one million rows 14.5 s 14.0 s 14.1 s brand == 'b_7', query 3.7 / 6.1 ms 3.5 / 4.4 ms 1.4 / 2.1 ms brand == 'b_7', search 4.0 / 5.2 ms 3.8 / 5.5 ms 2.5 / 2.7 ms price > 990, query rejected by Milvus 1.4 / 1.7 ms 1.4 / 2.2 ms price > 990, search rejected by Milvus 2.0 / 2.8 ms 2.9 / 3.8 ms ARRAY_CONTAINS(tags, 't_7'), query rejected by Milvus 11.8 / 12.8 ms 10.9 / 11.9 ms ARRAY_CONTAINS(tags, 't_7'), search rejected by Milvus 12.4 / 13.1 ms 14.4 / 15.3 ms doc["a"] > 990, query rejected by Milvus 62.8 / 64.0 ms 3.6 / 4.2 ms doc["a"] > 990, search rejected by Milvus 5.1 / 6.4 ms 4.9 / 6.1 ms Native columns cost nothing at write time and make three kinds of filter possible that text columns reject. The inverted index cuts the equality filter from 3.5 to 1.4 ms, and the JSON path index cuts the JSON range query from 63 to 3.6 ms. The numeric index changes nothing at this size, and a filtered vector search gains less than a plain query because the vector search dominates it. Which issue(s) this PR fixes: N/A Checks I've made sure the tests are passing. My commits are signed off (git commit -s) My PR title follows conventional commits format Testing Strategy Unit tests Integration tests Manual tests Testing is not required for this change Unit (mocked client): Json and arrays get nullable JSON and ARRAY columns with the element type, milvus.max_capacity and max_length, Map stays VARCHAR; milvus.index and milvus.json_indexes tags are passed to Milvus as indexes; the tags are validated; a missing index on an existing collection is logged. Unit (Milvus Lite): twelve complex values, empty arrays included, round-trip through online_write_batch and online_read, and a missing value reads as null without any tag; JSON null reads as a missing value and a top-level JSON string or number is rejected; a JSON column tagged not nullable rejects a missing value; a JSON path filter selects search hits and an array element filter selects rows to delete; the filter key syntax accepts paths and rejects anything else. Server: the same round-trip, the JSON path filter, the array element delete, and the scalar and JSON path indexes against a real server. Unit tests run on Milvus Lite 3.2.1 (pymilvus 3.0.2). The server test in sdk/python/tests/integration/online_store/test_milvus_remote.py passes against a local Milvus 3.0.2 server. Milvus Lite does not support JSON path indexes, so that part runs on the server only. The existing Milvus unit and server tests pass unchanged. Misc Stacked on #7119, whose commit is included here until it merges; review the last commit. Follows #7082, #7083 and #7084 and Simon Hearne's series #6882, #6895, #6917, #6921 and #6940. When #7025 lands, native_numeric_types will control these columns too, as it is an alias of the same flag.

Open Graph Description: What this PR does / why we need it: With enable_openai_compatible_store: true the Milvus online store already stores numeric features as native columns, so that filters compare numbers instead of t...

X Description: What this PR does / why we need it: With enable_openai_compatible_store: true the Milvus online store already stores numeric features as native columns, so that filters compare numbers instead of t...

Opengraph URL: https://github.com/feast-dev/feast/pull/7120

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:93b305a4-dd11-b83b-f269-c0081fde3b28
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idC4E8:3CBC2:DDCD09:122E8D9:6AFE9275
html-safe-nonce8be7be4ced8dd3775905f99419c0243f331bdaddf5b191f97913596681c32958
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDNEU4OjNDQkMyOkREQ0QwOToxMjJFOEQ5OjZBRkU5Mjc1IiwidmlzaXRvcl9pZCI6IjY2OTI5MzUxNjA3Nzk2NzQyMjkiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmac941e98348fa198fd09892d3e712eed568d4a3716e195db7bec13ed84f7068b55
hovercard-subject-tagpull_request:5224453549
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/feast-dev/feast/pull/7120/files
twitter:imagehttps://avatars.githubusercontent.com/u/10882086?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/10882086?s=400&v=4
og:image:altWhat this PR does / why we need it: With enable_openai_compatible_store: true the Milvus online store already stores numeric features as native columns, so that filters compare numbers instead of t...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
Noneb3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/feast-dev/feast git https://github.com/feast-dev/feast.git
octolytics-dimension-user_id57027613
octolytics-dimension-user_loginfeast-dev
octolytics-dimension-repository_id161133770
octolytics-dimension-repository_nwofeast-dev/feast
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id161133770
octolytics-dimension-repository_network_root_nwofeast-dev/feast
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release4a10310af51a1bfffbedc4fd81cb582acf0cb0ff
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/feast-dev/feast/pull/7120/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ffeast-dev%2Ffeast%2Fpull%2F7120%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ffeast-dev%2Ffeast%2Fpull%2F7120%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=feast-dev%2Ffeast
Reloadhttps://github.com/feast-dev/feast/pull/7120/files
Reloadhttps://github.com/feast-dev/feast/pull/7120/files
Reloadhttps://github.com/feast-dev/feast/pull/7120/files
feast-devhttps://github.com/feast-dev
feasthttps://github.com/feast-dev/feast
Codehttps://github.com/feast-dev/feast
Issues223 (223)https://github.com/feast-dev/feast/issues
Pull requests143 (143)https://github.com/feast-dev/feast/pulls
Discussionshttps://github.com/feast-dev/feast/discussions
Actionshttps://github.com/feast-dev/feast/actions
Security and quality1 (1)https://github.com/feast-dev/feast/security
Insightshttps://github.com/feast-dev/feast/insights
Sign up for GitHub https://github.com/signup?return_to=%2Ffeast-dev%2Ffeast%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Ffeast-dev%2Ffeast%2Fissues%2Fnew%2Fchoose
shuchuhttps://github.com/shuchu
feast-dev:masterhttps://github.com/feast-dev/feast/tree/master
jiwidi:feat/milvus-native-types-and-indexeshttps://github.com/jiwidi/feast/tree/feat/milvus-native-types-and-indexes
Conversation 2 https://github.com/feast-dev/feast/pull/7120
Commits 2 https://github.com/feast-dev/feast/pull/7120/commits
Checks 28 https://github.com/feast-dev/feast/pull/7120/checks
Files changed 4 https://github.com/feast-dev/feast/pull/7120/files
feat: Add native complex types and scalar index tags to the Milvus online store https://github.com/feast-dev/feast/pull/7120/files#top
Show all changes 2 commits https://github.com/feast-dev/feast/pull/7120/files
f9be11e fix: Round-trip every Feast type and null in the Milvus online store jiwidi Nov 9, 2026 https://github.com/feast-dev/feast/pull/7120/commits/f9be11eda7afb3dd7efe4028bd2eeec8fdbdd940
d825b1f feat: Add native complex types and scalar index tags to the Milvus on… jiwidi Nov 9, 2026 https://github.com/feast-dev/feast/pull/7120/commits/d825b1f146d3000affe8339ca54888c24d209363
Clear filters https://github.com/feast-dev/feast/pull/7120/files
Please reload this pagehttps://github.com/feast-dev/feast/pull/7120/files
Please reload this pagehttps://github.com/feast-dev/feast/pull/7120/files
milvus.md https://github.com/feast-dev/feast/pull/7120/files#diff-db3c469e983dc1586778989d91e337aca298d5e9333b4f4bdaf3e7d995c17880
milvus.py https://github.com/feast-dev/feast/pull/7120/files#diff-3f21025352e4840bcd8dd17ca243f97ab1e0b984b1bc51ea68bfeff59fef00f9
test_milvus_remote.py https://github.com/feast-dev/feast/pull/7120/files#diff-95f25767fa3351a959d205bceb3281385fc4cec3503a02e4a13abfdc15b4d46f
test_milvus_online_store.py https://github.com/feast-dev/feast/pull/7120/files#diff-450f1a3096f9e72985ee29b609a310067a6308850ec5765c0b9708a4830b0cfe
https://github.com/feast-dev/feast/blob/master/CODEOWNERS#L4
docs/reference/online-stores/milvus.mdhttps://github.com/feast-dev/feast/pull/7120/files#diff-db3c469e983dc1586778989d91e337aca298d5e9333b4f4bdaf3e7d995c17880
View file https://github.com/jiwidi/feast/blob/d825b1f146d3000affe8339ca54888c24d209363/docs/reference/online-stores/milvus.md
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/feast-dev/feast/pull/7120/{{ revealButtonHref }}
https://github.com/feast-dev/feast/pull/7120/files#diff-db3c469e983dc1586778989d91e337aca298d5e9333b4f4bdaf3e7d995c17880
https://github.com/feast-dev/feast/pull/7120/files#diff-db3c469e983dc1586778989d91e337aca298d5e9333b4f4bdaf3e7d995c17880
https://github.com/feast-dev/feast/pull/7120/files#diff-db3c469e983dc1586778989d91e337aca298d5e9333b4f4bdaf3e7d995c17880
https://github.com/feast-dev/feast/pull/7120/files#diff-db3c469e983dc1586778989d91e337aca298d5e9333b4f4bdaf3e7d995c17880
Please reload this pagehttps://github.com/feast-dev/feast/pull/7120/files
Please reload this pagehttps://github.com/feast-dev/feast/pull/7120/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.