Title: feat: Add native complex types and scalar index tags to the Milvus online store by jiwidi · Pull Request #7120 · feast-dev/feast · GitHub
Open Graph Title: feat: Add native complex types and scalar index tags to the Milvus online store by jiwidi · Pull Request #7120 · feast-dev/feast
X Title: feat: Add native complex types and scalar index tags to the Milvus online store by jiwidi · Pull Request #7120 · feast-dev/feast
Description: What this PR does / why we need it: With enable_openai_compatible_store: true the Milvus online store already stores numeric features as native columns, so that filters compare numbers instead of text. Arrays, Json, Map and Struct values were still base64 text, so a filter could not reach them and Milvus could not validate them. This PR completes the native storage behind the same flag: Feast type With enable_openai_compatible_store: true Array of String, Bool, Int32, Int64, Float32, Float64 or UnixTimestamp, without vector_index ARRAY of the element type, with at most milvus.max_capacity elements (default and maximum 4096) Json and Array(Json) JSON Both columns are nullable unless the field has milvus.nullable: "false", so a missing value stays null. A Json value must be an object or an array: PyMilvus cannot store a top-level string, number or boolean in a JSON column, and JSON null becomes a null column value. Map and Struct stay serialized, because JSON cannot hold their nested bytes, timestamps or empty typed lists. The column types of an existing collection decide how a row is written and read, as #7025 does for numeric columns, so collections created before this change keep working. Filter keys can address a JSON path or an array element with Milvus syntax, so retrieve_online_documents_v2 and online_delete can select rows by them: ComparisonFilter(type="gt", key='attributes["price"]', value=100) ComparisonFilter(type="eq", key="tags[0]", value="sale") A filter on a scalar column scans the collection unless the column has an index. Field tags now declare scalar indexes, which Feast creates with the collection, in the same way the milvus.partition_key tag declares the partition key: Field(name="brand_id", dtype=String, tags={"milvus.index": "INVERTED"}), Field(name="price", dtype=Float64, tags={"milvus.index": "STL_SORT"}), Field( name="attributes", dtype=Json, tags={"milvus.json_indexes": '[{"json_path": "attributes[\\"price\\"]", "json_cast_type": "double"}]'}, ), milvus.index names any scalar index type the server supports, milvus.index_params passes its parameters and milvus.json_indexes lists JSON path indexes of a JSON column. For an existing collection without them, Feast logs a warning, as it does for a missing partition key. The Elasticsearch online store indexes every feature value by default, Qdrant creates payload indexes on the fields its filters use, and Postgres indexes its filter columns; Milvus filters hit the feature columns, and those have no index unless one is declared. The docs describe the migration: the flag only applies when Feast creates a collection, a feast teardown drops the collections of the whole project, and a staged cutover avoids it. Benchmark. One million rows on a local single-node Milvus 3.0.2 in Docker, through PyMilvus: a 1000-value brand text column, a price number, a tags array of two to four values and a doc JSON object, plus an 8-dimensional HNSW vector. Three collections hold the same rows: text columns as before this PR, native columns, and native columns with an INVERTED index on brand and price and a JSON path index on doc["a"]. Each cell is the p50 / p95 latency of 30 runs after 5 warm-ups, for a plain query with the filter and for a top_k=10 vector search with the filter, measured after every index reported all rows indexed. text columns native columns native columns with indexes insert one million rows 14.5 s 14.0 s 14.1 s brand == 'b_7', query 3.7 / 6.1 ms 3.5 / 4.4 ms 1.4 / 2.1 ms brand == 'b_7', search 4.0 / 5.2 ms 3.8 / 5.5 ms 2.5 / 2.7 ms price > 990, query rejected by Milvus 1.4 / 1.7 ms 1.4 / 2.2 ms price > 990, search rejected by Milvus 2.0 / 2.8 ms 2.9 / 3.8 ms ARRAY_CONTAINS(tags, 't_7'), query rejected by Milvus 11.8 / 12.8 ms 10.9 / 11.9 ms ARRAY_CONTAINS(tags, 't_7'), search rejected by Milvus 12.4 / 13.1 ms 14.4 / 15.3 ms doc["a"] > 990, query rejected by Milvus 62.8 / 64.0 ms 3.6 / 4.2 ms doc["a"] > 990, search rejected by Milvus 5.1 / 6.4 ms 4.9 / 6.1 ms Native columns cost nothing at write time and make three kinds of filter possible that text columns reject. The inverted index cuts the equality filter from 3.5 to 1.4 ms, and the JSON path index cuts the JSON range query from 63 to 3.6 ms. The numeric index changes nothing at this size, and a filtered vector search gains less than a plain query because the vector search dominates it. Which issue(s) this PR fixes: N/A Checks I've made sure the tests are passing. My commits are signed off (git commit -s) My PR title follows conventional commits format Testing Strategy Unit tests Integration tests Manual tests Testing is not required for this change Unit (mocked client): Json and arrays get nullable JSON and ARRAY columns with the element type, milvus.max_capacity and max_length, Map stays VARCHAR; milvus.index and milvus.json_indexes tags are passed to Milvus as indexes; the tags are validated; a missing index on an existing collection is logged. Unit (Milvus Lite): twelve complex values, empty arrays included, round-trip through online_write_batch and online_read, and a missing value reads as null without any tag; JSON null reads as a missing value and a top-level JSON string or number is rejected; a JSON column tagged not nullable rejects a missing value; a JSON path filter selects search hits and an array element filter selects rows to delete; the filter key syntax accepts paths and rejects anything else. Server: the same round-trip, the JSON path filter, the array element delete, and the scalar and JSON path indexes against a real server. Unit tests run on Milvus Lite 3.2.1 (pymilvus 3.0.2). The server test in sdk/python/tests/integration/online_store/test_milvus_remote.py passes against a local Milvus 3.0.2 server. Milvus Lite does not support JSON path indexes, so that part runs on the server only. The existing Milvus unit and server tests pass unchanged. Misc Stacked on #7119, whose commit is included here until it merges; review the last commit. Follows #7082, #7083 and #7084 and Simon Hearne's series #6882, #6895, #6917, #6921 and #6940. When #7025 lands, native_numeric_types will control these columns too, as it is an alias of the same flag.
Open Graph Description: What this PR does / why we need it: With enable_openai_compatible_store: true the Milvus online store already stores numeric features as native columns, so that filters compare numbers instead of t...
X Description: What this PR does / why we need it: With enable_openai_compatible_store: true the Milvus online store already stores numeric features as native columns, so that filters compare numbers instead of t...
Opengraph URL: https://github.com/feast-dev/feast/pull/7120
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:93b305a4-dd11-b83b-f269-c0081fde3b28 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | C4E8:3CBC2:DDCD09:122E8D9:6AFE9275 |
| html-safe-nonce | 8be7be4ced8dd3775905f99419c0243f331bdaddf5b191f97913596681c32958 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDNEU4OjNDQkMyOkREQ0QwOToxMjJFOEQ5OjZBRkU5Mjc1IiwidmlzaXRvcl9pZCI6IjY2OTI5MzUxNjA3Nzk2NzQyMjkiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | 941e98348fa198fd09892d3e712eed568d4a3716e195db7bec13ed84f7068b55 |
| hovercard-subject-tag | pull_request:5224453549 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/feast-dev/feast/pull/7120/files |
| twitter:image | https://avatars.githubusercontent.com/u/10882086?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/10882086?s=400&v=4 |
| og:image:alt | What this PR does / why we need it: With enable_openai_compatible_store: true the Milvus online store already stores numeric features as native columns, so that filters compare numbers instead of t... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | b3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/feast-dev/feast git https://github.com/feast-dev/feast.git |
| octolytics-dimension-user_id | 57027613 |
| octolytics-dimension-user_login | feast-dev |
| octolytics-dimension-repository_id | 161133770 |
| octolytics-dimension-repository_nwo | feast-dev/feast |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 161133770 |
| octolytics-dimension-repository_network_root_nwo | feast-dev/feast |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 4a10310af51a1bfffbedc4fd81cb582acf0cb0ff |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width