Title: fix: Round-trip every Feast type and null in the Milvus online store by jiwidi · Pull Request #7119 · feast-dev/feast · GitHub
Open Graph Title: fix: Round-trip every Feast type and null in the Milvus online store by jiwidi · Pull Request #7119 · feast-dev/feast
X Title: fix: Round-trip every Feast type and null in the Milvus online store by jiwidi · Pull Request #7119 · feast-dev/feast
Description: What this PR does / why we need it: Every other online store returns a feature value as it was written, whatever its type, because Redis, Postgres, SQLite, DynamoDB, Elasticsearch, Qdrant and the rest store the serialized ValueProto. The Milvus online store keeps one column per feature, and with the default configuration it could not return most non-numeric types. On upstream master with Milvus Lite: Feast type Before this PR Bool the write works, the read raises TypeError UnixTimestamp no column is created, so every write of the feature view fails Array(String) the write works, the read raises AttributeError an array of numbers without vector_index declared as a vector of embedding_dim, so every write fails Bytes comes back as its base64 text, not the bytes Json comes back as the base64 of its protobuf, not the JSON text a missing value stored as "", 0 or false; the read of a numeric text column raises ValueError any Array, Map or Struct in a search hit comes back as text, because the search path decoded primitive types only a Bytes or UnixTimestamp entity column written as the text of its Python value, b'...' or a datetime The write path now stores every scalar natively or as text, bytes as base64, and every other value as the base64 of its ValueProto, for features and entity columns alike, and one helper rebuilds the ValueProto for reads and searches from every declared field type. UnixTimestamp gets a column. Only a field with vector_index=True becomes a vector, with the dimension from Field.vector_length before embedding_dim; the Elasticsearch, Postgres, Qdrant and MongoDB stores already pick the vector field this way. An empty text in a non-string column reads as null. Array(Bool) with vector_index=True now raises at collection creation: it was mapped to a binary vector but written unpacked and indexed with a float metric, so it could not work. A missing value is still written as the column's default when the column cannot hold NULL. The new milvus.nullable: "true" field tag makes a column nullable, so a missing value is stored and read as null, as every other store does: Field(name="rating", dtype=Float32, tags={"milvus.nullable": "true"}) Nothing here needs a flag. Every change fixes a call that failed or returned a wrong value, the read fixes apply to existing collections at once, and the new columns and the tag only apply when Feast creates a collection, like the partition key. Which issue(s) this PR fixes: N/A Checks I've made sure the tests are passing. My commits are signed off (git commit -s) My PR title follows conventional commits format Testing Strategy Unit tests Integration tests Manual tests Testing is not required for this change Unit (Milvus Lite): thirteen Feast values, empty bytes included, round-trip through online_write_batch and online_read, with and without native numeric types; a missing value in a text column reads as null; a milvus.nullable column stores null and a plain column keeps its default; search hits decode every type the same way as reads; Bytes and UnixTimestamp entity columns round-trip. Unit (mocked client): a numeric array without vector_index is a VARCHAR column, a vector_index field takes its vector_length, UnixTimestamp gets a column; the tag is rejected on entity, vector and partition key fields and for an invalid value; Array(Bool) with vector_index is rejected. Server: the same round-trip and the nullable column against a real server, with and without native numeric types. Unit tests run on Milvus Lite 3.2.1 (pymilvus 3.0.2). The server tests in sdk/python/tests/integration/online_store/test_milvus_remote.py pass against a local Milvus 3.0.2 server. The existing Milvus unit and server tests pass unchanged. Misc Follows #7082, #7083 and #7084 and Simon Hearne's series #6882, #6895, #6917, #6921 and #6940. The next PR builds on this one: native ARRAY and JSON columns and scalar index tags behind enable_openai_compatible_store.
Open Graph Description: What this PR does / why we need it: Every other online store returns a feature value as it was written, whatever its type, because Redis, Postgres, SQLite, DynamoDB, Elasticsearch, Qdrant and the r...
X Description: What this PR does / why we need it: Every other online store returns a feature value as it was written, whatever its type, because Redis, Postgres, SQLite, DynamoDB, Elasticsearch, Qdrant and the r...
Opengraph URL: https://github.com/feast-dev/feast/pull/7119
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:2728d87a-d9ef-9777-6751-b4de022624ec |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | D838:110AAF:215B0A:2E01AE:6AFE9560 |
| html-safe-nonce | 35ad59bf53d55393f53d7beed5cc7436e297e0ed53cb3a575cd8dc92b78a0b02 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJEODM4OjExMEFBRjoyMTVCMEE6MkUwMUFFOjZBRkU5NTYwIiwidmlzaXRvcl9pZCI6IjUwNzExOTM5MTg2NzgzNTc0NCIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 22b980567e6f14ad2e6cf463b14214f718f5776a59f0dd9998b5c32dee1b98a0 |
| hovercard-subject-tag | pull_request:5224452990 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/feast-dev/feast/pull/7119/files |
| twitter:image | https://avatars.githubusercontent.com/u/10882086?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/10882086?s=400&v=4 |
| og:image:alt | What this PR does / why we need it: Every other online store returns a feature value as it was written, whatever its type, because Redis, Postgres, SQLite, DynamoDB, Elasticsearch, Qdrant and the r... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | b3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/feast-dev/feast git https://github.com/feast-dev/feast.git |
| octolytics-dimension-user_id | 57027613 |
| octolytics-dimension-user_login | feast-dev |
| octolytics-dimension-repository_id | 161133770 |
| octolytics-dimension-repository_nwo | feast-dev/feast |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 161133770 |
| octolytics-dimension-repository_network_root_nwo | feast-dev/feast |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 4a10310af51a1bfffbedc4fd81cb582acf0cb0ff |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width