Title: feat: Add complex type support (Map, JSON, Struct) with schema validation by ntkathole · Pull Request #5974 · feast-dev/feast · GitHub
Open Graph Title: feat: Add complex type support (Map, JSON, Struct) with schema validation by ntkathole · Pull Request #5974 · feast-dev/feast
X Title: feat: Add complex type support (Map, JSON, Struct) with schema validation by ntkathole · Pull Request #5974 · feast-dev/feast
Description: What this PR does / why we need it: This PR adds comprehensive support for complex data types in Feast and implements schema validation across all compute engines: Commit 1 (970650b): Fix Map/Dict support and implement schema validation Fixed Map/Dict type support across offline stores, online stores, and type mappings Added enable_validation parameter to FeatureView, BatchFeatureView, and StreamFeatureView Implemented schema validation nodes in Local, Ray, and Spark compute engines (missing columns, type mismatch warnings) Added Spark and Milvus map type mappings Updated documentation for Map type backend mappings Commit 2 (ac28d4d): Add JSON and Struct complex data types Added JSON, JSON_LIST, STRUCT, STRUCT_LIST to the protobuf ValueType enum and Value message Introduced Json primitive type and Struct class (schema-aware structured type with named, typed fields) Implemented full type mapping pipeline: Feast ↔ Proto ↔ PyArrow ↔ backend-native types (PostgreSQL, BigQuery, Snowflake, Redshift, Spark, MSSQL, Athena, DynamoDB, Milvus) Added JSON well-formedness validation at both proto conversion level (always active) and validation node level (when enable_validation=True) Implemented Spark-native type validation using from_feast_to_spark_type() and _spark_types_compatible() — eliminates the PyArrow-to-Spark type mismatch gap Persisted Struct field schemas through the registry via Field tags (feast:struct_schema) Added Go SDK type conversion support for JSON and Struct types Added automatic serialization/deserialization for JSON type (Python dict/list ↔ JSON string) Single _validate_schema() method per engine (missing columns + type checks + JSON content validation) Commit 3 (a131b73): Modified default template with different types Added Json, Map and Struct in driver template
Open Graph Description: What this PR does / why we need it: This PR adds comprehensive support for complex data types in Feast and implements schema validation across all compute engines: Commit 1 (970650b): Fix Map/Dict ...
X Description: What this PR does / why we need it: This PR adds comprehensive support for complex data types in Feast and implements schema validation across all compute engines: Commit 1 (970650b): Fix Map/Dict ...
Opengraph URL: https://github.com/feast-dev/feast/pull/5974
X: @github
Domain: github.com
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:14073008-3760-c319-b00d-c47319a4c579 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | 9FC2:308E6B:13F4FC:1CC4BF:6A4E6501 |
| html-safe-nonce | 5dc4698b88de6c864464f7a7fba25e6521434843f38481aee0e6f282d6cfa2f4 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5RkMyOjMwOEU2QjoxM0Y0RkM6MUNDNEJGOjZBNEU2NTAxIiwidmlzaXRvcl9pZCI6IjMxNTQzOTY3NzkwOTAxNzUyMzMiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | fb7e590907f21a7f6b28c3a037bba86358e221f9400217c4ffae4310fd3b70ca |
| hovercard-subject-tag | pull_request:3289772551 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/feast-dev/feast/pull/5974/files |
| twitter:image | https://avatars.githubusercontent.com/u/16951913?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/16951913?s=400&v=4 |
| og:image:alt | What this PR does / why we need it: This PR adds comprehensive support for complex data types in Feast and implements schema validation across all compute engines: Commit 1 (970650b): Fix Map/Dict ... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 41b6ab3ba6d20a71766ac245b5a4a94c6fc672a9cd4da7d44c1b33ab8bf6a21c |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/feast-dev/feast git https://github.com/feast-dev/feast.git |
| octolytics-dimension-user_id | 57027613 |
| octolytics-dimension-user_login | feast-dev |
| octolytics-dimension-repository_id | 161133770 |
| octolytics-dimension-repository_nwo | feast-dev/feast |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 161133770 |
| octolytics-dimension-repository_network_root_nwo | feast-dev/feast |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | e6a744804e8e70f97b4d5a18a94dcc63db22f97a |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width