René's URL Explorer Experiment

Title: feat: MongoDB offline store by caseyclements · Pull Request #6138 · feast-dev/feast · GitHub

Open Graph Title: feat: MongoDB offline store by caseyclements · Pull Request #6138 · feast-dev/feast

X Title: feat: MongoDB offline store by caseyclements · Pull Request #6138 · feast-dev/feast

Description: What this PR does / why we need it: Adds a MongoDB offline store to Feast as a contrib plugin. Customers have been requesting this. All feature views share a single MongoDB collection (feature_history), discriminated by a feature_view field. Reads use aggregation pipelines backed by a compound index, giving O(log n) per-entity cost regardless of collection size. Note that we removed a second implementation that had originally been included. It did not scale well to production size documents relying on pandas for in-memory point-in-time joins. Design Single-collection schema — every document has the shape: { "entity_id": "", "feature_view": "driver_stats", "features": {"trips_today": 5, "rating": 4.8}, "event_timestamp": "2024-01-15T10:00:00Z", "created_at": "2024-01-15T10:00:01Z" } Compound index (created lazily on first use): (entity_id ASC, feature_view ASC, event_timestamp DESC, created_at DESC) This index supports every query the store issues. Query paths Path When Strategy Scoring entity_df has unique entity IDs $match → $sort → $group server-side dedup; returns ≤1 doc per (entity, FV). O(log P) per entity via compound index. Training entity_df has repeated entity IDs at different timestamps $match → $sort (no $group); point-in-time join via pandas.merge_asof in Python. K-collapse — feature views sharing the same join key set are batched into a single aggregation round-trip instead of one per FV. strict_pit — get_historical_features accepts strict_pit (default True). When True, documents whose timestamp is after the entity request time are returned as NULL (safe for training/evaluation). Set to False for real-time scoring where you always want the latest observation. What's included MongoDBOfflineStore, MongoDBOfflineStoreConfig, MongoDBSource, MongoDBRetrievalJob in sdk/python/feast/infra/offline_stores/contrib/mongodb_offline_store/mongodb.py offline_write_batch for ingesting feature data (append-only; conflict resolution at read time) persist support (writes joined results to Parquet) mongodb_to_feast_value_type type mapping in feast/type_map.py pymongo added to ci extras in pyproject.toml 17 unit tests in test_mongodb.py covering: pull_latest, pull_all, TTL filtering, K-collapse across multiple FVs, compound join keys, overlapping feature names with full_feature_names, INT32 entity keys, offline_write_batch round-trip, persist (write/overwrite/error), strict PIT nulls, chunk-size boundary, and strict_pit=False Why not universal tests? The single-collection design stores data in a different shape than the DataSourceCreator contract expects (the contract does not provide join-key information at data-creation time). The 17 dedicated unit tests validate the design end-to-end using Docker (testcontainers). The MongoDBDataSourceCreator remains in the codebase for future use when the universal contract is extended. Checks I've made sure the tests are passing. My commits are signed off (git commit -s) My PR title follows conventional commits format Testing Strategy Unit tests Integration tests Manual tests Testing is not required for this change Tests require Docker (ubuntu-latest in CI has it). Locally, tests are gated by @_requires_docker and skip gracefully when Docker is unavailable. Misc The offline store is marked as preview — all public entry points emit a RuntimeWarning.

Open Graph Description: What this PR does / why we need it: Adds a MongoDB offline store to Feast as a contrib plugin. Customers have been requesting this. All feature views share a single MongoDB collection (feature_hist...

X Description: What this PR does / why we need it: Adds a MongoDB offline store to Feast as a contrib plugin. Customers have been requesting this. All feature views share a single MongoDB collection (feature_hist...

Opengraph URL: https://github.com/feast-dev/feast/pull/6138

X: @github

direct link

Domain: github.com

Links:

Viewport: width=device-width