René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Revamp Data Quality Monitoring","articleBody":"**Is your feature request related to a problem? Please describe.**\n\nFeast defines and serves features via Feature Views, but it doesn’t provide a first-class way to monitor feature data quality and feature drift over time — especially for features produced via On Demand Transformations (ODT). \n\nToday, users often need to compare the “training distribution” (from a one-off static training dataset) with the distribution of feature values observed in production over time. This typically requires:\n\n- custom online feature logging pipelines,\n- bespoke Spark/SQL jobs to compute metrics and drift,\n- and external dashboards/alerting systems.\n\nThis becomes even more complex when ODT is involved, since transformed feature values are not easily reproducible unless transformation provenance and feature versions are tracked.\n\n**Describe the solution you'd like**\n\nIntroduce a new first-class concept in Feast: `DQMJob` (Data Quality Monitoring Job), which computes feature quality and drift metrics for one or more Feature Views over time and reports results in the Feast UI.\n\nProposed API:\n\n```python\nDQMJob(\n    feature_view_names: List[str],\n    data_source=None,  # should be the sink of the logged feature view values+metadata\n    features_to_exclude: Optional[List[str]] = None,\n    features_to_include: Optional[List[str]] = None,  # one or the other\n    time_interval: Literal[\"day\", \"week\", \"month\"] = \"day\",\n    feature_drift_metrics_config=None,  # schema-based defaults + per-feature overrides\n    baseline_config=None,  # static training dataset distribution reference\n    logging_config=None,   # simple: 100% or sampled logging\n)\n````\n\nKey components:\n\n1. **Simple feature logging to an offline sink**\n\n* During online feature retrieval, Feast can optionally emit feature logs to a configured offline sink (e.g., S3/GCS/ADLS or a warehouse table).\n* Logging supports either:\n\n  * 100% of requests, or\n  * sampled (e.g., p=0.01).\n* The sink is append-only and partitioned by time (e.g., dt=YYYY-MM-DD).\n* This offline dataset represents the “live” production feature values used by DQMJob.\n\n2. **ODT reproducibility via registry versioning**\n\n* For On Demand Transformations, feature logs include:\n\n  * input feature references + versions,\n  * output feature references + versions,\n  * transformation identifier (content hash),\n  * registry revision (or equivalent version id).\n* Transformation inputs/outputs and transformation code/artifact references are versioned in the Feast registry.\n* A content-addressed hash defines transformation identity and derived feature versions, enabling deterministic reproduction.\n\n3. **Baseline comparison against static training dataset**\n\n* Baseline is the distribution from a one-off training dataset run (static).\n* Each time bucket (e.g., daily) is compared against this fixed baseline using drift metrics appropriate to feature type.\n* Baseline is referenced deterministically (e.g., `baseline_dataset_uri` or a registry-managed `baseline_snapshot_id`).\n\n4. **Schema-aware metrics**\n\n* Quality metrics (examples):\n\n  * missing rate\n  * min/max/range\n  * mean/median\n  * unique count / cardinality\n* Drift metrics selected by schema:\n\n  * numeric: PSI, KS-statistic, etc.\n  * categorical: PSI, JS-divergence, entropy shift, etc.\n* Defaults inferred from feature schema with per-feature overrides.\n\n5. **Execution model**\n\n* `DQMJob` compiles into OfflineStore-specific execution plans (Spark DataFrame operations, warehouse SQL, etc.).\n* Jobs are intended to be run as batch jobs via orchestrators (Airflow, KFP, CLI, scripts).\n* OfflineStore implementations expose extensions for computing metrics efficiently.\n\n6. **Registry-managed metrics storage + Feast UI**\n\n* Computed metrics are written to a registry-managed metrics table (or registry-owned storage location).\n* Each metric record includes:\n\n  * feature_view_name\n  * feature_name\n  * feature_version / registry_revision\n  * metric_Name\n  * metric_value\n  * time_bucket\n  * baseline_reference\n* Feast UI queries this table to display:\n  * metric time series\n  * drift summaries\n  * per-feature quality views\n\n**Describe alternatives you've considered**\n\n* Integrating Feast with third-party observability tools (Evidently, WhyLabs, Arize, etc.) using custom logging pipelines.\n* Building custom Spark/SQL jobs on offline training data and online logs.\n* Monitoring only offline data (which misses what is actually served online, especially with ODT).\n\nWhile workable, these approaches require significant glue code and lack consistent offline/online symmetry within Feast.\n\n**Additional context**\n\n* Logging should be configurable for privacy and cost (sampling vs 100%).\n* Feature logs should be stored in batch-friendly formats (e.g., Parquet/Delta) and partitioned by time.\n* The design should preserve offline/online symmetry and leverage Feast registry versioning for reproducibility.\n* The system should remain backend-agnostic while using OfflineStore extensions for computation.\n","author":{"url":"https://github.com/franciscojavierarceo","@type":"Person","name":"franciscojavierarceo"},"datePublished":"2026-01-28T18:54:29.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/5919/feast/issues/5919"}