René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Improve Feature Server Observability","articleBody":"**Is your feature request related to a problem? Please describe.**\n\nThe Feature Server currently has limited built-in observability. While there is initial OpenTelemetry (OTEL) support, it does not expose standard service-level RED metrics (Rate, Errors, Duration) needed to operate the server in production.\n\nUsers need visibility into throughput, error rates, and latency for the core APIs:\n\n- `/get-online-features`\n- `/retrieve-online-documents`\n- `/push`\n- `/write-to-online-store`\n- `/materialize`\n- `/materialize-incremental`\n\nToday this often requires custom middleware or external tooling.\n\n**Describe the solution you'd like**\n\nExtend Feature Server’s OTEL integration to emit standard RED metrics out of the box, with useful Feast-specific breakdowns.\n\nMetrics per endpoint should include:\n\n- **Request rate/count**\n- **Error count/rate** (by HTTP status class, optionally basic error categories)\n- **Latency histograms** (supporting p50/p95/p99)\n\nAdditional breakdowns:\n\n- Break out metrics by **Feature Service name** (and **Feature View name** where applicable/available), with safeguards to limit label cardinality (e.g., allowlist and/or config flags; default off if needed).\n- Segment latency by **number of features requested** using configurable bins (e.g., `1–10`, `11–50`, `51–200`, `201+`).\n\nImplementation note (based on current `feature_server.py` structure):\n- Add a FastAPI middleware to record RED metrics for every request (keyed by `endpoint` + `status_class`).\n- Populate Feast-specific labels (feature_service / feature_view / feature_count_bin, etc.) inside the request handlers (since they come from the parsed request body) and attach them via `request.state` for the middleware to include.\n- For long-running endpoints (`/materialize*`), ensure duration histograms support multi-second/minute latencies.\n\nOptionally expose basic internal timings via tracing spans and/or additional histograms:\n- online store read/write duration\n- on-demand transformation execution duration\n- materialization step duration\n\nMetrics should use native OpenTelemetry APIs and work with common OTEL collectors/exporters (e.g., Prometheus, OpenTelemetry Collector, Grafana).\n\n**Describe alternatives you've considered**\n\n- Custom HTTP middleware\n- Service mesh / proxy metrics (miss Feast-specific context like FS/FV/feature_count)\n- Manual instrumentation\n\n**Additional context**\n\nStandard RED metrics with Feast-aware labels would significantly improve Feature Server operability and align Feast with common production observability practices, while keeping label cardinality under control.\n","author":{"url":"https://github.com/franciscojavierarceo","@type":"Person","name":"franciscojavierarceo"},"datePublished":"2026-01-28T19:23:07.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":5},"url":"https://github.com/5920/feast/issues/5920"}