Title: perf: Decouple metrics HTTP server into a dedicated child process by ntkathole · Pull Request #6855 · feast-dev/feast · GitHub
Open Graph Title: perf: Decouple metrics HTTP server into a dedicated child process by ntkathole · Pull Request #6855 · feast-dev/feast
X Title: perf: Decouple metrics HTTP server into a dedicated child process by ntkathole · Pull Request #6855 · feast-dev/feast
Description: Summary Replace the daemon thread that serves the Prometheus /metrics endpoint with a multiprocessing.Process so that scrape-time aggregation (MultiProcessCollector reads + text serialization) runs with its own GIL, fully isolated from request-serving workers and the Gunicorn master. Problem When Prometheus scrapes the /metrics endpoint, the current daemon thread performs: File listing of all .db files in PROMETHEUS_MULTIPROCESS_DIR Reading mmap-backed metric files from every Gunicorn worker Aggregating/merging metrics across workers Serializing to Prometheus text format All of this runs under the same GIL as the Gunicorn master process. With high metric cardinality (many label combinations), this scrape overhead can cause GIL contention that indirectly affects request latency. Solution Move the WSGI HTTP metrics server into a dedicated child process via multiprocessing.Process(daemon=True): _run_metrics_server() — new top-level function that serves as the child process entry point. It re-imports prometheus_client for macOS spawn safety and registers SIGTERM/SIGINT handlers for graceful shutdown. start_metrics_server() — now spawns a Process instead of a Thread. The function signature and call sites are unchanged. Background monitoring threads (resource, freshness) remain in-process because they only write to mmap-backed Gauges — an operation that is fast and does not benefit from process isolation. Why this is safe Aspect Before (thread) After (process) Metrics port :8000 in master process :8000 in child process GIL isolation Shared with master Fully isolated daemon=True Thread auto-joins on exit Process auto-terminates on parent exit Prometheus scrape Same metrics, same format Identical — MultiProcessCollector reads same mmap files Kubernetes/HPA Transparent Transparent — same port, same pod network namespace spawn safety N/A _run_metrics_server is top-level, re-imports deps
Open Graph Description: Summary Replace the daemon thread that serves the Prometheus /metrics endpoint with a multiprocessing.Process so that scrape-time aggregation (MultiProcessCollector reads + text serialization) runs...
X Description: Summary Replace the daemon thread that serves the Prometheus /metrics endpoint with a multiprocessing.Process so that scrape-time aggregation (MultiProcessCollector reads + text serialization) runs...
Opengraph URL: https://github.com/feast-dev/feast/pull/6855
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:307c9cce-81f8-6ee6-0444-3079c95ddba0 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | D85E:22FF8D:21F1C6:2EFC4F:6AFE9563 |
| html-safe-nonce | 8d30d336f7de73b1da25ab1b567e6e0f030fc501c87fb9aa12e75c778d52d587 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJEODVFOjIyRkY4RDoyMUYxQzY6MkVGQzRGOjZBRkU5NTYzIiwidmlzaXRvcl9pZCI6IjY4Mjk0MTMxNzU0NDg3MzcxMjMiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | a33a39ce3a377fb8fb3002ea163485a532221b228a56e86f12daf27ae87fd975 |
| hovercard-subject-tag | pull_request:4598707068 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/feast-dev/feast/pull/6855/files |
| twitter:image | https://avatars.githubusercontent.com/u/16951913?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/16951913?s=400&v=4 |
| og:image:alt | Summary Replace the daemon thread that serves the Prometheus /metrics endpoint with a multiprocessing.Process so that scrape-time aggregation (MultiProcessCollector reads + text serialization) runs... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | b3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/feast-dev/feast git https://github.com/feast-dev/feast.git |
| octolytics-dimension-user_id | 57027613 |
| octolytics-dimension-user_login | feast-dev |
| octolytics-dimension-repository_id | 161133770 |
| octolytics-dimension-repository_nwo | feast-dev/feast |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 161133770 |
| octolytics-dimension-repository_network_root_nwo | feast-dev/feast |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 4a10310af51a1bfffbedc4fd81cb582acf0cb0ff |
| ui-target | canary-2 |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width