Title: feast serve: Gunicorn worker deadlocks on startup when forked during a /metrics request (follow-up to #6647) · Issue #6928 · feast-dev/feast · GitHub
Open Graph Title: feast serve: Gunicorn worker deadlocks on startup when forked during a /metrics request (follow-up to #6647) · Issue #6928 · feast-dev/feast
X Title: feast serve: Gunicorn worker deadlocks on startup when forked during a /metrics request (follow-up to #6647) · Issue #6928 · feast-dev/feast
Description: Expected behavior With metrics enabled, Gunicorn replaces a worker after --max-requests requests and the new worker starts serving. Current behavior Occasionally the new worker never starts. The log ends with Worker exiting (pid: N) and ...
Open Graph Description: Expected behavior With metrics enabled, Gunicorn replaces a worker after --max-requests requests and the new worker starts serving. Current behavior Occasionally the new worker never starts. The lo...
X Description: Expected behavior With metrics enabled, Gunicorn replaces a worker after --max-requests requests and the new worker starts serving. Current behavior Occasionally the new worker never starts. The lo...
Opengraph URL: https://github.com/feast-dev/feast/issues/6928
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"feast serve: Gunicorn worker deadlocks on startup when forked during a /metrics request (follow-up to #6647)","articleBody":"### Expected behavior\n\nWith metrics enabled, Gunicorn replaces a worker after `--max-requests` requests and the new worker\nstarts serving.\n\n### Current behavior\n\nOccasionally the new worker never starts. The log ends with `Worker exiting (pid: N)` and there is no\n`Booting worker with pid: ...` after it. The worker process exists but is blocked forever; the master\nkeeps the listening socket open and the metrics server (a thread in the master) keeps answering\n`/metrics`, so TCP health checks still pass while no request is served. Gunicorn's worker timeout does\nnot kill it either (separate Gunicorn issue: a worker that hangs before its first `notify()` is never\ntimed out).\n\nIn our production (2 deployments, 26 pods, ~17,500 worker recycles per day) this happened 14 times\nin 7 days; each pod served nothing for ~1 h until its listen backlog filled up and Kubernetes' TCP\nliveness probe failed.\n\n### Root cause\n\nSame class of bug as #6647 (fixed for the freshness thread in #6648): a thread started in the\nGunicorn master before `fork()`.\n\n`feature_server.start_server()` calls `feast_metrics.start_metrics_server()` in the Gunicorn master,\nbefore `FeastServeApplication(...).run()` forks the workers. That starts the metrics HTTP server\n(`wsgiref` `make_server(...)`, now `_make_metrics_httpd`) as a daemon thread with the default\n`WSGIRequestHandler`, which writes an access-log line to `sys.stderr` for **every** request\n(`log_message`).\n\nWhile that thread is inside `sys.stderr.write`, it holds the internal lock of stderr's\n`BufferedWriter`. If Gunicorn forks a worker at that moment, the child inherits the lock in the locked\nstate, and the thread that would release it does not exist in the child. The child's first action is\n`self.log.info(\"Booting worker with pid: %s\", ...)` (`gunicorn/arbiter.py`, `spawn_worker`), which\nwrites to stderr and blocks forever. Python's logging module re-initialises its own handler locks\nafter fork; the `BufferedWriter` lock is not re-initialised.\n\nPython warns about exactly this when `feast serve` starts:\n\n```\ngunicorn/arbiter.py:666: DeprecationWarning: This process (pid=16635) is multi-threaded, use of fork() may lead to deadlocks in the child.\n pid = os.fork()\n```\n\n### Evidence\n\n- Python stack of a stuck worker (Linux, `faulthandler` dump), identical in all 10 hangs we captured:\n ```\n File \"/usr/local/lib/python3.12/logging/__init__.py\", line 1163 in emit # stream.write(...)\n File \"/usr/local/lib/python3.12/logging/__init__.py\", line 1028 in handle\n ...\n File \".../gunicorn/glogging.py\", line 277 in info\n File \".../gunicorn/arbiter.py\", line 680 in spawn_worker # \"Booting worker\"\n File \".../gunicorn/arbiter.py\", line 719 in spawn_workers\n File \".../gunicorn/arbiter.py\", line 634 in manage_workers\n File \".../gunicorn/arbiter.py\", line 228 in run\n File \".../feast/feature_server.py\", line 941 in start_server\n ```\n The stuck worker has 1 thread, waiting on a futex; the master has 2 threads (main + metrics server).\n- Native stack of a stuck worker (macOS `sample`, 879 of 879 samples):\n ```\n _io_TextIOWrapper_write → _textiowrapper_writeflush → _io_BufferedWriter_write\n → _enter_buffered_busy → PyThread_acquire_lock_timed → __psynch_cvwait\n ```\n- Production: in 12 of 12 hangs, a `/metrics` scrape was logged 0–1 s after the last `Worker exiting`;\n for normal recycles that happens 9.3% of the time (39 of 420).\n- Controlled test (Kubernetes, `--max-requests 20`, 4 back-to-back `/metrics` scrapers, a fresh pod\n every 100 forks): **5 hangs in 318 forks** without a change; **0 hangs in 2,000 forks** when only the\n access log of the metrics server is silenced (the change proposed in the PR). Locally (macOS, steps\n below): a hang after 12–29 forks without the change, 0 in 336 forks with it.\n\n### Steps to reproduce\n\nFeast 0.65.0 or `master`, no external services:\n\n```bash\nfeast init -m demo && cd demo/feature_repo\ncat > feature_store.yaml \u003c\u003c'EOF'\nproject: demo\nregistry: data/registry.db\nprovider: local\nonline_store:\n type: sqlite\n path: data/online_store.db\nentity_key_serialization_version: 3\nfeature_server:\n metrics:\n enabled: true\n freshness: false # only the metrics server thread (freshness was #6647)\nEOF\nmkdir -p data && feast apply\n\n# terminal 1: recycle the worker every 20 requests\nfeast serve --max-requests 20 --max-requests-jitter 0 2>&1 | tee serve.log\n\n# terminal 2: keep the metrics thread busy + send requests\nfor i in 1 2 3 4; do (while true; do curl -s -o /dev/null http://127.0.0.1:8000/metrics; done) & done\nwhile true; do curl -s -o /dev/null -m 2 http://127.0.0.1:6566/health; done\n```\n\nWithin seconds to a few minutes, `serve.log` stops at `Worker exiting (pid: N)` with no\n`Booting worker` after it, and `/health` stops answering. `py-spy dump --pid \u003cchild>` (or `sample`\non macOS) shows the child blocked in the stderr write.\n\n### Possible solution\n\n1. Minimal: give the metrics server a request handler that does not log\n (`make_server(..., handler_class=...)` with a no-op `log_message`), as prometheus_client's own\n `start_http_server` does (`_SilentHandler`). PR to follow. This removes the stderr write we observed.\n2. More complete: do not run any thread in the Gunicorn master while it forks, e.g. serve the\n multiprocess metrics from a separate process. Any other lock taken by the metrics thread could\n cause the same deadlock in the future.\n\n### Environment\n\nFeast 0.65.0 (also checked 0.66.0 and `master` @ bc5aeef), gunicorn 25.0.3, uvicorn-worker 0.3.0,\nPython 3.12, `python:3.12-slim` on Linux arm64 (EKS) and macOS.\n","author":{"url":"https://github.com/aborgatin","@type":"Person","name":"aborgatin"},"datePublished":"2026-10-02T10:08:50Z","url":"https://github.com/feast-dev/feast/issues/6928"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:e32cdbbf-cca1-e1ca-9e01-ce7aaeafbbb4 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | 8F72:132CCD:DD2938:13066A8:6AFE9521 |
| html-safe-nonce | fe3aa6eccb87cfc791e92fefb25c83ba2b51f42f58aeec49c7f7cae3b1fe4e93 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI4RjcyOjEzMkNDRDpERDI5Mzg6MTMwNjZBODo2QUZFOTUyMSIsInZpc2l0b3JfaWQiOiI3MjY1OTIxNzE4MzMyMzMxMjk3IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | ea88d75a5eef407e5ad1d2031ae19a32b1da4b65319882bf285bd60fad41d18f |
| hovercard-subject-tag | issue:5675946778 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/feast-dev/feast/6928/issue_layout |
| twitter:image | https://opengraph.githubassets.com/10c20f7b65ce37e0a7cc23eb96f26fa6d9f11b289c2cf3af12a76dcf8fb67fec/feast-dev/feast/issues/6928 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/10c20f7b65ce37e0a7cc23eb96f26fa6d9f11b289c2cf3af12a76dcf8fb67fec/feast-dev/feast/issues/6928 |
| og:image:alt | Expected behavior With metrics enabled, Gunicorn replaces a worker after --max-requests requests and the new worker starts serving. Current behavior Occasionally the new worker never starts. The lo... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | aborgatin |
| hostname | github.com |
| expected-hostname | github.com |
| None | b3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86 |
| turbo-cache-control | no-preview |
| go-import | github.com/feast-dev/feast git https://github.com/feast-dev/feast.git |
| octolytics-dimension-user_id | 57027613 |
| octolytics-dimension-user_login | feast-dev |
| octolytics-dimension-repository_id | 161133770 |
| octolytics-dimension-repository_nwo | feast-dev/feast |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 161133770 |
| octolytics-dimension-repository_network_root_nwo | feast-dev/feast |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 4a10310af51a1bfffbedc4fd81cb582acf0cb0ff |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width