Title: `feast serve`: /metrics stops responding after the first gunicorn worker exits (regression in 0.67.0, #6855) · Issue #7011 · feast-dev/feast · GitHub
Open Graph Title: `feast serve`: /metrics stops responding after the first gunicorn worker exits (regression in 0.67.0, #6855) · Issue #7011 · feast-dev/feast
X Title: `feast serve`: /metrics stops responding after the first gunicorn worker exits (regression in 0.67.0, #6855) · Issue #7011 · feast-dev/feast
Description: Current behavior Since 0.67.0 (#6855) the Prometheus metrics server runs in a multiprocessing.Process(daemon=True) started in the gunicorn master. Gunicorn workers are forked from the master and inherit multiprocessing's registry of chil...
Open Graph Description: Current behavior Since 0.67.0 (#6855) the Prometheus metrics server runs in a multiprocessing.Process(daemon=True) started in the gunicorn master. Gunicorn workers are forked from the master and in...
X Description: Current behavior Since 0.67.0 (#6855) the Prometheus metrics server runs in a multiprocessing.Process(daemon=True) started in the gunicorn master. Gunicorn workers are forked from the master and in...
Opengraph URL: https://github.com/feast-dev/feast/issues/7011
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"`feast serve`: /metrics stops responding after the first gunicorn worker exits (regression in 0.67.0, #6855)","articleBody":"### Current behavior\n\nSince 0.67.0 (#6855) the Prometheus metrics server runs in a `multiprocessing.Process(daemon=True)` started in the gunicorn master. Gunicorn workers are forked from the master and inherit `multiprocessing`'s registry of child processes, which contains the metrics process. When a worker exits (`--max-requests` recycle, worker timeout, `HUP` reload), Python's atexit handler `multiprocessing.util._exit_function` runs in the worker, treats the metrics process as its own daemon child and calls `terminate()` on it (SIGTERM). Then it fails to `join()` it:\n\n```\n[INFO] Maximum request limit of 3 exceeded. Terminating process.\n[INFO] Worker exiting (pid: 77539)\nException ignored in atexit callback: \u003cfunction _exit_function at 0x11b90c4a0>\nTraceback (most recent call last):\n File \".../multiprocessing/util.py\", line 360, in _exit_function\n p.join()\n File \".../multiprocessing/process.py\", line 147, in join\n assert self._parent_pid == os.getpid(), 'can only join a child process'\nAssertionError: can only join a child process\n[INFO] Booting worker with pid: 77567\n```\n\nFrom then on nothing listens on port 8000: `/metrics` returns connection refused until `feast serve` is restarted, while feature serving keeps working. Workers keep writing their metric files, but nothing serves them.\n\nAny `feast serve` with metrics enabled loses its metrics endpoint at the first worker restart, so with `--max-requests` this happens shortly after start. It happens with both multiprocessing start methods: fork (Linux) and spawn (macOS). 0.66.0 is not affected.\n\n### Expected behavior\n\nThe metrics server keeps running for the whole life of the gunicorn master, independent of worker restarts.\n\n### Steps to reproduce\n\n1. Create an empty folder with this `feature_store.yaml` (local registry and SQLite online store, metrics enabled):\n\n```yaml\nproject: demo\nregistry: data/registry.db\nprovider: local\nonline_store:\n type: sqlite\n path: data/online_store.db\nentity_key_serialization_version: 3\nfeature_server:\n metrics:\n enabled: true\n```\n\n2. In that folder, install Feast 0.67.0 and register the (empty) repo:\n\n```bash\npip install \"feast==0.67.0\"\nmkdir -p data && feast apply\n```\n\n3. Start the server with one worker that is restarted after 3 requests:\n\n```bash\nfeast serve --workers 1 --max-requests 3 --max-requests-jitter 0 > serve.log 2>&1 &\n```\n\n4. Once `curl -s -m 2 http://127.0.0.1:6566/health` answers, check `/metrics`, send 5 requests (the worker restarts after the 3rd), and check `/metrics` again:\n\n```bash\ncurl -s -m 5 -o /dev/null -w 'metrics before: %{http_code}\\n' http://127.0.0.1:8000/metrics\nfor i in 1 2 3 4 5; do curl -s -m 5 -o /dev/null -w \"request $i: %{http_code}\\n\" http://127.0.0.1:6566/health; done\nsleep 3\ncurl -s -m 5 -o /dev/null -w 'metrics after: %{http_code}\\n' http://127.0.0.1:8000/metrics\ngrep -m1 \"AssertionError: can only join a child process\" serve.log\n```\n\nOutput with 0.67.0:\n\n```\nmetrics before: 200\nrequest 1: 200\nrequest 2: 200\nrequest 3: 200\nrequest 4: 200\nrequest 5: 200\nmetrics after: 000\nAssertionError: can only join a child process\n```\n\n`000` means connection refused: nothing listens on port 8000 any more, while `/health` keeps returning 200.\n\nWith `feast==0.66.0` the same steps print `metrics after: 200` and no `AssertionError`.\n\nThe mechanism without Feast (plain Python 3.12; prints `False` on Linux and macOS):\n\n```python\nimport multiprocessing as mp, os, sys, time\n\ndef serve():\n while True:\n time.sleep(1)\n\nif __name__ == \"__main__\":\n p = mp.Process(target=serve, daemon=True) # like start_metrics_server() in the gunicorn master\n p.start(); time.sleep(0.5)\n pid = os.fork() # like a gunicorn worker\n if pid == 0:\n sys.exit(0) # worker exit runs multiprocessing's atexit -> terminate(p)\n os.waitpid(pid, 0); time.sleep(0.5)\n print(\"metrics process alive:\", p.is_alive()) # False\n```\n\n### Possible solutions\n\n- In each gunicorn worker (e.g. in `_gunicorn_post_worker_init`), remove the inherited metrics process from `multiprocessing`'s child registry, so the worker's exit no longer terminates it. The master still owns it and stops it on shutdown.\n- Start the metrics process with `os.fork()` or `subprocess` instead of `multiprocessing`, so forked workers don't track it, and have the master terminate it on exit.\n\n### Environment\n\n- Feast 0.67.0 (0.66.0 not affected), gunicorn 26.2.0, prometheus-client 0.24.1, Python 3.12\n- Linux (`python:3.12-slim`) and macOS 27\n","author":{"url":"https://github.com/aborgatin","@type":"Person","name":"aborgatin"},"datePublished":"2026-10-14T12:49:30Z","url":"https://github.com/feast-dev/feast/issues/7011"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:f23d6018-e430-4bdd-7c42-f944f81e3b10 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | A6A8:F21AC:1131DA:17F565:6AFE8F7B |
| html-safe-nonce | 43bc802ea13d7f4727fc785b271cfbe69007a4783194f21bb6258fdc3ed82d52 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBNkE4OkYyMUFDOjExMzFEQToxN0Y1NjU6NkFGRThGN0IiLCJ2aXNpdG9yX2lkIjoiMjMzODAwODQzNzcwMDMzMzQzNSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | c6f0c33d011425549bc9983dcbf71031e2b13717f34fe3f7b1544c11f26b0923 |
| hovercard-subject-tag | issue:5848088420 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/feast-dev/feast/7011/issue_layout |
| twitter:image | https://opengraph.githubassets.com/90c233e62717129ae8236e89eab32db332bce2f1d08d652df2b0658475dcbcc5/feast-dev/feast/issues/7011 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/90c233e62717129ae8236e89eab32db332bce2f1d08d652df2b0658475dcbcc5/feast-dev/feast/issues/7011 |
| og:image:alt | Current behavior Since 0.67.0 (#6855) the Prometheus metrics server runs in a multiprocessing.Process(daemon=True) started in the gunicorn master. Gunicorn workers are forked from the master and in... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | aborgatin |
| hostname | github.com |
| expected-hostname | github.com |
| None | b3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86 |
| turbo-cache-control | no-preview |
| go-import | github.com/feast-dev/feast git https://github.com/feast-dev/feast.git |
| octolytics-dimension-user_id | 57027613 |
| octolytics-dimension-user_login | feast-dev |
| octolytics-dimension-repository_id | 161133770 |
| octolytics-dimension-repository_nwo | feast-dev/feast |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 161133770 |
| octolytics-dimension-repository_network_root_nwo | feast-dev/feast |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 4a10310af51a1bfffbedc4fd81cb582acf0cb0ff |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width