René's URL Explorer Experiment


Title: `feast serve`: /metrics stops responding after the first gunicorn worker exits (regression in 0.67.0, #6855) · Issue #7011 · feast-dev/feast · GitHub

Open Graph Title: `feast serve`: /metrics stops responding after the first gunicorn worker exits (regression in 0.67.0, #6855) · Issue #7011 · feast-dev/feast

X Title: `feast serve`: /metrics stops responding after the first gunicorn worker exits (regression in 0.67.0, #6855) · Issue #7011 · feast-dev/feast

Description: Current behavior Since 0.67.0 (#6855) the Prometheus metrics server runs in a multiprocessing.Process(daemon=True) started in the gunicorn master. Gunicorn workers are forked from the master and inherit multiprocessing's registry of chil...

Open Graph Description: Current behavior Since 0.67.0 (#6855) the Prometheus metrics server runs in a multiprocessing.Process(daemon=True) started in the gunicorn master. Gunicorn workers are forked from the master and in...

X Description: Current behavior Since 0.67.0 (#6855) the Prometheus metrics server runs in a multiprocessing.Process(daemon=True) started in the gunicorn master. Gunicorn workers are forked from the master and in...

Opengraph URL: https://github.com/feast-dev/feast/issues/7011

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"`feast serve`: /metrics stops responding after the first gunicorn worker exits (regression in 0.67.0, #6855)","articleBody":"### Current behavior\n\nSince 0.67.0 (#6855) the Prometheus metrics server runs in a `multiprocessing.Process(daemon=True)` started in the gunicorn master. Gunicorn workers are forked from the master and inherit `multiprocessing`'s registry of child processes, which contains the metrics process. When a worker exits (`--max-requests` recycle, worker timeout, `HUP` reload), Python's atexit handler `multiprocessing.util._exit_function` runs in the worker, treats the metrics process as its own daemon child and calls `terminate()` on it (SIGTERM). Then it fails to `join()` it:\n\n```\n[INFO] Maximum request limit of 3 exceeded. Terminating process.\n[INFO] Worker exiting (pid: 77539)\nException ignored in atexit callback: \u003cfunction _exit_function at 0x11b90c4a0>\nTraceback (most recent call last):\n  File \".../multiprocessing/util.py\", line 360, in _exit_function\n    p.join()\n  File \".../multiprocessing/process.py\", line 147, in join\n    assert self._parent_pid == os.getpid(), 'can only join a child process'\nAssertionError: can only join a child process\n[INFO] Booting worker with pid: 77567\n```\n\nFrom then on nothing listens on port 8000: `/metrics` returns connection refused until `feast serve` is restarted, while feature serving keeps working. Workers keep writing their metric files, but nothing serves them.\n\nAny `feast serve` with metrics enabled loses its metrics endpoint at the first worker restart, so with `--max-requests` this happens shortly after start. It happens with both multiprocessing start methods: fork (Linux) and spawn (macOS). 0.66.0 is not affected.\n\n### Expected behavior\n\nThe metrics server keeps running for the whole life of the gunicorn master, independent of worker restarts.\n\n### Steps to reproduce\n\n1. Create an empty folder with this `feature_store.yaml` (local registry and SQLite online store, metrics enabled):\n\n```yaml\nproject: demo\nregistry: data/registry.db\nprovider: local\nonline_store:\n    type: sqlite\n    path: data/online_store.db\nentity_key_serialization_version: 3\nfeature_server:\n    metrics:\n        enabled: true\n```\n\n2. In that folder, install Feast 0.67.0 and register the (empty) repo:\n\n```bash\npip install \"feast==0.67.0\"\nmkdir -p data && feast apply\n```\n\n3. Start the server with one worker that is restarted after 3 requests:\n\n```bash\nfeast serve --workers 1 --max-requests 3 --max-requests-jitter 0 > serve.log 2>&1 &\n```\n\n4. Once `curl -s -m 2 http://127.0.0.1:6566/health` answers, check `/metrics`, send 5 requests (the worker restarts after the 3rd), and check `/metrics` again:\n\n```bash\ncurl -s -m 5 -o /dev/null -w 'metrics before: %{http_code}\\n' http://127.0.0.1:8000/metrics\nfor i in 1 2 3 4 5; do curl -s -m 5 -o /dev/null -w \"request $i: %{http_code}\\n\" http://127.0.0.1:6566/health; done\nsleep 3\ncurl -s -m 5 -o /dev/null -w 'metrics after: %{http_code}\\n' http://127.0.0.1:8000/metrics\ngrep -m1 \"AssertionError: can only join a child process\" serve.log\n```\n\nOutput with 0.67.0:\n\n```\nmetrics before: 200\nrequest 1: 200\nrequest 2: 200\nrequest 3: 200\nrequest 4: 200\nrequest 5: 200\nmetrics after: 000\nAssertionError: can only join a child process\n```\n\n`000` means connection refused: nothing listens on port 8000 any more, while `/health` keeps returning 200.\n\nWith `feast==0.66.0` the same steps print `metrics after: 200` and no `AssertionError`.\n\nThe mechanism without Feast (plain Python 3.12; prints `False` on Linux and macOS):\n\n```python\nimport multiprocessing as mp, os, sys, time\n\ndef serve():\n    while True:\n        time.sleep(1)\n\nif __name__ == \"__main__\":\n    p = mp.Process(target=serve, daemon=True)  # like start_metrics_server() in the gunicorn master\n    p.start(); time.sleep(0.5)\n    pid = os.fork()                             # like a gunicorn worker\n    if pid == 0:\n        sys.exit(0)                             # worker exit runs multiprocessing's atexit -> terminate(p)\n    os.waitpid(pid, 0); time.sleep(0.5)\n    print(\"metrics process alive:\", p.is_alive())   # False\n```\n\n### Possible solutions\n\n- In each gunicorn worker (e.g. in `_gunicorn_post_worker_init`), remove the inherited metrics process from `multiprocessing`'s child registry, so the worker's exit no longer terminates it. The master still owns it and stops it on shutdown.\n- Start the metrics process with `os.fork()` or `subprocess` instead of `multiprocessing`, so forked workers don't track it, and have the master terminate it on exit.\n\n### Environment\n\n- Feast 0.67.0 (0.66.0 not affected), gunicorn 26.2.0, prometheus-client 0.24.1, Python 3.12\n- Linux (`python:3.12-slim`) and macOS 27\n","author":{"url":"https://github.com/aborgatin","@type":"Person","name":"aborgatin"},"datePublished":"2026-10-14T12:49:30Z","url":"https://github.com/feast-dev/feast/issues/7011"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:f23d6018-e430-4bdd-7c42-f944f81e3b10
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-idA6A8:F21AC:1131DA:17F565:6AFE8F7B
html-safe-nonce43bc802ea13d7f4727fc785b271cfbe69007a4783194f21bb6258fdc3ed82d52
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBNkE4OkYyMUFDOjExMzFEQToxN0Y1NjU6NkFGRThGN0IiLCJ2aXNpdG9yX2lkIjoiMjMzODAwODQzNzcwMDMzMzQzNSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmacc6f0c33d011425549bc9983dcbf71031e2b13717f34fe3f7b1544c11f26b0923
hovercard-subject-tagissue:5848088420
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/feast-dev/feast/7011/issue_layout
twitter:imagehttps://opengraph.githubassets.com/90c233e62717129ae8236e89eab32db332bce2f1d08d652df2b0658475dcbcc5/feast-dev/feast/issues/7011
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/90c233e62717129ae8236e89eab32db332bce2f1d08d652df2b0658475dcbcc5/feast-dev/feast/issues/7011
og:image:altCurrent behavior Since 0.67.0 (#6855) the Prometheus metrics server runs in a multiprocessing.Process(daemon=True) started in the gunicorn master. Gunicorn workers are forked from the master and in...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernameaborgatin
hostnamegithub.com
expected-hostnamegithub.com
Noneb3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86
turbo-cache-controlno-preview
go-importgithub.com/feast-dev/feast git https://github.com/feast-dev/feast.git
octolytics-dimension-user_id57027613
octolytics-dimension-user_loginfeast-dev
octolytics-dimension-repository_id161133770
octolytics-dimension-repository_nwofeast-dev/feast
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id161133770
octolytics-dimension-repository_network_root_nwofeast-dev/feast
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release4a10310af51a1bfffbedc4fd81cb582acf0cb0ff
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/feast-dev/feast/issues/7011#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ffeast-dev%2Ffeast%2Fissues%2F7011
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ffeast-dev%2Ffeast%2Fissues%2F7011
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=feast-dev%2Ffeast
Reloadhttps://github.com/feast-dev/feast/issues/7011
Reloadhttps://github.com/feast-dev/feast/issues/7011
Reloadhttps://github.com/feast-dev/feast/issues/7011
feast-devhttps://github.com/feast-dev
feasthttps://github.com/feast-dev/feast
Codehttps://github.com/feast-dev/feast
Issues223 (223)https://github.com/feast-dev/feast/issues
Pull requests143 (143)https://github.com/feast-dev/feast/pulls
Discussionshttps://github.com/feast-dev/feast/discussions
Actionshttps://github.com/feast-dev/feast/actions
Security and quality1 (1)https://github.com/feast-dev/feast/security
Insightshttps://github.com/feast-dev/feast/insights
https://github.com/feast-dev/feast
New issuehttps://github.com/login?return_to=https://github.com/feast-dev/feast/issues/7011
New issuehttps://github.com/login?return_to=https://github.com/feast-dev/feast/issues/7011
#7012https://github.com/feast-dev/feast/pull/7012
feast serve: /metrics stops responding after the first gunicorn worker exits (regression in 0.67.0, #6855)https://github.com/feast-dev/feast/issues/7011#top
#7012https://github.com/feast-dev/feast/pull/7012
https://github.com/aborgatin
aborgatinhttps://github.com/aborgatin
on Oct 14, 2026https://github.com/feast-dev/feast/issues/7011#issue-5848088420
#6855https://github.com/feast-dev/feast/pull/6855
https://github.com/Mrun25
Mrun25https://github.com/Mrun25
on Oct 15, 2026https://github.com/feast-dev/feast/issues/7011#issuecomment-6198173411
@ntkatholehttps://github.com/ntkathole
ntkatholehttps://github.com/ntkathole
on Oct 18, 2026https://github.com/feast-dev/feast/issues/7011#event-33573282619
fix: Keep the metrics server running when a Gunicorn worker exitshttps://github.com/aborgatin/feast/commit/1f0df5e644164e07f62e9fa30e9d959231a51338
https://docs.github.com/articles/closing-issues-via-commit-messages
1f0df5ehttps://github.com/aborgatin/feast/commit/1f0df5e644164e07f62e9fa30e9d959231a51338
ntkatholehttps://github.com/ntkathole
completedhttps://github.com/feast-dev/feast/issues?q=is%3Aissue%20state%3Aclosed%20archived%3Afalse%20reason%3Acompleted
#7012https://github.com/feast-dev/feast/pull/7012
on Oct 18, 2026https://github.com/feast-dev/feast/issues/7011#event-33573290251
ntkatholehttps://github.com/ntkathole
on Oct 18, 2026https://github.com/feast-dev/feast/issues/7011#event-33575884812
fix: Keep the metrics server running when a Gunicorn worker exits (fe…https://github.com/patelchaitany/feast/commit/68c5d541c1ad728db6cf43ccaf815b24bea76b20
fe…https://github.com/feast-dev/feast/pull/7012
https://docs.github.com/articles/closing-issues-via-commit-messages
68c5d54https://github.com/patelchaitany/feast/commit/68c5d541c1ad728db6cf43ccaf815b24bea76b20
franciscojavierarceohttps://github.com/franciscojavierarceo
on Nov 10, 2026https://github.com/feast-dev/feast/issues/7011#event-35734988350
chore(release): release 0.68.0https://github.com/feast-dev/feast/commit/fb5e5d5e1df1903f41ac738cd49adac55ef44b88
fb5e5d5https://github.com/feast-dev/feast/commit/fb5e5d5e1df1903f41ac738cd49adac55ef44b88
shuchuhttps://github.com/shuchu
on Nov 11, 2026https://github.com/feast-dev/feast/issues/7011#event-35759492023
chore(release): release 0.68.0https://github.com/shuchu/feast/commit/5b94c9eb145d3fe7b84782133961639dc87ee951
5b94c9ehttps://github.com/shuchu/feast/commit/5b94c9eb145d3fe7b84782133961639dc87ee951
Sign up for freehttps://github.com/signup?return_to=https://github.com/feast-dev/feast/issues/7011
Sign in to commenthttps://github.com/login?return_to=https://github.com/feast-dev/feast/issues/7011
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.