René's URL Explorer Experiment


Title: feast serve: Gunicorn worker deadlocks on startup when forked during a /metrics request (follow-up to #6647) · Issue #6928 · feast-dev/feast · GitHub

Open Graph Title: feast serve: Gunicorn worker deadlocks on startup when forked during a /metrics request (follow-up to #6647) · Issue #6928 · feast-dev/feast

X Title: feast serve: Gunicorn worker deadlocks on startup when forked during a /metrics request (follow-up to #6647) · Issue #6928 · feast-dev/feast

Description: Expected behavior With metrics enabled, Gunicorn replaces a worker after --max-requests requests and the new worker starts serving. Current behavior Occasionally the new worker never starts. The log ends with Worker exiting (pid: N) and ...

Open Graph Description: Expected behavior With metrics enabled, Gunicorn replaces a worker after --max-requests requests and the new worker starts serving. Current behavior Occasionally the new worker never starts. The lo...

X Description: Expected behavior With metrics enabled, Gunicorn replaces a worker after --max-requests requests and the new worker starts serving. Current behavior Occasionally the new worker never starts. The lo...

Opengraph URL: https://github.com/feast-dev/feast/issues/6928

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"feast serve: Gunicorn worker deadlocks on startup when forked during a /metrics request (follow-up to #6647)","articleBody":"### Expected behavior\n\nWith metrics enabled, Gunicorn replaces a worker after `--max-requests` requests and the new worker\nstarts serving.\n\n### Current behavior\n\nOccasionally the new worker never starts. The log ends with `Worker exiting (pid: N)` and there is no\n`Booting worker with pid: ...` after it. The worker process exists but is blocked forever; the master\nkeeps the listening socket open and the metrics server (a thread in the master) keeps answering\n`/metrics`, so TCP health checks still pass while no request is served. Gunicorn's worker timeout does\nnot kill it either (separate Gunicorn issue: a worker that hangs before its first `notify()` is never\ntimed out).\n\nIn our production (2 deployments, 26 pods, ~17,500 worker recycles per day) this happened 14 times\nin 7 days; each pod served nothing for ~1 h until its listen backlog filled up and Kubernetes' TCP\nliveness probe failed.\n\n### Root cause\n\nSame class of bug as #6647 (fixed for the freshness thread in #6648): a thread started in the\nGunicorn master before `fork()`.\n\n`feature_server.start_server()` calls `feast_metrics.start_metrics_server()` in the Gunicorn master,\nbefore `FeastServeApplication(...).run()` forks the workers. That starts the metrics HTTP server\n(`wsgiref` `make_server(...)`, now `_make_metrics_httpd`) as a daemon thread with the default\n`WSGIRequestHandler`, which writes an access-log line to `sys.stderr` for **every** request\n(`log_message`).\n\nWhile that thread is inside `sys.stderr.write`, it holds the internal lock of stderr's\n`BufferedWriter`. If Gunicorn forks a worker at that moment, the child inherits the lock in the locked\nstate, and the thread that would release it does not exist in the child. The child's first action is\n`self.log.info(\"Booting worker with pid: %s\", ...)` (`gunicorn/arbiter.py`, `spawn_worker`), which\nwrites to stderr and blocks forever. Python's logging module re-initialises its own handler locks\nafter fork; the `BufferedWriter` lock is not re-initialised.\n\nPython warns about exactly this when `feast serve` starts:\n\n```\ngunicorn/arbiter.py:666: DeprecationWarning: This process (pid=16635) is multi-threaded, use of fork() may lead to deadlocks in the child.\n  pid = os.fork()\n```\n\n### Evidence\n\n- Python stack of a stuck worker (Linux, `faulthandler` dump), identical in all 10 hangs we captured:\n  ```\n  File \"/usr/local/lib/python3.12/logging/__init__.py\", line 1163 in emit      # stream.write(...)\n  File \"/usr/local/lib/python3.12/logging/__init__.py\", line 1028 in handle\n  ...\n  File \".../gunicorn/glogging.py\", line 277 in info\n  File \".../gunicorn/arbiter.py\", line 680 in spawn_worker                      # \"Booting worker\"\n  File \".../gunicorn/arbiter.py\", line 719 in spawn_workers\n  File \".../gunicorn/arbiter.py\", line 634 in manage_workers\n  File \".../gunicorn/arbiter.py\", line 228 in run\n  File \".../feast/feature_server.py\", line 941 in start_server\n  ```\n  The stuck worker has 1 thread, waiting on a futex; the master has 2 threads (main + metrics server).\n- Native stack of a stuck worker (macOS `sample`, 879 of 879 samples):\n  ```\n  _io_TextIOWrapper_write → _textiowrapper_writeflush → _io_BufferedWriter_write\n    → _enter_buffered_busy → PyThread_acquire_lock_timed → __psynch_cvwait\n  ```\n- Production: in 12 of 12 hangs, a `/metrics` scrape was logged 0–1 s after the last `Worker exiting`;\n  for normal recycles that happens 9.3% of the time (39 of 420).\n- Controlled test (Kubernetes, `--max-requests 20`, 4 back-to-back `/metrics` scrapers, a fresh pod\n  every 100 forks): **5 hangs in 318 forks** without a change; **0 hangs in 2,000 forks** when only the\n  access log of the metrics server is silenced (the change proposed in the PR). Locally (macOS, steps\n  below): a hang after 12–29 forks without the change, 0 in 336 forks with it.\n\n### Steps to reproduce\n\nFeast 0.65.0 or `master`, no external services:\n\n```bash\nfeast init -m demo && cd demo/feature_repo\ncat > feature_store.yaml \u003c\u003c'EOF'\nproject: demo\nregistry: data/registry.db\nprovider: local\nonline_store:\n    type: sqlite\n    path: data/online_store.db\nentity_key_serialization_version: 3\nfeature_server:\n    metrics:\n        enabled: true\n        freshness: false   # only the metrics server thread (freshness was #6647)\nEOF\nmkdir -p data && feast apply\n\n# terminal 1: recycle the worker every 20 requests\nfeast serve --max-requests 20 --max-requests-jitter 0 2>&1 | tee serve.log\n\n# terminal 2: keep the metrics thread busy + send requests\nfor i in 1 2 3 4; do (while true; do curl -s -o /dev/null http://127.0.0.1:8000/metrics; done) & done\nwhile true; do curl -s -o /dev/null -m 2 http://127.0.0.1:6566/health; done\n```\n\nWithin seconds to a few minutes, `serve.log` stops at `Worker exiting (pid: N)` with no\n`Booting worker` after it, and `/health` stops answering. `py-spy dump --pid \u003cchild>` (or `sample`\non macOS) shows the child blocked in the stderr write.\n\n### Possible solution\n\n1. Minimal: give the metrics server a request handler that does not log\n   (`make_server(..., handler_class=...)` with a no-op `log_message`), as prometheus_client's own\n   `start_http_server` does (`_SilentHandler`). PR to follow. This removes the stderr write we observed.\n2. More complete: do not run any thread in the Gunicorn master while it forks, e.g. serve the\n   multiprocess metrics from a separate process. Any other lock taken by the metrics thread could\n   cause the same deadlock in the future.\n\n### Environment\n\nFeast 0.65.0 (also checked 0.66.0 and `master` @ bc5aeef), gunicorn 25.0.3, uvicorn-worker 0.3.0,\nPython 3.12, `python:3.12-slim` on Linux arm64 (EKS) and macOS.\n","author":{"url":"https://github.com/aborgatin","@type":"Person","name":"aborgatin"},"datePublished":"2026-10-02T10:08:50Z","url":"https://github.com/feast-dev/feast/issues/6928"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:e32cdbbf-cca1-e1ca-9e01-ce7aaeafbbb4
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-id8F72:132CCD:DD2938:13066A8:6AFE9521
html-safe-noncefe3aa6eccb87cfc791e92fefb25c83ba2b51f42f58aeec49c7f7cae3b1fe4e93
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI4RjcyOjEzMkNDRDpERDI5Mzg6MTMwNjZBODo2QUZFOTUyMSIsInZpc2l0b3JfaWQiOiI3MjY1OTIxNzE4MzMyMzMxMjk3IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0=
visitor-hmacea88d75a5eef407e5ad1d2031ae19a32b1da4b65319882bf285bd60fad41d18f
hovercard-subject-tagissue:5675946778
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/feast-dev/feast/6928/issue_layout
twitter:imagehttps://opengraph.githubassets.com/10c20f7b65ce37e0a7cc23eb96f26fa6d9f11b289c2cf3af12a76dcf8fb67fec/feast-dev/feast/issues/6928
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/10c20f7b65ce37e0a7cc23eb96f26fa6d9f11b289c2cf3af12a76dcf8fb67fec/feast-dev/feast/issues/6928
og:image:altExpected behavior With metrics enabled, Gunicorn replaces a worker after --max-requests requests and the new worker starts serving. Current behavior Occasionally the new worker never starts. The lo...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernameaborgatin
hostnamegithub.com
expected-hostnamegithub.com
Noneb3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86
turbo-cache-controlno-preview
go-importgithub.com/feast-dev/feast git https://github.com/feast-dev/feast.git
octolytics-dimension-user_id57027613
octolytics-dimension-user_loginfeast-dev
octolytics-dimension-repository_id161133770
octolytics-dimension-repository_nwofeast-dev/feast
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id161133770
octolytics-dimension-repository_network_root_nwofeast-dev/feast
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release4a10310af51a1bfffbedc4fd81cb582acf0cb0ff
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/feast-dev/feast/issues/6928#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ffeast-dev%2Ffeast%2Fissues%2F6928
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ffeast-dev%2Ffeast%2Fissues%2F6928
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=feast-dev%2Ffeast
Reloadhttps://github.com/feast-dev/feast/issues/6928
Reloadhttps://github.com/feast-dev/feast/issues/6928
Reloadhttps://github.com/feast-dev/feast/issues/6928
feast-devhttps://github.com/feast-dev
feasthttps://github.com/feast-dev/feast
Codehttps://github.com/feast-dev/feast
Issues223 (223)https://github.com/feast-dev/feast/issues
Pull requests143 (143)https://github.com/feast-dev/feast/pulls
Discussionshttps://github.com/feast-dev/feast/discussions
Actionshttps://github.com/feast-dev/feast/actions
Security and quality1 (1)https://github.com/feast-dev/feast/security
Insightshttps://github.com/feast-dev/feast/insights
https://github.com/feast-dev/feast
New issuehttps://github.com/login?return_to=https://github.com/feast-dev/feast/issues/6928
New issuehttps://github.com/login?return_to=https://github.com/feast-dev/feast/issues/6928
#6929https://github.com/feast-dev/feast/pull/6929
feast serve: Gunicorn worker deadlocks on startup when forked during a /metrics request (follow-up to #6647)https://github.com/feast-dev/feast/issues/6928#top
#6929https://github.com/feast-dev/feast/pull/6929
https://github.com/aborgatin
aborgatinhttps://github.com/aborgatin
on Oct 2, 2026https://github.com/feast-dev/feast/issues/6928#issue-5675946778
#6647https://github.com/feast-dev/feast/issues/6647
#6648https://github.com/feast-dev/feast/pull/6648
bc5aeefhttps://github.com/feast-dev/feast/commit/bc5aeef7c8146dd0879a4dc8925e840247b7366f
aborgatinhttps://github.com/aborgatin
on Oct 6, 2026https://github.com/feast-dev/feast/issues/6928#event-32578485770
fix: Do not log /metrics requests from the pre-fork metrics server th…https://github.com/aborgatin/feast/commit/8e0f1e4f30ee9c92baddfd869f2cb53476854288
https://docs.github.com/articles/closing-issues-via-commit-messages
8e0f1e4https://github.com/aborgatin/feast/commit/8e0f1e4f30ee9c92baddfd869f2cb53476854288
ntkatholehttps://github.com/ntkathole
on Oct 7, 2026https://github.com/feast-dev/feast/issues/6928#event-32659645802
fix: Do not log /metrics requests from the pre-fork metrics server th…https://github.com/aborgatin/feast/commit/9237fb099825a080c992b3b4baaebb0de707ff87
https://docs.github.com/articles/closing-issues-via-commit-messages
9237fb0https://github.com/aborgatin/feast/commit/9237fb099825a080c992b3b4baaebb0de707ff87
ntkatholehttps://github.com/ntkathole
completedhttps://github.com/feast-dev/feast/issues?q=is%3Aissue%20state%3Aclosed%20archived%3Afalse%20reason%3Acompleted
#6929https://github.com/feast-dev/feast/pull/6929
on Oct 7, 2026https://github.com/feast-dev/feast/issues/6928#event-32659653963
ntkatholehttps://github.com/ntkathole
on Oct 7, 2026https://github.com/feast-dev/feast/issues/6928#event-32659704156
fix: Do not log /metrics requests from the pre-fork metrics server th…https://github.com/ntkathole/feast/commit/7087f1f91495c61b8a8c9d79231049b5271111e3
https://docs.github.com/articles/closing-issues-via-commit-messages
7087f1fhttps://github.com/ntkathole/feast/commit/7087f1f91495c61b8a8c9d79231049b5271111e3
chore(release): release 0.67.0https://github.com/feast-dev/feast/commit/2dfb9e6d1a37719fd11028d916094050f1cdd552
https://docs.github.com/articles/closing-issues-via-commit-messages
2dfb9e6https://github.com/feast-dev/feast/commit/2dfb9e6d1a37719fd11028d916094050f1cdd552
fix: Merge upstream feast-dev/feast master - resolve CSV conflict (#247)https://github.com/opendatahub-io/feast/commit/2b0b7d75acd0a1fbee53a0e77a81010905a82fbc
#247https://github.com/opendatahub-io/feast/pull/247
https://docs.github.com/articles/closing-issues-via-commit-messages
2b0b7d7https://github.com/opendatahub-io/feast/commit/2b0b7d75acd0a1fbee53a0e77a81010905a82fbc
Sign up for freehttps://github.com/signup?return_to=https://github.com/feast-dev/feast/issues/6928
Sign in to commenthttps://github.com/login?return_to=https://github.com/feast-dev/feast/issues/6928
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.