Title: Intermittent deadlock on `feast serve` startup under gunicorn: feature-freshness thread starts pre-fork and can freeze an import lock forever · Issue #6647 · feast-dev/feast · GitHub
Open Graph Title: Intermittent deadlock on `feast serve` startup under gunicorn: feature-freshness thread starts pre-fork and can freeze an import lock forever · Issue #6647 · feast-dev/feast
X Title: Intermittent deadlock on `feast serve` startup under gunicorn: feature-freshness thread starts pre-fork and can freeze an import lock forever · Issue #6647 · feast-dev/feast
Description: Describe the bug feast serve (running under gunicorn, uvicorn_worker.UvicornWorker) can hang forever at startup with no error, no traceback, and no log output beyond "Waiting for application startup.". It's intermittent — most starts suc...
Open Graph Description: Describe the bug feast serve (running under gunicorn, uvicorn_worker.UvicornWorker) can hang forever at startup with no error, no traceback, and no log output beyond "Waiting for application startu...
X Description: Describe the bug feast serve (running under gunicorn, uvicorn_worker.UvicornWorker) can hang forever at startup with no error, no traceback, and no log output beyond "Waiting for application s...
Opengraph URL: https://github.com/feast-dev/feast/issues/6647
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Intermittent deadlock on `feast serve` startup under gunicorn: feature-freshness thread starts pre-fork and can freeze an import lock forever","articleBody":"\n## Describe the bug\n\n`feast serve` (running under gunicorn, `uvicorn_worker.UvicornWorker`) can hang forever at startup with **no error, no traceback, and no log output** beyond `\"Waiting for application startup.\"`. It's intermittent — most starts succeed, but occasionally a worker freezes permanently and never becomes ready.\n\n## Root cause\n\n`feast_metrics.start_metrics_server()` is called from `feature_server.start_server()`, which runs in the **gunicorn master process**, before `FeastServeApplication(...).run()` triggers gunicorn's `fork()` of the worker process(es).\n\n`start_metrics_server()` correctly defers CPU/memory resource monitoring to *after* the fork (there's an explicit `start_resource_monitoring=not uses_gunicorn` check, plus a `post_worker_init` hook that calls `init_worker_monitoring()`). However, it does **not** apply the same treatment to the feature-freshness thread:\n\n```python\nif _config.freshness:\n freshness_thread = threading.Thread(\n target=monitor_freshness, args=(store, 30), daemon=True\n )\n freshness_thread.start()\n```\n\nThis thread is started unconditionally in the master (whenever `feature_server.metrics.freshness` is enabled), and its very first action — with no delay — is `update_feature_freshness(store)` → `store.list_feature_views(...)`, which lazily builds the registry for the first time. For registry backends that need a lazy DBAPI import (e.g. the SQL registry importing `pymysql` via SQLAlchemy's `create_engine()`), this means a background thread in the gunicorn master may be **mid-import**, holding CPython's per-module import lock, at the exact moment gunicorn forks a worker.\n\nPOSIX `fork()` only duplicates the calling thread into the child process — every other thread that existed in the parent (including this freshness thread) simply ceases to exist in the worker. If the fork happens while that thread holds a module's import lock, the lock is left permanently in the \"locked\"/`_initializing` state inside the new worker, and there is no longer any thread anywhere that can finish that import and release it. When the worker's own FastAPI startup later tries to build its own registry and hits the same import, it deadlocks forever.\n\nThis matches a known general class of hazard (fork() + threads + an in-flight import lock — see e.g. https://bugs.python.org/issue6380), and it explains the intermittency exactly: it only manifests if the fork lands inside that narrow timing window.\n\nWe confirmed this live by attaching `py-spy` to a genuinely frozen worker process and dumping its stack — the main thread was blocked acquiring CPython's per-module import lock inside `SQLAlchemy`'s lazy `import_dbapi()` (dialects/mysql/pymysql.py), called from `SqlRegistry.__init__` → `feature_store.py`'s `registry` property → `refresh_registry()` → `feature_server.py`'s `async_refresh()` → the FastAPI `lifespan`.\n\nWe also verified against CPython's actual `importlib._bootstrap._find_and_load` that the per-module import lock is **only** acquired if a module is missing from `sys.modules` or still mid-initialization — once a module has fully finished importing, every later import statement, from any thread, before or after any number of forks, returns the cached module without ever touching the lock again. This is why the freshness thread's *pre-fork* timing is what makes it dangerous, and it points at the fix.\n\n## To Reproduce\n\n1. Configure a registry backend that performs a lazy import on first use (e.g. `registry_type: sql` with a MySQL connection string, which lazily imports `pymysql` via SQLAlchemy).\n2. Enable `feature_server.metrics.freshness: true` in `feature_store.yaml`.\n3. Run `feast serve --metrics` under gunicorn (Linux, non-Windows path) with 1+ workers.\n4. Restart the process repeatedly (e.g. in a container orchestrator that recycles the pod/process on every deploy or crash). Occasionally — timing-dependent, not consistently reproducible on demand — a worker will hang forever at `\"Waiting for application startup.\"` with zero output.\n\n## Expected behavior\n\nThe feature-freshness thread (and ideally anything else `start_metrics_server()` spawns pre-fork that can trigger a lazy import) should be deferred to run after gunicorn forks its workers, the same way resource monitoring already is — e.g. via the existing `post_worker_init` hook — rather than starting unconditionally in the master before the fork.\n\n## Workaround\n\nForce the DBAPI driver (e.g. `pymysql`) to finish importing once, synchronously, as the very first statement of the process — before `feast`'s CLI runs, before any thread exists, and before gunicorn forks anything. A tiny wrapper around the CLI entrypoint works:\n\n```python\nimport pymysql # must be first — see explanation above\nimport sys\nfrom feast.cli.cli import cli\n\nif __name__ == \"__main__\":\n sys.exit(cli())\n```\n\n...invoked instead of the `feast` console script (e.g. `python entrypoint.py serve ...`). This is unrelated to thread timing luck — once the module is fully imported, the import lock is never touched again for it, regardless of what any other thread does afterward.\n\n## Environment\n\n- feast version: 0.62.0\n- Python: 3.11\n- OS: Linux (containerized), gunicorn + `uvicorn_worker.UvicornWorker`\n- Registry: SQL registry (`registry_type: sql`, MySQL via pymysql)\n","author":{"url":"https://github.com/casaar97","@type":"Person","name":"casaar97"},"datePublished":"2026-07-27T11:05:56Z","url":"https://github.com/feast-dev/feast/issues/6647"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:a2181178-607d-db45-55f8-d54d7e68546a |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | 8D9E:20D634:DE2B4C:1285410:6AFE981B |
| html-safe-nonce | 8841bef4276b95bb20c51ae93f59967ec88b873c90ac0330c89d05e47507fd7b |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI4RDlFOjIwRDYzNDpERTJCNEM6MTI4NTQxMDo2QUZFOTgxQiIsInZpc2l0b3JfaWQiOiI0OTAyNjM4ODAwMDQ5NDQ0ODkxIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | 1ef5449f607c42160425e4bf7ac24617745aadd40f2589524a694840460d7e88 |
| hovercard-subject-tag | issue:4987215344 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/feast-dev/feast/6647/issue_layout |
| twitter:image | https://opengraph.githubassets.com/512ff466daa8273e63d3a4b225147c5a360cd17a0cefec5872be6781d2407643/feast-dev/feast/issues/6647 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/512ff466daa8273e63d3a4b225147c5a360cd17a0cefec5872be6781d2407643/feast-dev/feast/issues/6647 |
| og:image:alt | Describe the bug feast serve (running under gunicorn, uvicorn_worker.UvicornWorker) can hang forever at startup with no error, no traceback, and no log output beyond "Waiting for application startu... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | casaar97 |
| hostname | github.com |
| expected-hostname | github.com |
| None | b3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86 |
| turbo-cache-control | no-preview |
| go-import | github.com/feast-dev/feast git https://github.com/feast-dev/feast.git |
| octolytics-dimension-user_id | 57027613 |
| octolytics-dimension-user_login | feast-dev |
| octolytics-dimension-repository_id | 161133770 |
| octolytics-dimension-repository_nwo | feast-dev/feast |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 161133770 |
| octolytics-dimension-repository_network_root_nwo | feast-dev/feast |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 4a10310af51a1bfffbedc4fd81cb582acf0cb0ff |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width