Title: Feature server: long GC pauses cause latency spikes on online reads · Issue #7088 · feast-dev/feast · GitHub
Open Graph Title: Feature server: long GC pauses cause latency spikes on online reads · Issue #7088 · feast-dev/feast
X Title: Feature server: long GC pauses cause latency spikes on online reads · Issue #7088 · feast-dev/feast
Description: Problem With feast serve (gunicorn + UvicornWorker), about 0.3% of /get-online-features requests take 50–100 ms, while p50 is 7 ms. The online store read timer shows the same spikes, so it looks like a slow Redis read. Redis is fine. The...
Open Graph Description: Problem With feast serve (gunicorn + UvicornWorker), about 0.3% of /get-online-features requests take 50–100 ms, while p50 is 7 ms. The online store read timer shows the same spikes, so it looks li...
X Description: Problem With feast serve (gunicorn + UvicornWorker), about 0.3% of /get-online-features requests take 50–100 ms, while p50 is 7 ms. The online store read timer shows the same spikes, so it looks li...
Opengraph URL: https://github.com/feast-dev/feast/issues/7088
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Feature server: long GC pauses cause latency spikes on online reads","articleBody":"## Problem\n\nWith `feast serve` (gunicorn + UvicornWorker), about 0.3% of `/get-online-features` requests take 50–100 ms, while p50 is 7 ms. The `online store read` timer shows the same spikes, so it looks like a slow Redis read. Redis is fine.\n\nThe cause is the Python cyclic GC. After startup a worker holds about 185k tracked objects: the registry, feature definitions, protobuf classes and modules. They live until the process exits, but every gen 2 collection walks all of them. On our setup one gen 2 collection takes 50–75 ms and frees nothing. It runs on the event loop thread, so every request waiting on the loop at that moment gets slower, including the timed Redis `await`.\n\n## Proposal\n\nAdd an opt-in `feast serve --gc-freeze` flag (and `feature_server.gc_freeze` in `feature_store.yaml`). It runs `gc.collect()` and `gc.freeze()` in the gunicorn master after the app and registry are loaded and before the workers fork. Workers inherit the frozen state, including workers restarted by `--max-requests`, so there is no extra cost per worker. Off by default.\n\nThis is a common fix for pre-fork servers (Instagram, Locust, vLLM `freeze_gc_heap`, Solara `gc_freeze`). Gunicorn has no built-in option (benoitc/gunicorn#1640).\n\nI have a PR ready.\n\n## Results\n\nLocal Docker setup from the repro steps below: one worker, 15 req/s for 10 min, one entity per request, 4 feature views with 137 features.\n\n| Setup | Requests | > 50 ms | p99.9 | max |\n|---|---|---|---|---|\n| master | 8,991 | 16 (0.18%) | 61 ms | 75 ms |\n| `--gc-freeze` | 8,933 | 0 | 29 ms | 37 ms |\n\nEach case was run twice with the same result. p50 and worker memory stay the same.\n\n## Repro\n\n1. Create a feature repo with a Redis online store, one entity and 4 feature views with about 140 Float32 features. Write rows for 20k entities.\n2. Start `feast serve --max-requests 100000` (one worker).\n3. Send requests for one random entity at 15 req/s with random gaps (open loop) for 10 min and record the client latency.\n4. Count the requests over 50 ms, then repeat with `feast serve --gc-freeze`.\n\nI can share my Docker setup and load script if that helps.\n\n## Related\n\n#7072 also saw this on the client side: `gc.freeze()` took p50 from 63 to 34 ms there. This issue is only about the feature server.\n\n","author":{"url":"https://github.com/aborgatin","@type":"Person","name":"aborgatin"},"datePublished":"2026-10-29T09:40:12Z","url":"https://github.com/feast-dev/feast/issues/7088"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:61898c6c-6005-be03-33c5-5b7455924950 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | B06C:20741A:D90A39:11DEF9B:6AFE922F |
| html-safe-nonce | ed4b7e56c3d95e43b711b28299089ba914c6111efee80f5ad67f908dc8d7481f |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCMDZDOjIwNzQxQTpEOTBBMzk6MTFERUY5Qjo2QUZFOTIyRiIsInZpc2l0b3JfaWQiOiIyNzMwOTgzMDUwMzM3NzUxNTk5IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | 982d727cc9b55294cdd3dd13fa5b4bfed8801aad1d76651503742c09f7d04e32 |
| hovercard-subject-tag | issue:6096955005 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/feast-dev/feast/7088/issue_layout |
| twitter:image | https://opengraph.githubassets.com/650d0b1bd9292c470888a87ac5a6120257074966cafee8324e29e99367dd3263/feast-dev/feast/issues/7088 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/650d0b1bd9292c470888a87ac5a6120257074966cafee8324e29e99367dd3263/feast-dev/feast/issues/7088 |
| og:image:alt | Problem With feast serve (gunicorn + UvicornWorker), about 0.3% of /get-online-features requests take 50–100 ms, while p50 is 7 ms. The online store read timer shows the same spikes, so it looks li... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | aborgatin |
| hostname | github.com |
| expected-hostname | github.com |
| None | b3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86 |
| turbo-cache-control | no-preview |
| go-import | github.com/feast-dev/feast git https://github.com/feast-dev/feast.git |
| octolytics-dimension-user_id | 57027613 |
| octolytics-dimension-user_login | feast-dev |
| octolytics-dimension-repository_id | 161133770 |
| octolytics-dimension-repository_nwo | feast-dev/feast |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 161133770 |
| octolytics-dimension-repository_network_root_nwo | feast-dev/feast |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 4a10310af51a1bfffbedc4fd81cb582acf0cb0ff |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width