René's URL Explorer Experiment


Title: Feature server: long GC pauses cause latency spikes on online reads · Issue #7088 · feast-dev/feast · GitHub

Open Graph Title: Feature server: long GC pauses cause latency spikes on online reads · Issue #7088 · feast-dev/feast

X Title: Feature server: long GC pauses cause latency spikes on online reads · Issue #7088 · feast-dev/feast

Description: Problem With feast serve (gunicorn + UvicornWorker), about 0.3% of /get-online-features requests take 50–100 ms, while p50 is 7 ms. The online store read timer shows the same spikes, so it looks like a slow Redis read. Redis is fine. The...

Open Graph Description: Problem With feast serve (gunicorn + UvicornWorker), about 0.3% of /get-online-features requests take 50–100 ms, while p50 is 7 ms. The online store read timer shows the same spikes, so it looks li...

X Description: Problem With feast serve (gunicorn + UvicornWorker), about 0.3% of /get-online-features requests take 50–100 ms, while p50 is 7 ms. The online store read timer shows the same spikes, so it looks li...

Opengraph URL: https://github.com/feast-dev/feast/issues/7088

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Feature server: long GC pauses cause latency spikes on online reads","articleBody":"## Problem\n\nWith `feast serve` (gunicorn + UvicornWorker), about 0.3% of `/get-online-features` requests take 50–100 ms, while p50 is 7 ms. The `online store read` timer shows the same spikes, so it looks like a slow Redis read. Redis is fine.\n\nThe cause is the Python cyclic GC. After startup a worker holds about 185k tracked objects: the registry, feature definitions, protobuf classes and modules. They live until the process exits, but every gen 2 collection walks all of them. On our setup one gen 2 collection takes 50–75 ms and frees nothing. It runs on the event loop thread, so every request waiting on the loop at that moment gets slower, including the timed Redis `await`.\n\n## Proposal\n\nAdd an opt-in `feast serve --gc-freeze` flag (and `feature_server.gc_freeze` in `feature_store.yaml`). It runs `gc.collect()` and `gc.freeze()` in the gunicorn master after the app and registry are loaded and before the workers fork. Workers inherit the frozen state, including workers restarted by `--max-requests`, so there is no extra cost per worker. Off by default.\n\nThis is a common fix for pre-fork servers (Instagram, Locust, vLLM `freeze_gc_heap`, Solara `gc_freeze`). Gunicorn has no built-in option (benoitc/gunicorn#1640).\n\nI have a PR ready.\n\n## Results\n\nLocal Docker setup from the repro steps below: one worker, 15 req/s for 10 min, one entity per request, 4 feature views with 137 features.\n\n| Setup | Requests | > 50 ms | p99.9 | max |\n|---|---|---|---|---|\n| master | 8,991 | 16 (0.18%) | 61 ms | 75 ms |\n| `--gc-freeze` | 8,933 | 0 | 29 ms | 37 ms |\n\nEach case was run twice with the same result. p50 and worker memory stay the same.\n\n## Repro\n\n1. Create a feature repo with a Redis online store, one entity and 4 feature views with about 140 Float32 features. Write rows for 20k entities.\n2. Start `feast serve --max-requests 100000` (one worker).\n3. Send requests for one random entity at 15 req/s with random gaps (open loop) for 10 min and record the client latency.\n4. Count the requests over 50 ms, then repeat with `feast serve --gc-freeze`.\n\nI can share my Docker setup and load script if that helps.\n\n## Related\n\n#7072 also saw this on the client side: `gc.freeze()` took p50 from 63 to 34 ms there. This issue is only about the feature server.\n\n","author":{"url":"https://github.com/aborgatin","@type":"Person","name":"aborgatin"},"datePublished":"2026-10-29T09:40:12Z","url":"https://github.com/feast-dev/feast/issues/7088"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:61898c6c-6005-be03-33c5-5b7455924950
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-idB06C:20741A:D90A39:11DEF9B:6AFE922F
html-safe-nonceed4b7e56c3d95e43b711b28299089ba914c6111efee80f5ad67f908dc8d7481f
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCMDZDOjIwNzQxQTpEOTBBMzk6MTFERUY5Qjo2QUZFOTIyRiIsInZpc2l0b3JfaWQiOiIyNzMwOTgzMDUwMzM3NzUxNTk5IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0=
visitor-hmac982d727cc9b55294cdd3dd13fa5b4bfed8801aad1d76651503742c09f7d04e32
hovercard-subject-tagissue:6096955005
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/feast-dev/feast/7088/issue_layout
twitter:imagehttps://opengraph.githubassets.com/650d0b1bd9292c470888a87ac5a6120257074966cafee8324e29e99367dd3263/feast-dev/feast/issues/7088
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/650d0b1bd9292c470888a87ac5a6120257074966cafee8324e29e99367dd3263/feast-dev/feast/issues/7088
og:image:altProblem With feast serve (gunicorn + UvicornWorker), about 0.3% of /get-online-features requests take 50–100 ms, while p50 is 7 ms. The online store read timer shows the same spikes, so it looks li...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernameaborgatin
hostnamegithub.com
expected-hostnamegithub.com
Noneb3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86
turbo-cache-controlno-preview
go-importgithub.com/feast-dev/feast git https://github.com/feast-dev/feast.git
octolytics-dimension-user_id57027613
octolytics-dimension-user_loginfeast-dev
octolytics-dimension-repository_id161133770
octolytics-dimension-repository_nwofeast-dev/feast
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id161133770
octolytics-dimension-repository_network_root_nwofeast-dev/feast
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release4a10310af51a1bfffbedc4fd81cb582acf0cb0ff
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/feast-dev/feast/issues/7088#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ffeast-dev%2Ffeast%2Fissues%2F7088
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ffeast-dev%2Ffeast%2Fissues%2F7088
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=feast-dev%2Ffeast
Reloadhttps://github.com/feast-dev/feast/issues/7088
Reloadhttps://github.com/feast-dev/feast/issues/7088
Reloadhttps://github.com/feast-dev/feast/issues/7088
feast-devhttps://github.com/feast-dev
feasthttps://github.com/feast-dev/feast
Codehttps://github.com/feast-dev/feast
Issues223 (223)https://github.com/feast-dev/feast/issues
Pull requests143 (143)https://github.com/feast-dev/feast/pulls
Discussionshttps://github.com/feast-dev/feast/discussions
Actionshttps://github.com/feast-dev/feast/actions
Security and quality1 (1)https://github.com/feast-dev/feast/security
Insightshttps://github.com/feast-dev/feast/insights
https://github.com/feast-dev/feast
New issuehttps://github.com/login?return_to=https://github.com/feast-dev/feast/issues/7088
New issuehttps://github.com/login?return_to=https://github.com/feast-dev/feast/issues/7088
#7089https://github.com/feast-dev/feast/pull/7089
Feature server: long GC pauses cause latency spikes on online readshttps://github.com/feast-dev/feast/issues/7088#top
#7089https://github.com/feast-dev/feast/pull/7089
https://github.com/aborgatin
aborgatinhttps://github.com/aborgatin
on Oct 29, 2026https://github.com/feast-dev/feast/issues/7088#issue-6096955005
benoitc/gunicorn#1640https://github.com/benoitc/gunicorn/issues/1640
#7072https://github.com/feast-dev/feast/issues/7072
shuchuhttps://github.com/shuchu
completedhttps://github.com/feast-dev/feast/issues?q=is%3Aissue%20state%3Aclosed%20archived%3Afalse%20reason%3Acompleted
#7089https://github.com/feast-dev/feast/pull/7089
on Nov 5, 2026https://github.com/feast-dev/feast/issues/7088#event-35299149492
Sign up for freehttps://github.com/signup?return_to=https://github.com/feast-dev/feast/issues/7088
Sign in to commenthttps://github.com/login?return_to=https://github.com/feast-dev/feast/issues/7088
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.