René's URL Explorer Experiment


Title: Tolerate etcd stalls on a loaded macOS build host (k3s exits on lost leader election) · Issue #252 · ExaDev/github-runner · GitHub

Open Graph Title: Tolerate etcd stalls on a loaded macOS build host (k3s exits on lost leader election) · Issue #252 · ExaDev/github-runner

X Title: Tolerate etcd stalls on a loaded macOS build host (k3s exits on lost leader election) · Issue #252 · ExaDev/github-runner

Description: On a macOS build host, the k3s node container of the runner cluster has been stopped and restarted nine times, most recently with an exit code of 0 and no out-of-memory kill, then restarted by the unless-stopped policy within 0.2 seconds...

Open Graph Description: On a macOS build host, the k3s node container of the runner cluster has been stopped and restarted nine times, most recently with an exit code of 0 and no out-of-memory kill, then restarted by the ...

X Description: On a macOS build host, the k3s node container of the runner cluster has been stopped and restarted nine times, most recently with an exit code of 0 and no out-of-memory kill, then restarted by the ...

Opengraph URL: https://github.com/ExaDev/github-runner/issues/252

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Tolerate etcd stalls on a loaded macOS build host (k3s exits on lost leader election)","articleBody":"On a macOS build host, the k3s node container of the runner cluster has been stopped and restarted nine times, most recently with an exit code of 0 and no out-of-memory kill, then restarted by the `unless-stopped` policy within 0.2 seconds. CI jobs running on that host's runners are lost each time.\n\n**Cause.** The node container's own log, a few lines before each restart, ends with `level=fatal msg=\"leaderelection lost for k3s-etcd\"` followed by `leaderelection lost for k3s`. k3s exits 0 when its embedded etcd loses its leader lease. Just before the last exit etcd was slow: read-only range requests took about 300 ms against an expected 100 ms and a transaction failed its retries. The slowness coincided with a host-side fault: the macOS file-event daemon (`fseventsd`) had run at full CPU for eleven days with swap near its ceiling, so the VM that hosts Docker was starved of I/O. After that daemon was restarted, swap use fell from about 16 GB to about 6 GB. The earlier restarts fall in the same swap-pressure period. The memory-pressure regrader is ruled out (its container logged nothing, the node carries no regrade level and the container's cumulative memory stall is about a second).\n\n**Remaining work, deferred until the next restart shows it is still needed.** A host stall long enough to miss the election timeout takes the control plane down and every CI job on its runners with it. Set the role's etcd heartbeat and election timeouts from the host's measured disk latency instead of the defaults (the node entrypoint already reads `K3S_ETCD_HEARTBEAT_INTERVAL` and `K3S_ETCD_ELECTION_TIMEOUT`), and alert when the host's swap use stays above half its ceiling.\n\n**Revisit trigger:** the next `leaderelection lost` restart after the file-event daemon was restarted.\n\n## Update: a lost voter hides the real stall rate\n\nWith the control plane at three etcd members and one of them down, the remaining two are both needed for quorum, and every Raft heartbeat to the dead member is dropped with \"sending buffer is full\". Those drops are noise from the dead member, not a measure of host load. Once the third member was restored, the dropped heartbeats and unreachable-peer warnings fell from about 1,600 per two minutes to none.\n\nThe k3s container on a loaded macOS build host still exits 0 on `leaderelection lost` when a heavy build or review job runs on the host: it restarted twice in one day, each time right after heavy runs, and every runner pod on that host died with it. With three voters the cluster stays up through such a restart. The hardening above (tolerating etcd stalls on that host) is still open, and so is keeping heavy build jobs off the voters.\n","author":{"url":"https://github.com/Mearman","@type":"Person","name":"Mearman"},"datePublished":"2026-11-14T06:33:36Z","url":"https://github.com/ExaDev/github-runner/issues/252"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:4ac8ea94-d924-0562-f71b-49ae5cada672
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-idDFF8:2140E0:2A810E:3A11F4:6B006963
html-safe-noncedb6192dc1fc1b2eb677d68ba933399cb5060c4f63a954b102dddf4d152e21b22
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJERkY4OjIxNDBFMDoyQTgxMEU6M0ExMUY0OjZCMDA2OTYzIiwidmlzaXRvcl9pZCI6IjM4MTI3NzgxNjk5NDYxMDYyMTEiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmac9ea124a9057b0ba48473cb4447c5e10e50baa795fbe575231a816d5ea3abc3b5
hovercard-subject-tagissue:6386387880
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/ExaDev/github-runner/252/issue_layout
twitter:imagehttps://opengraph.githubassets.com/f21080961c1e167f062697b11713cce25e6c044e34417bffdb4eb9ba2fb30129/ExaDev/github-runner/issues/252
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/f21080961c1e167f062697b11713cce25e6c044e34417bffdb4eb9ba2fb30129/ExaDev/github-runner/issues/252
og:image:altOn a macOS build host, the k3s node container of the runner cluster has been stopped and restarted nine times, most recently with an exit code of 0 and no out-of-memory kill, then restarted by the ...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernameMearman
hostnamegithub.com
expected-hostnamegithub.com
None6527e91c60c0b29d16606ff32ac5df8ecdc37a2a62205ea057a14d7762c870fa
turbo-cache-controlno-preview
go-importgithub.com/ExaDev/github-runner git https://github.com/ExaDev/github-runner.git
octolytics-dimension-user_id140118018
octolytics-dimension-user_loginExaDev
octolytics-dimension-repository_id1388126104
octolytics-dimension-repository_nwoExaDev/github-runner
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id1388126104
octolytics-dimension-repository_network_root_nwoExaDev/github-runner
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release46f1cd2ae3cd8ae1a81019b5d619742e76f13bbd
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/ExaDev/github-runner/issues/252#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FExaDev%2Fgithub-runner%2Fissues%2F252
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FExaDev%2Fgithub-runner%2Fissues%2F252
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=ExaDev%2Fgithub-runner
Reloadhttps://github.com/ExaDev/github-runner/issues/252
Reloadhttps://github.com/ExaDev/github-runner/issues/252
Reloadhttps://github.com/ExaDev/github-runner/issues/252
ExaDevhttps://github.com/ExaDev
github-runnerhttps://github.com/ExaDev/github-runner
Codehttps://github.com/ExaDev/github-runner
Issues46 (46)https://github.com/ExaDev/github-runner/issues
Pull requests6 (6)https://github.com/ExaDev/github-runner/pulls
Actionshttps://github.com/ExaDev/github-runner/actions
Security and qualityhttps://github.com/ExaDev/github-runner/security
Insightshttps://github.com/ExaDev/github-runner/insights
https://github.com/ExaDev/github-runner
New issuehttps://github.com/login?return_to=https://github.com/ExaDev/github-runner/issues/252
New issuehttps://github.com/login?return_to=https://github.com/ExaDev/github-runner/issues/252
Tolerate etcd stalls on a loaded macOS build host (k3s exits on lost leader election)https://github.com/ExaDev/github-runner/issues/252#top
deferredDeferred until a stated triggerhttps://github.com/ExaDev/github-runner/issues?q=state%3Aopen%20label%3A%22deferred%22
https://github.com/Mearman
Mearmanhttps://github.com/Mearman
on Nov 14, 2026https://github.com/ExaDev/github-runner/issues/252#issue-6386387880
Mearmanhttps://github.com/Mearman
deferredDeferred until a stated triggerhttps://github.com/ExaDev/github-runner/issues?q=state%3Aopen%20label%3A%22deferred%22
on Nov 14, 2026https://github.com/ExaDev/github-runner/issues/252#event-36105787589
Mearmanhttps://github.com/Mearman
on Nov 14, 2026https://github.com/ExaDev/github-runner/issues/252#event-36106303630
Sign up for freehttps://github.com/signup?return_to=https://github.com/ExaDev/github-runner/issues/252
Sign in to commenthttps://github.com/login?return_to=https://github.com/ExaDev/github-runner/issues/252
deferredDeferred until a stated triggerhttps://github.com/ExaDev/github-runner/issues?q=state%3Aopen%20label%3A%22deferred%22
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.