Title: Tolerate etcd stalls on a loaded macOS build host (k3s exits on lost leader election) · Issue #252 · ExaDev/github-runner · GitHub
Open Graph Title: Tolerate etcd stalls on a loaded macOS build host (k3s exits on lost leader election) · Issue #252 · ExaDev/github-runner
X Title: Tolerate etcd stalls on a loaded macOS build host (k3s exits on lost leader election) · Issue #252 · ExaDev/github-runner
Description: On a macOS build host, the k3s node container of the runner cluster has been stopped and restarted nine times, most recently with an exit code of 0 and no out-of-memory kill, then restarted by the unless-stopped policy within 0.2 seconds...
Open Graph Description: On a macOS build host, the k3s node container of the runner cluster has been stopped and restarted nine times, most recently with an exit code of 0 and no out-of-memory kill, then restarted by the ...
X Description: On a macOS build host, the k3s node container of the runner cluster has been stopped and restarted nine times, most recently with an exit code of 0 and no out-of-memory kill, then restarted by the ...
Opengraph URL: https://github.com/ExaDev/github-runner/issues/252
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Tolerate etcd stalls on a loaded macOS build host (k3s exits on lost leader election)","articleBody":"On a macOS build host, the k3s node container of the runner cluster has been stopped and restarted nine times, most recently with an exit code of 0 and no out-of-memory kill, then restarted by the `unless-stopped` policy within 0.2 seconds. CI jobs running on that host's runners are lost each time.\n\n**Cause.** The node container's own log, a few lines before each restart, ends with `level=fatal msg=\"leaderelection lost for k3s-etcd\"` followed by `leaderelection lost for k3s`. k3s exits 0 when its embedded etcd loses its leader lease. Just before the last exit etcd was slow: read-only range requests took about 300 ms against an expected 100 ms and a transaction failed its retries. The slowness coincided with a host-side fault: the macOS file-event daemon (`fseventsd`) had run at full CPU for eleven days with swap near its ceiling, so the VM that hosts Docker was starved of I/O. After that daemon was restarted, swap use fell from about 16 GB to about 6 GB. The earlier restarts fall in the same swap-pressure period. The memory-pressure regrader is ruled out (its container logged nothing, the node carries no regrade level and the container's cumulative memory stall is about a second).\n\n**Remaining work, deferred until the next restart shows it is still needed.** A host stall long enough to miss the election timeout takes the control plane down and every CI job on its runners with it. Set the role's etcd heartbeat and election timeouts from the host's measured disk latency instead of the defaults (the node entrypoint already reads `K3S_ETCD_HEARTBEAT_INTERVAL` and `K3S_ETCD_ELECTION_TIMEOUT`), and alert when the host's swap use stays above half its ceiling.\n\n**Revisit trigger:** the next `leaderelection lost` restart after the file-event daemon was restarted.\n\n## Update: a lost voter hides the real stall rate\n\nWith the control plane at three etcd members and one of them down, the remaining two are both needed for quorum, and every Raft heartbeat to the dead member is dropped with \"sending buffer is full\". Those drops are noise from the dead member, not a measure of host load. Once the third member was restored, the dropped heartbeats and unreachable-peer warnings fell from about 1,600 per two minutes to none.\n\nThe k3s container on a loaded macOS build host still exits 0 on `leaderelection lost` when a heavy build or review job runs on the host: it restarted twice in one day, each time right after heavy runs, and every runner pod on that host died with it. With three voters the cluster stays up through such a restart. The hardening above (tolerating etcd stalls on that host) is still open, and so is keeping heavy build jobs off the voters.\n","author":{"url":"https://github.com/Mearman","@type":"Person","name":"Mearman"},"datePublished":"2026-11-14T06:33:36Z","url":"https://github.com/ExaDev/github-runner/issues/252"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:4ac8ea94-d924-0562-f71b-49ae5cada672 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | DFF8:2140E0:2A810E:3A11F4:6B006963 |
| html-safe-nonce | db6192dc1fc1b2eb677d68ba933399cb5060c4f63a954b102dddf4d152e21b22 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJERkY4OjIxNDBFMDoyQTgxMEU6M0ExMUY0OjZCMDA2OTYzIiwidmlzaXRvcl9pZCI6IjM4MTI3NzgxNjk5NDYxMDYyMTEiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | 9ea124a9057b0ba48473cb4447c5e10e50baa795fbe575231a816d5ea3abc3b5 |
| hovercard-subject-tag | issue:6386387880 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/ExaDev/github-runner/252/issue_layout |
| twitter:image | https://opengraph.githubassets.com/f21080961c1e167f062697b11713cce25e6c044e34417bffdb4eb9ba2fb30129/ExaDev/github-runner/issues/252 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/f21080961c1e167f062697b11713cce25e6c044e34417bffdb4eb9ba2fb30129/ExaDev/github-runner/issues/252 |
| og:image:alt | On a macOS build host, the k3s node container of the runner cluster has been stopped and restarted nine times, most recently with an exit code of 0 and no out-of-memory kill, then restarted by the ... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | Mearman |
| hostname | github.com |
| expected-hostname | github.com |
| None | 6527e91c60c0b29d16606ff32ac5df8ecdc37a2a62205ea057a14d7762c870fa |
| turbo-cache-control | no-preview |
| go-import | github.com/ExaDev/github-runner git https://github.com/ExaDev/github-runner.git |
| octolytics-dimension-user_id | 140118018 |
| octolytics-dimension-user_login | ExaDev |
| octolytics-dimension-repository_id | 1388126104 |
| octolytics-dimension-repository_nwo | ExaDev/github-runner |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 1388126104 |
| octolytics-dimension-repository_network_root_nwo | ExaDev/github-runner |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 46f1cd2ae3cd8ae1a81019b5d619742e76f13bbd |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width