Title: Measure each host's fitness to hold an etcd vote and report a role controller's decisions in dry-run · Issue #182 · ExaDev/github-runner · GitHub
Open Graph Title: Measure each host's fitness to hold an etcd vote and report a role controller's decisions in dry-run · Issue #182 · ExaDev/github-runner
X Title: Measure each host's fitness to hold an etcd vote and report a role controller's decisions in dry-run · Issue #182 · ExaDev/github-runner
Description: The role controller of the dynamic regrading epic decides which eligible hosts hold the etcd votes. This sub-issue builds what it decides from, and runs the decision in dry-run only: it reads, computes and reports, and never changes a ro...
Open Graph Description: The role controller of the dynamic regrading epic decides which eligible hosts hold the etcd votes. This sub-issue builds what it decides from, and runs the decision in dry-run only: it reads, comp...
X Description: The role controller of the dynamic regrading epic decides which eligible hosts hold the etcd votes. This sub-issue builds what it decides from, and runs the decision in dry-run only: it reads, comp...
Opengraph URL: https://github.com/ExaDev/github-runner/issues/182
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Measure each host's fitness to hold an etcd vote and report a role controller's decisions in dry-run","articleBody":"The role controller of the dynamic regrading epic decides which eligible hosts hold the etcd votes. This sub-issue builds what it decides from, and runs the decision in dry-run only: it reads, computes and reports, and never changes a role. It needs no other sub-issue and changes no membership, so it can run on a live cluster as soon as it ships.\n\n**Probe.** A `fleet-role-probe` DaemonSet on every node except etcd-only servers (which have no kubelet; their side is read from their peers), with host networking so it measures the path the node's own etcd and kubelet use (on a Docker node that is the k3s container's network, where the node's Tailscale client lives). Every sample period it records, per node:\n\n- the TCP round trip to every voter's mesh address on the etcd peer port, so loss shows as tail latency rather than being lost;\n- on a voter, etcd's own `etcd_network_peer_round_trip_time_seconds`, `etcd_disk_wal_fsync_duration_seconds`, `etcd_disk_backend_commit_duration_seconds` and `etcd_server_leader_changes_seen_total` from the metrics listener k3s keeps on `127.0.0.1:2381`;\n- on a non-voter, an fdatasync probe in the k3s data directory with the write size etcd's own fio check uses, once per window, never on a voter (whose etcd is measured directly);\n- memory pressure and MemAvailable, read with the regrade core's own functions;\n- the link the default route leaves by: wired or Wi-Fi from `/sys/class/net/\u003ciface>`, `unknown` where the node cannot see the host's interfaces (a Docker VM).\n\nResults go to a ConfigMap per node, in a status shape the controller and the heartbeat read.\n\n**Controller.** A `fleet-role` Deployment in the platform namespace, built and released like `node-recovery`, with `FLEET_ROLE_DRY_RUN=true` by default (the autoscaler's convention) and a `fleet-role-status` ConfigMap the heartbeat republishes beside `autoscaler-status`. Each pass it computes, for every eligible host, fit or unfit and why, and the swap it would make, logging the decision and the reason. Thresholds, all derived:\n\n- round trip: unfit when the p99 to any voter exceeds the election timeout divided by ten (etcd's rule that the election timeout is at least ten round trips); fit for promotion only at or below the election timeout divided by twenty (the 0.5 end of etcd's heartbeat band); the election timeout is the cluster's configured value, or k3s's 1000 ms when unset;\n- disk: WAL fsync p99 under 10 ms and backend commit p99 under 25 ms (etcd's FAQ);\n- memory: never over the regrade limit in the window, and MemAvailable at least etcd's storage quota (2 GiB unless set);\n- stability: no NotReady spell and no k3s restart in the window;\n- window: 100 samples per peer pair, the fewest that give a p99 a sample of its own, at `github_runner_cluster_role_sample_seconds` (15 by default, node-recovery's poll);\n- rank among the fit: wired before Wi-Fi before unknown, then p99 round trip, then fsync.\n\nThe decision rules are the ones the epic states: a swap only when an unpinned voter has been unfit for a whole window and a fit eligible candidate ranks above it, never while any voter is below `full`, not Ready, or etcd reports a learner, an unstarted member or no leader, a target voter count that is odd, at least 3 and at most `github_runner_cluster_max_servers` minus one, and a cooldown of k3s's learner stall time plus one window after any membership change.\n\n**Inventory.** `github_runner_cluster_voter_eligible` and `github_runner_cluster_voter_pinned` per host and `github_runner_cluster_voters` for the group, published as Node labels by the role; the role refuses an even target, a target under 3 or without a place of headroom below `max_servers`, and fewer eligible hosts than the target.\n\nTests: `tests/unit` for every threshold and decision rule against recorded probe output, including the hysteresis band, the window boundary, ties and each refusal. A mesh scenario runs the probe and the controller in dry-run on the three-server cluster, makes one voter slow with `tc netem` on its mesh interface, and checks the controller reports it unfit and names the swap, then pauses a voter and checks it reports that it would not act. Every probe is then run against each real host the fleet marks eligible, in dry-run, and its readings compared with a reading taken by hand on that host (`ip route`, the sysfs entry, `ping`, the etcd metrics), because a stand-in that lacks the real structure hides bugs: a `find -L` over sysfs once passed stand-ins and failed on a real host.\n","author":{"url":"https://github.com/Mearman","@type":"Person","name":"Mearman"},"datePublished":"2026-11-03T07:02:04Z","url":"https://github.com/ExaDev/github-runner/issues/182"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:e8b46d4a-f53e-e6e1-32f0-521c1970189d |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | ED12:A5865:5FD576:830779:6B0076B4 |
| html-safe-nonce | b98ff015eb90e7983587f1019caf8dcd56a611e9cb46876b59e405c97a997455 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJFRDEyOkE1ODY1OjVGRDU3Njo4MzA3Nzk6NkIwMDc2QjQiLCJ2aXNpdG9yX2lkIjoiNzMyNjQ4NTkxMTE2NjgwOTc4MCIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | ff43afbc9a5afc834465a92e872ffadc5b7ce98f8c8e3388b6e8f78800b0d183 |
| hovercard-subject-tag | issue:6177867756 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/ExaDev/github-runner/182/issue_layout |
| twitter:image | https://opengraph.githubassets.com/c361539861a9a8745381db45c8211ac7d49da60bbd41b7bab6bbcc5a8c28d3a3/ExaDev/github-runner/issues/182 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/c361539861a9a8745381db45c8211ac7d49da60bbd41b7bab6bbcc5a8c28d3a3/ExaDev/github-runner/issues/182 |
| og:image:alt | The role controller of the dynamic regrading epic decides which eligible hosts hold the etcd votes. This sub-issue builds what it decides from, and runs the decision in dry-run only: it reads, comp... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | Mearman |
| hostname | github.com |
| expected-hostname | github.com |
| None | 75c2b65dd205f26961d0c29afed1a8d0bb7cca9f05780571dfd1ef93eb539d16 |
| turbo-cache-control | no-preview |
| go-import | github.com/ExaDev/github-runner git https://github.com/ExaDev/github-runner.git |
| octolytics-dimension-user_id | 140118018 |
| octolytics-dimension-user_login | ExaDev |
| octolytics-dimension-repository_id | 1388126104 |
| octolytics-dimension-repository_nwo | ExaDev/github-runner |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 1388126104 |
| octolytics-dimension-repository_network_root_nwo | ExaDev/github-runner |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 84e563eb4b55f8fa102f325f117533cee9e3df4a |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width