Title: Promote a fit standby to an etcd voter and demote an unfit voter, one swap at a time · Issue #183 · ExaDev/github-runner · GitHub
Open Graph Title: Promote a fit standby to an etcd voter and demote an unfit voter, one swap at a time · Issue #183 · ExaDev/github-runner
X Title: Promote a fit standby to an etcd voter and demote an unfit voter, one swap at a time · Issue #183 · ExaDev/github-runner
Description: With native servers, the systemd regrade actuator and the dry-run controller in place, the controller gets the power to act: promote an eligible agent to a voter and demote an unfit voter to an agent, as one swap, when FLEET_ROLE_DRY_RUN...
Open Graph Description: With native servers, the systemd regrade actuator and the dry-run controller in place, the controller gets the power to act: promote an eligible agent to a voter and demote an unfit voter to an age...
X Description: With native servers, the systemd regrade actuator and the dry-run controller in place, the controller gets the power to act: promote an eligible agent to a voter and demote an unfit voter to an age...
Opengraph URL: https://github.com/ExaDev/github-runner/issues/183
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Promote a fit standby to an etcd voter and demote an unfit voter, one swap at a time","articleBody":"With native servers, the systemd regrade actuator and the dry-run controller in place, the controller gets the power to act: promote an eligible agent to a voter and demote an unfit voter to an agent, as one swap, when `FLEET_ROLE_DRY_RUN=false`. Part of the dynamic regrading epic, whose body has the reasoning and sources for each step.\n\n**A swap, step by step**, all under the `fleet-regrade` Lease, which the controller renews for as long as the swap runs:\n\n1. Check again that every voter is at `full`, Ready, and that etcd has a leader and no learner or unstarted member. Take an on-demand snapshot on a voter (`k3s etcd-snapshot save`) and record its name.\n2. Set `github-runner.exadev.com/desired-role=server` on the candidate's Node. The host's actuator cordons it, waits for its runner pods to finish (a planned change never cuts a job; past the operation timeout the swap is abandoned and the node uncordoned), drains it, writes the role file and a join URL naming a current voter, stops the agent unit and starts the server unit. k3s joins it as a learner and the etcd leader promotes it once it has caught up. The step succeeds when it is a voter and its Node is Ready on a heartbeat after the start.\n3. Check health again, now with four voters.\n4. Set `desired-role=agent` on the outgoing voter. Its actuator steps it to etcd-only through regrading (drained, no API server, so k3s's leader-elected controllers run elsewhere), then the controller sets `etcd.k3s.cattle.io/remove=true` on its Node. k3s's member controller on another server removes the member; the outgoing server's etcd logs that it was removed and writes its tombstone. The actuator then stops the server unit, moves `server/db` aside (keeping only the newest such copy), clears the regrade level, removes the `control-plane`, `etcd` and `master` role labels and k3s's etcd annotations from the Node, writes the role file and starts the agent unit, and uncordons the node once it is Ready.\n5. Release the Lease and start the cooldown.\n\n**Failure at each step.** A candidate that does not become a voter (k3s removes a learner stalled for five minutes, or the step times out) is rolled back: the controller removes a learner that is still listed, which never touches quorum, and the actuator stops the server unit, moves its `server/db` aside and starts the agent again, so membership is as it was. A failure between steps 3 and 4 leaves four healthy voters, with the same tolerance as three; the controller's next action, before any other, is to finish the removal, and it alerts through the heartbeat while the count is even. A removal the member controller rejects is retried only while the remaining voters are all healthy. If the outgoing server's agent does not come back, its member is already gone and quorum is untouched; node-recovery handles the node as any agent.\n\n**Docker hosts.** Today's voters run under Compose, so the Docker side needs the same role switch: demotion stops the server service and starts the agent profile with the server's node password copied into the agent's node volume (the two services use different volumes, and a new password is rejected), and promotion is the reverse. The Compose project records the role on the host so a role run does not undo it, and the entrypoint never passes `--cluster-init` once the host has joined a cluster, so a former bootstrap host that is promoted again joins the existing cluster.\n\n**Interactions.** The regrade core's `VOTER_LABEL` selection stays right because a demoted node loses the etcd label. node-recovery leaves alone the node the Lease holder names, so a node restarting into its new role is not tainted out of service mid-swap, and its prune pass sees a demoted node as an agent. The heartbeat and `scripts/fleet-alert.sh` report a swap in progress rather than an outage.\n\n**What stays manual.** Removing a voter that is down and not coming back, restoring after quorum loss, and changing the target count. Promotion of a host the role has not pre-provisioned is refused.\n\nTests: the mesh harness proves every step with real k3s and real etcd: a native standby promoted and a Docker server demoted in one swap, with writes succeeding throughout and three voters at the end; the same swap with the outgoing voter paused after step 3 (the removal still completes); a candidate whose peer port is blocked (k3s removes the learner after five minutes and the rollback leaves membership unchanged); a swap refused while a voter is paused before step 1; the demoted node's labels and annotations gone, its `server/db` moved aside, and a later promotion of the former bootstrap host joining the existing cluster (one cluster ID before and after). Unit tests cover the controller's state machine for every failure branch. On the live fleet the first swap is run by hand with the controller's command while the owner watches, before the flag is turned off for good.\n\n\n\n## Implementation plan (decided)\n\nRead from `origin/main` at 1b781aa (v2.31.0). Paths are relative to `roles/github_runner_cluster/` unless they start with `scripts/`, `tests/`, `plugins/` or `roles/`.\n\n### (a) What already exists on main, and what does not\n\n**Exists:**\n\n- **Native role switching through Ansible.**\n - The role file and unit guard: `files/native/fleet-role-is.sh`, used by `ExecCondition` in `templates/native/k3s.service.j2:33`.\n - An eligible host keeps the role its file records: `cluster_native_role`, `plugins/filter/cluster_native.py:94-114`.\n - Eligible hosts are staged as standbys: `tasks/native/role.yml:46`.\n - Promotion in place: the `promote` turn in `tasks/native/server_turn.yml:65-162`.\n - Demotion: `tasks/native/demote.yml`. It refuses unless three other Ready voters remain (line 30), writes the role file before asking for removal (54), annotates `etcd.k3s.cattle.io/remove` (61), waits for `removed-node-name` (72), moves `server/db` aside keeping one copy (108), and strips the role labels and etcd annotations (120).\n - Draining with a runner wait, uncordoning if it gives up: `tasks/native/drain.yml`.\n - A former member never bootstraps: `templates/native/join.yaml.j2` and `tasks/native/role.yml:61-92`.\n- **Regrading:**\n - The levels and records: `files/regrade/fleet-regrade-k3s.sh`, `fleet-regrade-systemd.sh` and `fleet-regrade-docker.sh`.\n - The Lease: `acquire_lease`, `release_lease` and `lease_state` in `files/regrade/fleet-regrade-core.sh:184-250`.\n - `select_api_server` over `KUBE_SERVERS`, which already lists eligible hosts (`templates/regrade.env.j2:28-32`).\n- **The dry-run controller:** `scripts/fleet-role.sh`, `scripts/fleet-role.jq` and `scripts/fleet-role-probe.sh`. The settings checks are `tasks/voters.yml`. Eligible hosts' TLS SANs come from `plugins/filter/cluster_topology.py:123`. The heartbeat republishes `fleet-role-status` (`scripts/heartbeat.sh:132`). `scripts/fleet-alert.sh:89` already ignores a server carrying a regrade level.\n\n**Does not exist:**\n\n- `desired-role` appears nowhere in `scripts`, `roles` or `tests`.\n- No host-side actuator, and no process on an agent host that could act on the API at all.\n- The controller cannot write. `scripts/fleet-role.sh:43` refuses `FLEET_ROLE_DRY_RUN` other than true, and its ClusterRole is `get, list` on nodes (`roles/github_runner_arc/templates/fleet-role.yaml.j2`, `node_rules`).\n- Nothing renews a Lease: the core acquires once, sized at three operation timeouts (`defaults/main.yml:50`).\n- Nothing triggers a snapshot.\n- `scripts/node-recovery.sh` never reads a Lease.\n- The heartbeat has no swap fields.\n- Docker hosts have no role switch or recorded role. `tasks/main.yml:131-182` picks the profile and `.env` template from the inventory role.\n- **The Docker entrypoint passes `--cluster-init` whenever `K3S_JOIN_SERVER_URL` is empty** (`files/k3s/node-entrypoint.sh:177-182`). A demoted bootstrap host with its `server/db` moved aside would therefore start a second cluster on its next promotion.\n- No test removes a member with real k3s. The `native_server` mesh scenario only promotes, and `demote.yml` is tested only against stand-ins (`tests/unit/test_native_server.py:161`).\n\n### Where issue 183 is wrong or inconsistent with the code (each decided)\n\n1. **Who cordons and drains.** The issue gives this to the host actuator. An agent host has no API credential that can evict pods, and its own k3s is down during the switch.\n - Decided: the controller does everything on the API: cordon, the runner wait, drain, the taint, the level label, member removal, label clean-up and uncordon.\n - The actuator does only host-local work. Its credential can read Nodes and patch one ConfigMap, nothing more.\n\n2. **The snapshot.** \"`k3s etcd-snapshot save` on a voter\" cannot run from the controller pod, which has neither the k3s binary nor the server token.\n - Decided: the controller asks one remaining voter's actuator through a Node annotation.\n - The controller then checks that an `ETCDSnapshotFile` with that name reports `readyToUse`. k3s v1.34 publishes these objects.\n\n3. **Renewing the Lease.** The epic says the holder renews it; the core never does. Decided: add `renew_lease` to `fleet-regrade-core.sh`, which the controller calls on every pass. A regrade step keeps its sized duration.\n\n4. **node-recovery and the Lease holder.** \"node-recovery leaves alone the node the Lease holder names\" has nothing to build on, and a Lease held by the controller names no node.\n - Decided: a Lease annotation lists the swap's nodes.\n - node-recovery also stops tainting a server held at etcd-only by regrading. Today it applies `out-of-service` to one after `NODE_RECOVERY_AFTER_SECONDS`, and a swap's outgoing voter passes through etcd-only.\n\n5. **Copying only the node password on a Docker switch is not enough.** The agent service keeps its own Tailscale state (`files/docker-compose.yml:62` against `:106`). It would come up as a new device under the same hostname, the tailnet would give it a suffixed name, and every join URL and SAN by name would break.\n - Decided: copy the node password and the Tailscale state between the two services' volumes, with both containers stopped.\n\n6. **\"VOTER_LABEL stays right because a demoted node loses the etcd label.\"** k3s does not remove that label (`demote.yml` says so).\n - A learner also carries `node-role.kubernetes.io/etcd` before it votes, so anything that reads a voter from that label (`scripts/fleet-role.jq:66`, the core's `VOTER_LABEL`, node-recovery) counts a learner.\n - Decided: the controller removes the label in step 4, and judges a voter by `EtcdIsVoter=True` as well as the label.\n\n7. **Step 4 writes the role file after the removal; that is unsafe.**\n - Once removed, k3s writes `server/db/etcd/tombstone`. On its next start, `Register` backs the datastore up and rejoins as a learner (`pkg/etcd/etcd.go:682-689` at v1.34.1+k3s1). A unit restarted by `Restart=always`, or a container restarted by `restart: unless-stopped`, would therefore come back as a member.\n - Decided: switch the role file to agent after the etcd-only restart and before the removal, as `demote.yml:54` already does.\n - Also refuse any server start while a tombstone exists. This is the only guard that holds when the outgoing voter was down during the removal.\n\n8. **The join URL alone does not change what a native server waits for.** The unit's `GITHUB_RUNNER_JOIN_HOST` is fixed when Ansible writes it (`templates/native/k3s.service.j2:16`), and `files/native/tailscale-up-wait.sh` waits for that host without a bound. A join drop-in alone would still wait on the inventory's bootstrap host, which may by then be an agent or down. Decided: the actuator writes a persistent unit drop-in naming the same host as the join drop-in.\n\n9. **The wait for runner pods.** The issue bounds it by the operation timeout; the role bounds it by `github_runner_cluster_native_runner_wait_timeout` (`defaults/main.yml:303`, `tasks/native/drain.yml:29`). Decided: the controller uses the runner wait timeout, so both paths agree.\n\n10. **Issue 184's claim that regrading is a prerequisite on every eligible server.** The swap uses the level mechanism, which works whether or not regrading is on: the Docker entrypoint's `set_regrade_flags` (`files/k3s/node-entrypoint.sh:57-66`) and the native `ExecStartPre` (pinned by `tests/unit/test_native_regrade.py:73`). It does not use the regrade watcher. Decided: the regrade service is not a prerequisite, and #140 stays separate.\n\n### The PRs, in dependency order\n\nA and B can land in any order. C needs A. D1 needs C. D2 needs D1. E1 is independent. E2 needs B, D2 and E1. F needs E2.\n\n#### PR A: `fix(cluster): never start a second cluster from a former member, and keep a retired server from starting`\n\n**Files:** `files/k3s/node-entrypoint.sh`, `files/docker-compose.yml`, `files/native/fleet-role-is.sh`, `tasks/main.yml`, the role `README.md`, `tests/unit/test_node_entrypoint.py` (new), `tests/unit/test_native_server.py`, `tests/mesh/run.sh`.\n\n**Interfaces:**\n\n- **A role directory.** `{{ github_runner_cluster_dir }}/fleet-role.d/` is bind-mounted read-only at `/etc/github-runner` into both k3s services. It is a directory, not two files, because an atomic `mv` gives a file a new inode, which a single-file bind mount does not see.\n - It holds `role` (`server` or `agent`) and `join-url`.\n - The role creates the directory empty.\n - An absent `role` file behaves exactly as today.\n- **The entrypoint's guards.** When `role` names the other role, or (on a server) `/var/lib/rancher/k3s/server/db/etcd/tombstone` exists, the entrypoint logs why and runs `exec sleep infinity`. It never exits, because `unless-stopped` would restart a container that exits.\n- **The join URL** is the `join-url` file when it is not empty, otherwise `K3S_JOIN_SERVER_URL`.\n- **`--cluster-init`:**\n - With no join URL and `server/db` present: passed as today (k3s ignores it).\n - With no join URL and `server/db.demoted` present: k3s is not started, and the container exits 1 with a message.\n - With neither: passed (a first bootstrap).\n- **Native:** `fleet-role-is.sh server` also exits 1 (start skipped) while a tombstone exists.\n\n**Reuses:** the existing `mesh_server_args` structure, and `join_host_of` and `wait_for_mesh_host` with the chosen URL.\n\n**Tests:**\n\n- Unit:\n - the join URL's precedence;\n - `--cluster-init` in each of the three datastore states;\n - the idle guard for the wrong role;\n - the idle guard on a tombstone;\n - no change with no role file;\n - the native condition on a tombstone.\n- Mesh scenario `rejoin`, with real k3s and etcd:\n - stop the bootstrap Docker server;\n - move `server/db` to `db.demoted` in its volume;\n - start it with no `join-url`: it refuses;\n - write a `join-url` naming node 2: it joins as a learner and is promoted;\n - the etcd cluster ID (`etcdctl endpoint status -w json`) is the same before and after;\n - a role file naming `agent` leaves the server container idle with no k3s process.\n\n**Size:** 8 files, 10 tests (9 unit, 1 mesh scenario).\n\n#### PR B: `feat(arc): leave alone the nodes of a role swap and servers stepped down by regrading in node-recovery`\n\n**Files:** `scripts/node-recovery.sh`, `roles/github_runner_arc/templates/node-recovery.yaml.j2`, `roles/github_runner_arc/defaults/main.yml`, `tests/unit/test_node_recovery.py`, `tests/unit/test_node_recovery_role.py`, the README.\n\n**Interfaces:**\n\n- node-recovery reads `kube-system/fleet-regrade` through a new Role with `get` on that name only.\n- While the Lease is unexpired (the core's `lease_state` rule, reimplemented in node-recovery's jq), node-recovery leaves alone:\n - the nodes listed in its annotation `github-runner.exadev.com/swap-nodes` (comma-separated);\n - its `holderIdentity` when that is a node name.\n- It never taints a node carrying `github-runner.exadev.com/regrade-level`.\n\n**Tests:**\n\n- exempt while held;\n- acted on once expired;\n- a regrade holder exempted;\n- the level label exempted;\n- the taint cleared as before;\n- the prune pass unchanged for a node with no server labels;\n- the RBAC renders scoped to the Lease name.\n\n**Size:** 6 files, 7 tests.\n\n#### PR C: `feat(cluster): record a Docker host's role on the host and keep it across runs`\n\n**Files:**\n\n- `templates/env.j2`, which becomes one template holding both services' variables; `templates/env-agent.j2` is removed;\n- `plugins/filter/cluster_role.py`, new: `cluster_host_role`, moved from `cluster_native_role`, which is removed with every caller updated;\n- `tasks/native/role.yml`, `tasks/docker_role.yml` (new), `tasks/main.yml`, `tasks/start_servers.yml`, `tasks/regrade_service.yml`;\n- `tasks/role_lease_guard.yml`, new;\n- `tests/unit/test_docker_role.py` (new), `tests/unit/test_native_node.py`, the README.\n\n**Interfaces:**\n\n- An eligible Docker host's role lives in `fleet-role.d/role`. It is written from the inventory only when absent (`force: false`), as the native role file is.\n- `main.yml` chooses the profile, the container to start and the regrade service from the recorded role.\n- `start_servers.yml` gives a turn to every play host whose recorded role is `server`, not to `topology.servers`.\n- A host that is not eligible, and whose running container's role differs from its inventory role, is refused with a message. Today the second profile would start beside the first.\n- Every run refuses to change a unit or container while the `fleet-regrade` Lease is held by `fleet-role`.\n\n**Tests:**\n\n- the recorded role kept over the inventory;\n- written only when absent;\n- the merged `.env` renders the same configuration for the `k3s` and `k3s-agent` services as before (`docker compose config` hashes are equal, so no container is recreated);\n- the turn list follows the recorded role;\n- the refusal on a role mismatch;\n- the refusal while the Lease is held;\n- the regrade service only on a server;\n- `--check` reports the refusals.\n\n**Size:** 12 files, 12 tests.\n\n#### PR D1: `feat(cluster): switch a native host's role from a host-side actuator driven by its Node`\n\n**Files:**\n\n- New: `files/role/fleet-role-actuator.sh` (the command and platform-neutral logic), `files/role/fleet-role-actuator-systemd.sh`, `templates/native/fleet-role-actuator.service.j2`, `templates/role-actuator.env.j2`, `tasks/role_actuator.yml`, `tests/role_actuator/` (fakes reusing `tests/regrade/fake-*.sh`), `tests/unit/test_role_actuator.py`.\n- Changed: `templates/native/fleet-regrade.service.j2` (gains `ExecCondition=... fleet-role-is server`), `tasks/native/demote.yml` and `tasks/native/server_turn.yml` (refactored), `defaults/main.yml`, `meta/argument_specs.yml`, the README.\n\n**Interfaces fixed here:**\n\n- **Node label** `github-runner.exadev.com/desired-role` (`server` or `agent`). Only the controller sets it. It is persistent, and never a kubelet registration label.\n- **Node annotations,** all under `github-runner.exadev.com/`:\n - `role-swap`: the swap id, `\u003cUTC %Y%m%dT%H%M%SZ>-\u003cout>-\u003cin>`;\n - `role-step`: `promote`, `step-down` or `rollback`;\n - `role-join-node`: the node name of the voter to join;\n - `snapshot-request`: a swap id.\n- **The Lease,** `kube-system/fleet-regrade`:\n - `holderIdentity: fleet-role`;\n - annotations `github-runner.exadev.com/swap-id` and `github-runner.exadev.com/swap-nodes`.\n- **Actuator state.** The ConfigMap `fleet-role-actuators` in the platform namespace holds one key per node: `{\"swap\",\"step\",\"phase\":\"running|done|failed|refused\",\"role\",\"started_at\",\"at\",\"message\",\"snapshot\"}`.\n- **Credential.**\n - ServiceAccount `fleet-role-actuator`: a ClusterRole with `get, list, watch` on nodes, and a Role with `get, patch` on that ConfigMap by name.\n - A token Secret, created through `tasks/native/api.yml`'s API host.\n - A kubeconfig, mode 0600, at `/etc/rancher/k3s/fleet-role-actuator.kubeconfig`.\n - `KUBE_SERVERS` comes from the same expression `templates/regrade.env.j2` uses.\n - A file mapping each server and eligible host to its join URL is written from the topology's `peer_server_urls` and `address`.\n- **Commands and exit statuses.** The commands are `fleet-role-actuator watch|status|promote ID|step-down ID|finish-demote|rollback ID|snapshot ID`.\n - 0: done.\n - 1: failed. The host is left with one role's unit at most, and a role file naming it.\n - 2: refused. That means one of:\n - the Lease is not held by `fleet-role` with this swap id;\n - the host is not provisioned (native: no `server.yaml`);\n - the role named is not a role.\n - 3: not yet (`finish-demote` before the member is gone).\n- **Authority to act.** Every step that could add a member needs the Lease with a matching `swap-id`. `finish-demote` needs only role file `agent` together with a tombstone, `removed-node-name` naming its member, or no etcd label. Completing the demotion of a removed member can never cost quorum, so a host that was down during its swap finishes on return.\n- **Promote:**\n - role file `server`;\n - `server.yaml.d/20-fleet-role-join.yaml` with `server:`;\n - a persistent unit drop-in setting `GITHUB_RUNNER_JOIN_HOST`;\n - `systemctl disable --now k3s-agent`, then `enable --now k3s`;\n - report `started_at`.\n- **Step-down:**\n - `write_level \"$LEVEL_FILE\" etcd-only`, `restart_into etcd-only` and `stop_pod_shims` (reused from `fleet-regrade-k3s.sh` and `fleet-regrade-systemd.sh`);\n - then role file `agent`;\n - and `systemctl stop fleet-regrade`.\n- **Finish-demote and rollback:**\n - `disable --now k3s`;\n - move `server/db` to `db.demoted`, keeping one copy;\n - remove the level file and the join drop-ins;\n - `enable --now k3s-agent`.\n- **Snapshot:** `k3s etcd-snapshot save --name fleet-role-\u003cid>`.\n- **Refactor.** `demote.yml` and `server_turn.yml` call `promote-local` and `demote-local` for their host-local part. Ansible and the actuator then share one implementation.\n\n**Reuses:** `kubectl_`, `select_api_server`, `lease_state`, `micro_time` and `now_epoch` (core); `write_level`, `restart_into` and `etcd_serving` (k3s); `stop_pod_shims`, `on_server` and `restart_server` (systemd); `fleet-role-is.sh`.\n\n**Tests:**\n\n- Unit:\n - each command's success, failure, refusal and status;\n - refusal without the Lease or with a stale swap id;\n - `finish-demote` without the Lease;\n - a tombstone treated as removed;\n - one `db.demoted` copy kept;\n - the drop-ins written and removed;\n - the regrade unit's condition;\n - the RBAC renders scoped;\n - the token stays out of the output;\n - `demote.yml` and `server_turn.yml` unchanged in behaviour against the existing harness.\n- Mesh scenario `role_actuator`, native part, with the test acting as the controller (setting the Lease and annotations by hand) and real k3s:\n - the native standby is promoted: learner, then voter, writes succeed;\n - stepped down: etcd serves and no shim is left;\n - removed through the annotation;\n - finished to an agent: the same password hash and Tailscale IP, `db.demoted` present, both units never active together.\n\n**Size:** 16 files, 18 tests (17 unit, 1 mesh scenario).\n\n#### PR D2: `feat(cluster): switch a Docker host's role from the same actuator`\n\n**Files:** `files/role/fleet-role-actuator-docker.sh` (new), `files/role/Dockerfile` (new: Alpine with `bash jq docker-cli docker-cli-compose kubectl`, built locally as the regrade image is, scripts mounted), `files/node-identity.sh` (new; the copy pipe from `tasks/node_password.yml`, generalised), `files/docker-compose.yml`, `files/regrade/fleet-regrade.sh`, `tasks/role_actuator.yml`, `tests/unit/test_role_actuator.py`, `tests/mesh/run.sh`, the README.\n\n**Interfaces:**\n\n- **A Compose service `role-actuator`** with profile `role-actuator`, added for every server and eligible host while the controller is on. It mounts the Docker socket, the project directory at its own absolute path, and the kubeconfig.\n- **Promote:**\n - write `fleet-role.d/role=server` and `join-url`;\n - `docker compose --profile agent stop k3s-agent`;\n - copy the identity: `${project}_k3s-agent-node` to `_k3s-node`, and `_k3s-agent-tailscale-data` to `_k3s-tailscale-data`, through a throwaway container from the node image, logging nothing;\n - `--profile server up -d k3s`.\n- **Step-down** reuses `fleet-regrade-docker.sh` (`write_level`, `restart_into`) and then writes the role file.\n- **Finish-demote:**\n - stop the server;\n - in its volume, move `server/db` aside and remove the level file;\n - copy the identity back;\n - `up -d k3s-agent`;\n - stop the `regrade` service.\n- The regrade wrapper idles on a non-server host.\n\n**Reuses:** `on_server`, `restart_server` and `etcd_serving` (docker actuator), and the seeding pipe from `tasks/node_password.yml`.\n\n**Tests:**\n\n- Unit:\n - each command against the fake `docker` and `kubectl`;\n - the identity copy in both directions, never with either container running;\n - the volume names derived from the project;\n - no secret in the output;\n - the regrade wrapper idling.\n- Mesh scenario `role_actuator`, Docker part:\n - a Docker server is demoted by hand-set annotations;\n - the same Tailscale IP and password hash;\n - one Node;\n - `db.demoted` in `k3s-data`;\n - the agent container Ready;\n - then promoted again, joining the same cluster ID.\n\n**Size:** 9 files, 12 tests (11 unit, 1 mesh scenario).\n\n#### PR E1: `feat(role): judge voters by etcd's own vote and let a holder renew the regrade Lease`\n\n**Files:** `files/regrade/fleet-regrade-core.sh`, `scripts/fleet-role.jq`, `tests/regrade/regrade-lease.sh`, `tests/unit/test_fleet_role.py`, `tests/unit/fleet_role_support.py`.\n\n**Interfaces:**\n\n- `renew_lease HOLDER`: replaces the Lease with a new `renewTime`, keeping `acquireTime` and annotations; fails if the holder changed.\n- `fleet-role.jq` gains `$config.expected_voters`: the target, or the target plus one while a swap holds a new voter.\n- A voter needs the etcd label and `EtcdIsVoter=True`. A node with the label but not the condition is a block, \"member but not a voter\".\n\n**Tests:**\n\n- renew keeps the holder;\n- renew refused for another holder;\n- annotations survive a renewal;\n- a learner by label is not a voter and blocks;\n- the expected count of target plus one;\n- the existing gate tests unchanged.\n\n**Size:** 5 files, 8 tests.\n\n#### PR E2: `feat(role): carry out a voter swap, by decision or by command`\n\n**Files:** `scripts/fleet-role.sh` (subcommands, resumable passes), `scripts/fleet-role-swap.jq` (new; pure next-action function), `roles/github_runner_arc/templates/fleet-role.yaml.j2`, `roles/github_runner_arc/defaults/main.yml`, `roles/github_runner_arc/meta/argument_specs.yml`, `roles/github_runner_arc/README.md`, the repository README (Role controller section), `fleet-role/Dockerfile`, `tests/unit/test_fleet_role_swap.py` (new), `tests/mesh/run.sh`.\n\n**Command line and modes:**\n\n- `fleet-role.sh` with no arguments runs one pass.\n- `fleet-role.sh swap OUT IN` records a manual request in `fleet-role-status` (`request: {out, in, at}`). Exit statuses:\n - 0: accepted, every gate holding now;\n - 1: refused, with each reason;\n - 2: usage error, or an unknown or ineligible node.\n- `fleet-role.sh status` prints the swap and its step.\n- Run as `kubectl -n github-runner-platform exec deploy/fleet-role -- /app/scripts/fleet-role.sh swap \u003cout> \u003cin>`.\n- `FLEET_ROLE_DRY_RUN=true` reports automatic swaps only. A manual request is carried out.\n- `false` carries out automatic decisions as well.\n- A swap already in progress (a `role-swap` annotation on a Node) is always driven to completion or rolled back, whatever the setting, so an even count is never left behind.\n- A manual swap honours every gate but the fitness verdict and the cooldown. It refuses a pinned, ineligible or unprovisioned node.\n\n**New settings:** `FLEET_ROLE_OPERATION_TIMEOUT_SECONDS`, `FLEET_ROLE_RUNNER_WAIT_SECONDS`, `FLEET_ROLE_LEASE_SECONDS` (the regrade Lease duration) and `FLEET_ROLE_ACTUATOR_CONFIGMAP`, each from the cluster role's values.\n\n**RBAC (the controller only; the probe keeps read-only):**\n\n- nodes: `patch`;\n- pods: `list` in all namespaces;\n- `pods/eviction`: `create`;\n- `apps/daemonsets`: `get`;\n- `coordination.k8s.io/leases`: `create`, plus `get, update` on `fleet-regrade`;\n- `k3s.cattle.io/etcdsnapshotfiles`: `get, list`;\n- the actuator ConfigMap: `get`.\n\n**The state machine.** Each pass renews the Lease and takes at most one action, so a rescheduled controller pod resumes from the Node annotations and the status.\n\n1. The gates (`blocks` empty, the Lease free). Take the Lease. Request a snapshot from a remaining voter and wait for `readyToUse`.\n2. Cordon the candidate and wait for runner pods. Past the runner wait timeout, abandon: uncordon, release the Lease, no cooldown. Otherwise drain, set `desired-role=server` with `role-step=promote` and `role-join-node`. Wait for `EtcdIsVoter=True` and Ready on a heartbeat after `started_at` (the shell form of `tasks/native/server_ready.yml`), then uncordon. The deadline is k3s's learner stall time plus the operation timeout.\n3. The gates again, with `expected_voters` at the target plus one.\n4. Taint, cordon and drain the outgoing node; label it `regrade-level=etcd-only`; set `desired-role=agent` with `role-step=step-down`. When the actuator reports done, annotate `remove=true` and wait for `removed-node-name`.\n - If the actuator does not report and the node is not Ready, remove anyway while the other voters are healthy, as the epic allows.\n - If k3s clears the annotation (`ErrMemberNotEnoughStarted`), set it again only while the others are healthy.\n - Then wait for the actuator's `finish-demote`. Remove the `control-plane`, `etcd` and `master` labels, the `etcd.k3s.cattle.io/*` annotations, the level label and the taint. Wait for Ready, then uncordon. A host that does not come back is left to node-recovery, with `outgoing_pending` recorded.\n5. Clear the swap annotations, release the Lease, and record the change for the cooldown.\n\n**Rollback.** A candidate that is not a voter by the deadline is rolled back:\n\n- if its Node has etcd annotations, annotate `remove`;\n- `desired-role=agent` with `role-step=rollback`;\n- clear the server labels and uncordon;\n- done once every voter's probe reports `known_members` equal to the target.\n\n**Status.** `fleet-role-status` gains `swap` (`id, out, in, step, started_at, deadline, manual, snapshot`), `last_swap` (`outcome: completed|rolled-back|abandoned`, with reasons), `request` and the real `dry_run`.\n\n**Tests:**\n\n- Unit, one per branch:\n - each step's success;\n - abandon on the runner wait;\n - rollback on the deadline, with and without etcd annotations;\n - a rollback that waits on `known_members`;\n - removal rejected, then retried;\n - removal held while a remaining voter is unhealthy;\n - removal of an outgoing voter that is down;\n - resume after a restart at each step;\n - the Lease taken by regrade mid-swap (wait, never act);\n - an even count finished before any new decision;\n - dry-run carrying out a manual request only;\n - a manual request refused for a pinned or unprovisioned node, and with a paused voter;\n - no second swap in the cooldown;\n - Lease renewal on every pass;\n - snapshot wait and timeout;\n - RBAC renders;\n - the probe keeps read-only rules.\n- Mesh, real k3s and etcd:\n - `role_swap`: three Docker servers and a native standby. A manual swap brings the native host in and the Docker bootstrap host out, with writes succeeding throughout, three `EtcdIsVoter` voters at the end, the demoted node's labels and annotations gone, `db.demoted` in its volume, its identity kept and the Lease released. A second swap then brings the former bootstrap host back in, joining the existing cluster: one cluster ID before and after.\n - `role_swap_paused`: the outgoing k3s container is paused after step 3. The removal still completes, with three voters. On unpause, the tombstone guard keeps it out (the member count stays three through a learner stall time) and the actuator finishes it as an agent.\n - `role_swap_blocked_peer`: TCP 2380 is dropped in the candidate's namespace. k3s removes the learner after five minutes, and the rollback leaves the member list and the voters as they were.\n - `role_swap_refused`: a voter is paused before step 1. `swap` exits 1 naming it, and no label or Lease is set.\n - `role_controller` is extended: with dry-run off after the voter slowed by `tc netem` is reported unfit, the controller swaps it for the Docker standby. This proves a Docker promotion of a host that was never a server.\n\n**Size:** 11 files, 26 tests (22 unit, 4 new mesh scenarios and 1 extended).\n\n#### PR F: `feat(heartbeat): report a voter swap in progress and alert on an even voter count left behind`\n\n**Files:** `scripts/heartbeat.sh`, `scripts/fleet-alert.sh`, `tests/unit/test_heartbeat.py`, `tests/unit/test_fleet_alert.py`, the README.\n\n**Interfaces.** `fleet-health.json` gains:\n\n- `role_swap`: `fleet-role-status`'s `swap`, or null. It is published whatever the health, unlike the healthy-only status file.\n- `etcd_voters`: `{count, target, even_since}`, counted by `EtcdIsVoter`.\n- `not_ready_nodes[].role_swap`: a boolean.\n\n`fleet-alert.sh` changes:\n\n- It skips `server_not_ready` for a swap node until `role_swap.deadline`.\n- New kind `role_swap_overdue`: now is past the deadline plus `FLEET_ALERT_AFTER_SECONDS`.\n- New kind `voters_even`: `even_since` older than the window, with no swap inside its deadline.\n\n**Tests:**\n\n- the fields with and without a swap;\n- not-ready entries flagged;\n- `even_since` carried from the previous tick;\n- no alert for a swap node within the deadline;\n- an overdue swap alerts;\n- an even count alerts after the window;\n- not while a swap is within its deadline;\n- recovery clears both.\n\n**Size:** 5 files, 9 tests.\n\n### (b) The Docker role switch, in one place\n\n- **Profiles.** Compose keeps the `server` and `agent` profiles. One `.env` holds both services' variables, and the role recorded at `fleet-role.d/role` decides which profile Ansible and the actuator start (PR C).\n- **Never both.** The entrypoint idles when the role file names the other role, so the two services never run together.\n- **Identity.** The node password and the Tailscale state are copied between the two services' volumes while both containers are stopped (PR D2). The host keeps its node name, password Secret, Tailscale device and mesh IP, so etcd's peer URL is unchanged.\n- **`--cluster-init`.** It is never passed once the host holds `server/db.demoted` or a `join-url`. A former member with neither refuses to start (PR A).\n- **Ansible never undoes the controller.** An eligible host's role file is written only when absent. Every run refuses to change a container while the Lease is held by `fleet-role` (PR C).\n\n### (c) Interactions\n\n- **node-recovery:** the swap nodes listed on the Lease and any regrade-level node are exempt (PR B). Once its labels are gone, a demoted node is an agent to the prune pass. Its existing server refusal (`scripts/node-recovery.sh:68`) reads those labels, so it is right only after step 4's clean-up. That is why the clean-up comes before the agent's Ready wait ends the swap.\n- **Regrade core:** a Lease held by `fleet-role` refuses every regrade step (`acquire_lease`, `fleet-regrade-core.sh:231`). `release_lease` never frees it (`fleet-regrade-core.sh:248`). If the controller is gone long enough for the Lease to expire, regrade may take it, and the controller waits before acting again. The swap's record lives on the Nodes and in the status, not on the Lease, so nothing is lost.\n- **Heartbeat:** it publishes the swap and the voter count (PR F). An etcd-only outgoing voter is already excluded by its `regrade_level`.\n- **fleet-alert:** it holds off on swap nodes until the deadline, and alerts on an overdue swap or an even count left behind (PR F).\n- **The probe:** its DaemonSet follows the etcd label and the eligible label, so a demoted node keeps its pod while it stays eligible. A restart in the swap window marks the swapped hosts unfit for one window, and the cooldown already outlasts that window.\n\n### What stays manual, and a No\n\nThe manual tasks stay as issue 183 lists them: removing a voter that is down and not coming back, restoring after quorum loss, changing the target count, and promoting a host that has not been provisioned (refused in D1 and E2).\n\n**No: an Ansible-driven role switch for a Docker host.** For an eligible host, the controller's `swap` command covers it, and PR C refuses the mismatch that would otherwise start a second container silently. Revisit when an inventory needs to move a non-eligible Docker host between roles by hand.\n","author":{"url":"https://github.com/Mearman","@type":"Person","name":"Mearman"},"datePublished":"2026-11-03T07:02:05Z","url":"https://github.com/ExaDev/github-runner/issues/183"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:f976dd88-2680-94fe-fc13-4e9e4e95cb49 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | AC40:39E772:552F6B:740022:6B0072B9 |
| html-safe-nonce | 637ec0d3ec0ff3034550f1868f94dfd584425d7317a5238c145fb0c9e25550db |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBQzQwOjM5RTc3Mjo1NTJGNkI6NzQwMDIyOjZCMDA3MkI5IiwidmlzaXRvcl9pZCI6IjE1OTYxODM3ODY1NzQzNDQ4ODkiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | 654ad9c668ec36a1f0657f4092006e60c38b609ac9f4a8d5403168f341072cb6 |
| hovercard-subject-tag | issue:6177867928 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/ExaDev/github-runner/183/issue_layout |
| twitter:image | https://opengraph.githubassets.com/21fc9afdca83bc86258bd99e99d1c38b063b7480861b0519ec2cfb1f83e4dd49/ExaDev/github-runner/issues/183 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/21fc9afdca83bc86258bd99e99d1c38b063b7480861b0519ec2cfb1f83e4dd49/ExaDev/github-runner/issues/183 |
| og:image:alt | With native servers, the systemd regrade actuator and the dry-run controller in place, the controller gets the power to act: promote an eligible agent to a voter and demote an unfit voter to an age... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | Mearman |
| hostname | github.com |
| expected-hostname | github.com |
| None | 75c2b65dd205f26961d0c29afed1a8d0bb7cca9f05780571dfd1ef93eb539d16 |
| turbo-cache-control | no-preview |
| go-import | github.com/ExaDev/github-runner git https://github.com/ExaDev/github-runner.git |
| octolytics-dimension-user_id | 140118018 |
| octolytics-dimension-user_login | ExaDev |
| octolytics-dimension-repository_id | 1388126104 |
| octolytics-dimension-repository_nwo | ExaDev/github-runner |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 1388126104 |
| octolytics-dimension-repository_network_root_nwo | ExaDev/github-runner |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 84e563eb4b55f8fa102f325f117533cee9e3df4a |
| ui-target | canary-2 |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width