Title: Regrade a native server under memory pressure with a systemd actuator · Issue #181 · ExaDev/github-runner · GitHub
Open Graph Title: Regrade a native server under memory pressure with a systemd actuator · Issue #181 · ExaDev/github-runner
X Title: Regrade a native server under memory pressure with a systemd actuator · Issue #181 · ExaDev/github-runner
Description: Memory-pressure regrading (github_runner_cluster_regrade) has a Docker actuator only. A native server (the native server mode sub-issue) needs a systemd actuator implementing the core's three functions, levels, current_level and apply_le...
Open Graph Description: Memory-pressure regrading (github_runner_cluster_regrade) has a Docker actuator only. A native server (the native server mode sub-issue) needs a systemd actuator implementing the core's three funct...
X Description: Memory-pressure regrading (github_runner_cluster_regrade) has a Docker actuator only. A native server (the native server mode sub-issue) needs a systemd actuator implementing the core's three f...
Opengraph URL: https://github.com/ExaDev/github-runner/issues/181
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Regrade a native server under memory pressure with a systemd actuator","articleBody":"Memory-pressure regrading (`github_runner_cluster_regrade`) has a Docker actuator only. A native server (the native server mode sub-issue) needs a systemd actuator implementing the core's three functions, `levels`, `current_level` and `apply_level NAME`, with `files/regrade/fleet-regrade-core.sh` used as it is. Part of the dynamic regrading epic.\n\n**Levels** are the same three: `full`, `control-plane` and `etcd-only`. Between `full` and `control-plane` nothing restarts; the core's taint and cordon are the difference. etcd-only is reached by configuration and a restart of `k3s.service`: the actuator writes the level to `/var/lib/rancher/k3s/regrade-level` (persistent, so a stepped-down server stays down across a restart or a reboot, as the Docker actuator's does), and the unit's `ExecStartPre` renders `/etc/rancher/k3s/server.yaml.d/50-regrade.yaml` from it (`disable-apiserver`, `disable-controller-manager`, `disable-scheduler`, `disable-agent` at etcd-only, empty otherwise) and records the level k3s starts at in `/run/github-runner-k3s/regrade-level`, which `current_level` reads, since k3s hides its arguments from `/proc`. Both files are written atomically.\n\n**Waiting for the step.** After a restart `apply_level` returns only when etcd serves again (k3s's `ETCD server is now running` in the journal since the restart, the line the Docker actuator waits for) and, for a level that runs the agent, when the Node is Ready on a heartbeat later than the restart, because the core releases the Lease as soon as it returns.\n\n**Pods at etcd-only.** The unit's `KillMode=process` leaves pod shims running when k3s restarts without its agent, so pods that survived the drain (DaemonSet pods) would keep running with no kubelet and keep their memory, which defeats the step. Entering etcd-only stops the remaining shims, as `k3s-killall.sh` does for containers; leaving it starts the agent, which recreates them.\n\n**The service.** `fleet-regrade.service` runs the core and this actuator with the host's own `/proc/pressure/memory` and `/proc/meminfo`, `KUBECTL=\"k3s kubectl\"` and the admin kubeconfig, and the environment file the role already templates. The role's refusals carry over unchanged (not on an agent, not on the only server, not on a `headscale_in_cluster` bootstrap server, not turned off below `full`), plus a kernel without PSI is refused before anything changes, as the Docker wrapper stops for it.\n\n**Reaching the API.** `KUBE_SERVER` is today the first other server in inventory order, fixed when the role runs. Once roles change at run time that server may be an agent, so the core takes a list of candidate servers (every other server and every eligible host, by their certificate names) and uses the first whose `/readyz` answers, never its own. This is the one change to the core, and the Docker actuator takes it too.\n\nTests: `tests/regrade` gains the systemd actuator against the existing fake kubectl and a fake `systemctl` and journal, for every level change in both directions and the waits. The mesh scenario with the native server steps it down to etcd-only and back under simulated pressure with the real unit, real k3s and a real drop-in: membership unchanged, the vote kept (a write succeeds with one other server stopped), the level kept across `systemctl restart` and across stopping and starting the systemd container, and no shim left at etcd-only. On a real native server, `fleet-regrade status` runs before regrading is turned on.\n","author":{"url":"https://github.com/Mearman","@type":"Person","name":"Mearman"},"datePublished":"2026-11-03T07:02:03Z","url":"https://github.com/ExaDev/github-runner/issues/181"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:536f4984-0ebf-f223-6831-61544177b042 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | A8EA:22F32B:54645F:766CED:6B0076B6 |
| html-safe-nonce | b67e8496cc6060f286b1abaa1ebab4e48f839ecf71225a568a46fcc2a3bf12b3 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBOEVBOjIyRjMyQjo1NDY0NUY6NzY2Q0VEOjZCMDA3NkI2IiwidmlzaXRvcl9pZCI6IjI0NzY2MjEwOTQwNjUzMDUyNzAiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | e7a8d8c30021bbca88e20376fd51e757613cce15d785fb7044aec5516cc0d11f |
| hovercard-subject-tag | issue:6177867574 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/ExaDev/github-runner/181/issue_layout |
| twitter:image | https://opengraph.githubassets.com/fa85a39f2e376ae63c8d9291d53ec66e29a0b16fd8a809f962a413e1f356c4fd/ExaDev/github-runner/issues/181 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/fa85a39f2e376ae63c8d9291d53ec66e29a0b16fd8a809f962a413e1f356c4fd/ExaDev/github-runner/issues/181 |
| og:image:alt | Memory-pressure regrading (github_runner_cluster_regrade) has a Docker actuator only. A native server (the native server mode sub-issue) needs a systemd actuator implementing the core's three funct... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | Mearman |
| hostname | github.com |
| expected-hostname | github.com |
| None | 75c2b65dd205f26961d0c29afed1a8d0bb7cca9f05780571dfd1ef93eb539d16 |
| turbo-cache-control | no-preview |
| go-import | github.com/ExaDev/github-runner git https://github.com/ExaDev/github-runner.git |
| octolytics-dimension-user_id | 140118018 |
| octolytics-dimension-user_login | ExaDev |
| octolytics-dimension-repository_id | 1388126104 |
| octolytics-dimension-repository_nwo | ExaDev/github-runner |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 1388126104 |
| octolytics-dimension-repository_network_root_nwo | ExaDev/github-runner |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 84e563eb4b55f8fa102f325f117533cee9e3df4a |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width