Title: A control-plane node's Tailscale key expired: etcd voter cut off from the tailnet · Issue #256 · ExaDev/github-runner · GitHub
Open Graph Title: A control-plane node's Tailscale key expired: etcd voter cut off from the tailnet · Issue #256 · ExaDev/github-runner
X Title: A control-plane node's Tailscale key expired: etcd voter cut off from the tailnet · Issue #256 · ExaDev/github-runner
Description: A control-plane node (an etcd voter) lost its Tailscale link: tailscale ping to it reports "peer's node key has expired", ssh over the tailnet times out, and etcd on the control plane logs "timed out waiting for read index response (loca...
Open Graph Description: A control-plane node (an etcd voter) lost its Tailscale link: tailscale ping to it reports "peer's node key has expired", ssh over the tailnet times out, and etcd on the control plane logs "timed o...
X Description: A control-plane node (an etcd voter) lost its Tailscale link: tailscale ping to it reports "peer's node key has expired", ssh over the tailnet times out, and etcd on the control plane...
Opengraph URL: https://github.com/ExaDev/github-runner/issues/256
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"A control-plane node's Tailscale key expired: etcd voter cut off from the tailnet","articleBody":"A control-plane node (an etcd voter) lost its Tailscale link: `tailscale ping` to it reports \"peer's node key has expired\", ssh over the tailnet times out, and etcd on the control plane logs \"timed out waiting for read index response (local node might have slow network)\" while nodes flap NotReady and Ready. Quorum still holds with the other two voters, but the cluster is degraded until the key is renewed.\n\nRe-keying a node is an action on the owner's Tailscale account, so it is left to them, the same as the offline laptop node in #255.\n\nRevisit trigger: the owner re-authenticates the node on the tailnet. Verify afterwards with `tailscale ping` to it, no read-index timeouts in the k3s log, and all voters Ready.\n\n\n\n## Update, 2026-11-16 11:08\n\nThe control-plane container on one voter restarted at 09:10 (a clean exit, not an out-of-memory kill; the cause is not confirmed), and the node cut off from the tailnet now reports NotReady. The cluster is back with two of its three voters Ready, so it has no margin: losing one more voter breaks quorum. Only two etcd read-index timeouts were logged in the last 90 minutes, so the control plane is calm for now. Re-authenticating the cut-off node is still the fix.\n\n\n\n## Update, 2026-11-16 11:32\n\nThe control-plane container restarted a second time at 11:13 (the first was 09:10), each time a graceful, clean exit that the container runtime then brought straight back (`[graceful-termination] secure server is exiting`, exit code 0, not an out-of-memory kill). The second restart also logged the container's own Tailscale starting a fresh login. The cause is not established; the two restarts are about two hours apart, and the cluster is at two of three voters. It is tracked here because the restarts lengthen the window in which one more voter lost breaks quorum.\n\n\n\n## Update, 2026-11-16 14:52\n\nThe control-plane container has restarted nine more times since the 11:13 restart: the restart count read 20 then and reads 29 now, with the latest start at 14:50. It is running, not out-of-memory killed and exited 0 on the last restart, so each is still a clean exit followed by an immediate restart by the runtime. Cluster state at 14:52: five nodes Ready (the other voters among them), three NotReady, so the cluster is still at two of three voters. The restarts are frequent enough that a lost second voter during one would break quorum; re-authenticating the third voter remains the fix that adds margin.\n\n\n\n## Update, 2026-11-16 16:13\n\nThe control-plane container restarted once more at 16:09 (restart count 29 to 30), after about an hour and a quarter without one since 14:50, so the restarts continue but at no regular interval. Cluster state at 16:13 is unchanged: five nodes Ready, three NotReady, still two of three voters.\n\n\n\n## Update, 2026-11-16 16:42\n\nThe control-plane container restarted twice more (count 30 at 16:09, 32 at 16:37). The log just before the 16:37 start shows its own Tailscale starting from nothing: \"monitor: gateway and self IP changed\", \"load netmap from cache: netmap cache is not available\", \"Tailscale is starting\", a fresh registration. So each restart brings the container's Tailscale up cold and it re-registers with the control server; the netmap is not kept across restarts. The restarts are still clean exits that the runtime brings straight back. What triggers the exit is not in that log; Docker events for the container are the next place to read. Cluster state is unchanged: five nodes Ready, three NotReady, still two of three voters.\n\n\n\n## Update, 2026-11-16 18:03\n\nThe control-plane container has restarted three more times since the 16:37 entry (count 34 at 17:22, 37 at 17:53, latest start 17:53). Over the afternoon the interval between restarts has shortened to roughly half an hour or less. Each is still a clean exit followed by an immediate restart by the runtime, with its Tailscale coming up cold. Cluster state is unchanged: five nodes Ready, three NotReady, two of three voters.\n\n\n\n## Update, 2026-11-16 20:40\n\nThe third voter is back. The mini node has been Ready since 19:23:57 UTC, after its Colima virtual machine was found dead and restarted (the Docker socket answered \"Invalid virtual machine state\"; `colima status` returned no runtime). That, and not only the logged-out host Tailscale, was keeping the node NotReady. The cluster is at three of three voters Ready (m1, m3, mini) with the Air and the Linux laptop still NotReady. The mini's host Tailscale is still logged out (`tailscale status` reports \"Logged out\"), and the node is reaching the cluster through the runner container's own Tailscale address, so re-authenticating the host is no longer what stands between the cluster and quorum margin, but the host is still off the tailnet. The control-plane container has not restarted since 19:11 UTC (count 40), about 1.5 hours.\n\n\n\n## Update, 2026-11-17 00:34\n\nThe mini node went NotReady again at 00:13:34 UTC (\"Kubelet stopped posting node status\"), about five hours after it came back at 19:23 UTC, so the cluster is back to two of three voters. The mini host itself is healthy (load 1.8, reachable over the LAN); its Colima status and virtual machine state were not readable in this check, so whether the VM died again is not established. The control-plane container restarted twice more in the evening (count 40 at 19:11, 42 at 00:23). The earlier note that the mini was no longer the blocker holds only while its VM stays up: this is the second time its kubelet has stopped posting status.\n","author":{"url":"https://github.com/Mearman","@type":"Person","name":"Mearman"},"datePublished":"2026-11-16T03:36:02Z","url":"https://github.com/ExaDev/github-runner/issues/256"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:e15a311b-f230-6530-6146-37516982f326 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | D54E:1ED25A:469FA4:60B81E:6B007048 |
| html-safe-nonce | e2f5078f0701fdf071463e52cc933cfc57e0df9ea26dc5c233cdedef6aadb67d |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJENTRFOjFFRDI1QTo0NjlGQTQ6NjBCODFFOjZCMDA3MDQ4IiwidmlzaXRvcl9pZCI6IjQ0NjUyODEyNzUyNzU1Mzg1MDQiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | bcc5d35c664130ba37d549436b9fb074d2135a6e030f113db9dd334c4049b75a |
| hovercard-subject-tag | issue:6413617066 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/ExaDev/github-runner/256/issue_layout |
| twitter:image | https://opengraph.githubassets.com/83e65b43f9d9d6000d3b70be053be696ce8e4d6090b02825a7bb95db60810439/ExaDev/github-runner/issues/256 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/83e65b43f9d9d6000d3b70be053be696ce8e4d6090b02825a7bb95db60810439/ExaDev/github-runner/issues/256 |
| og:image:alt | A control-plane node (an etcd voter) lost its Tailscale link: tailscale ping to it reports "peer's node key has expired", ssh over the tailnet times out, and etcd on the control plane logs "timed o... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | Mearman |
| hostname | github.com |
| expected-hostname | github.com |
| None | 6527e91c60c0b29d16606ff32ac5df8ecdc37a2a62205ea057a14d7762c870fa |
| turbo-cache-control | no-preview |
| go-import | github.com/ExaDev/github-runner git https://github.com/ExaDev/github-runner.git |
| octolytics-dimension-user_id | 140118018 |
| octolytics-dimension-user_login | ExaDev |
| octolytics-dimension-repository_id | 1388126104 |
| octolytics-dimension-repository_nwo | ExaDev/github-runner |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 1388126104 |
| octolytics-dimension-repository_network_root_nwo | ExaDev/github-runner |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 46f1cd2ae3cd8ae1a81019b5d619742e76f13bbd |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width