René's URL Explorer Experiment


Title: A control-plane node's Tailscale key expired: etcd voter cut off from the tailnet · Issue #256 · ExaDev/github-runner · GitHub

Open Graph Title: A control-plane node's Tailscale key expired: etcd voter cut off from the tailnet · Issue #256 · ExaDev/github-runner

X Title: A control-plane node's Tailscale key expired: etcd voter cut off from the tailnet · Issue #256 · ExaDev/github-runner

Description: A control-plane node (an etcd voter) lost its Tailscale link: tailscale ping to it reports "peer's node key has expired", ssh over the tailnet times out, and etcd on the control plane logs "timed out waiting for read index response (loca...

Open Graph Description: A control-plane node (an etcd voter) lost its Tailscale link: tailscale ping to it reports "peer's node key has expired", ssh over the tailnet times out, and etcd on the control plane logs "timed o...

X Description: A control-plane node (an etcd voter) lost its Tailscale link: tailscale ping to it reports "peer's node key has expired", ssh over the tailnet times out, and etcd on the control plane...

Opengraph URL: https://github.com/ExaDev/github-runner/issues/256

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"A control-plane node's Tailscale key expired: etcd voter cut off from the tailnet","articleBody":"A control-plane node (an etcd voter) lost its Tailscale link: `tailscale ping` to it reports \"peer's node key has expired\", ssh over the tailnet times out, and etcd on the control plane logs \"timed out waiting for read index response (local node might have slow network)\" while nodes flap NotReady and Ready. Quorum still holds with the other two voters, but the cluster is degraded until the key is renewed.\n\nRe-keying a node is an action on the owner's Tailscale account, so it is left to them, the same as the offline laptop node in #255.\n\nRevisit trigger: the owner re-authenticates the node on the tailnet. Verify afterwards with `tailscale ping` to it, no read-index timeouts in the k3s log, and all voters Ready.\n\n\n\n## Update, 2026-11-16 11:08\n\nThe control-plane container on one voter restarted at 09:10 (a clean exit, not an out-of-memory kill; the cause is not confirmed), and the node cut off from the tailnet now reports NotReady. The cluster is back with two of its three voters Ready, so it has no margin: losing one more voter breaks quorum. Only two etcd read-index timeouts were logged in the last 90 minutes, so the control plane is calm for now. Re-authenticating the cut-off node is still the fix.\n\n\n\n## Update, 2026-11-16 11:32\n\nThe control-plane container restarted a second time at 11:13 (the first was 09:10), each time a graceful, clean exit that the container runtime then brought straight back (`[graceful-termination] secure server is exiting`, exit code 0, not an out-of-memory kill). The second restart also logged the container's own Tailscale starting a fresh login. The cause is not established; the two restarts are about two hours apart, and the cluster is at two of three voters. It is tracked here because the restarts lengthen the window in which one more voter lost breaks quorum.\n\n\n\n## Update, 2026-11-16 14:52\n\nThe control-plane container has restarted nine more times since the 11:13 restart: the restart count read 20 then and reads 29 now, with the latest start at 14:50. It is running, not out-of-memory killed and exited 0 on the last restart, so each is still a clean exit followed by an immediate restart by the runtime. Cluster state at 14:52: five nodes Ready (the other voters among them), three NotReady, so the cluster is still at two of three voters. The restarts are frequent enough that a lost second voter during one would break quorum; re-authenticating the third voter remains the fix that adds margin.\n\n\n\n## Update, 2026-11-16 16:13\n\nThe control-plane container restarted once more at 16:09 (restart count 29 to 30), after about an hour and a quarter without one since 14:50, so the restarts continue but at no regular interval. Cluster state at 16:13 is unchanged: five nodes Ready, three NotReady, still two of three voters.\n\n\n\n## Update, 2026-11-16 16:42\n\nThe control-plane container restarted twice more (count 30 at 16:09, 32 at 16:37). The log just before the 16:37 start shows its own Tailscale starting from nothing: \"monitor: gateway and self IP changed\", \"load netmap from cache: netmap cache is not available\", \"Tailscale is starting\", a fresh registration. So each restart brings the container's Tailscale up cold and it re-registers with the control server; the netmap is not kept across restarts. The restarts are still clean exits that the runtime brings straight back. What triggers the exit is not in that log; Docker events for the container are the next place to read. Cluster state is unchanged: five nodes Ready, three NotReady, still two of three voters.\n\n\n\n## Update, 2026-11-16 18:03\n\nThe control-plane container has restarted three more times since the 16:37 entry (count 34 at 17:22, 37 at 17:53, latest start 17:53). Over the afternoon the interval between restarts has shortened to roughly half an hour or less. Each is still a clean exit followed by an immediate restart by the runtime, with its Tailscale coming up cold. Cluster state is unchanged: five nodes Ready, three NotReady, two of three voters.\n\n\n\n## Update, 2026-11-16 20:40\n\nThe third voter is back. The mini node has been Ready since 19:23:57 UTC, after its Colima virtual machine was found dead and restarted (the Docker socket answered \"Invalid virtual machine state\"; `colima status` returned no runtime). That, and not only the logged-out host Tailscale, was keeping the node NotReady. The cluster is at three of three voters Ready (m1, m3, mini) with the Air and the Linux laptop still NotReady. The mini's host Tailscale is still logged out (`tailscale status` reports \"Logged out\"), and the node is reaching the cluster through the runner container's own Tailscale address, so re-authenticating the host is no longer what stands between the cluster and quorum margin, but the host is still off the tailnet. The control-plane container has not restarted since 19:11 UTC (count 40), about 1.5 hours.\n\n\n\n## Update, 2026-11-17 00:34\n\nThe mini node went NotReady again at 00:13:34 UTC (\"Kubelet stopped posting node status\"), about five hours after it came back at 19:23 UTC, so the cluster is back to two of three voters. The mini host itself is healthy (load 1.8, reachable over the LAN); its Colima status and virtual machine state were not readable in this check, so whether the VM died again is not established. The control-plane container restarted twice more in the evening (count 40 at 19:11, 42 at 00:23). The earlier note that the mini was no longer the blocker holds only while its VM stays up: this is the second time its kubelet has stopped posting status.\n","author":{"url":"https://github.com/Mearman","@type":"Person","name":"Mearman"},"datePublished":"2026-11-16T03:36:02Z","url":"https://github.com/ExaDev/github-runner/issues/256"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:e15a311b-f230-6530-6146-37516982f326
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-idD54E:1ED25A:469FA4:60B81E:6B007048
html-safe-noncee2f5078f0701fdf071463e52cc933cfc57e0df9ea26dc5c233cdedef6aadb67d
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJENTRFOjFFRDI1QTo0NjlGQTQ6NjBCODFFOjZCMDA3MDQ4IiwidmlzaXRvcl9pZCI6IjQ0NjUyODEyNzUyNzU1Mzg1MDQiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmacbcc5d35c664130ba37d549436b9fb074d2135a6e030f113db9dd334c4049b75a
hovercard-subject-tagissue:6413617066
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/ExaDev/github-runner/256/issue_layout
twitter:imagehttps://opengraph.githubassets.com/83e65b43f9d9d6000d3b70be053be696ce8e4d6090b02825a7bb95db60810439/ExaDev/github-runner/issues/256
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/83e65b43f9d9d6000d3b70be053be696ce8e4d6090b02825a7bb95db60810439/ExaDev/github-runner/issues/256
og:image:altA control-plane node (an etcd voter) lost its Tailscale link: tailscale ping to it reports "peer's node key has expired", ssh over the tailnet times out, and etcd on the control plane logs "timed o...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernameMearman
hostnamegithub.com
expected-hostnamegithub.com
None6527e91c60c0b29d16606ff32ac5df8ecdc37a2a62205ea057a14d7762c870fa
turbo-cache-controlno-preview
go-importgithub.com/ExaDev/github-runner git https://github.com/ExaDev/github-runner.git
octolytics-dimension-user_id140118018
octolytics-dimension-user_loginExaDev
octolytics-dimension-repository_id1388126104
octolytics-dimension-repository_nwoExaDev/github-runner
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id1388126104
octolytics-dimension-repository_network_root_nwoExaDev/github-runner
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release46f1cd2ae3cd8ae1a81019b5d619742e76f13bbd
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/ExaDev/github-runner/issues/256#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FExaDev%2Fgithub-runner%2Fissues%2F256
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FExaDev%2Fgithub-runner%2Fissues%2F256
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=ExaDev%2Fgithub-runner
Reloadhttps://github.com/ExaDev/github-runner/issues/256
Reloadhttps://github.com/ExaDev/github-runner/issues/256
Reloadhttps://github.com/ExaDev/github-runner/issues/256
ExaDevhttps://github.com/ExaDev
github-runnerhttps://github.com/ExaDev/github-runner
Codehttps://github.com/ExaDev/github-runner
Issues46 (46)https://github.com/ExaDev/github-runner/issues
Pull requests6 (6)https://github.com/ExaDev/github-runner/pulls
Actionshttps://github.com/ExaDev/github-runner/actions
Security and qualityhttps://github.com/ExaDev/github-runner/security
Insightshttps://github.com/ExaDev/github-runner/insights
https://github.com/ExaDev/github-runner
New issuehttps://github.com/login?return_to=https://github.com/ExaDev/github-runner/issues/256
New issuehttps://github.com/login?return_to=https://github.com/ExaDev/github-runner/issues/256
A control-plane node's Tailscale key expired: etcd voter cut off from the tailnethttps://github.com/ExaDev/github-runner/issues/256#top
deferredDeferred until a stated triggerhttps://github.com/ExaDev/github-runner/issues?q=state%3Aopen%20label%3A%22deferred%22
https://github.com/Mearman
Mearmanhttps://github.com/Mearman
on Nov 16, 2026https://github.com/ExaDev/github-runner/issues/256#issue-6413617066
#255https://github.com/ExaDev/github-runner/issues/255
Mearmanhttps://github.com/Mearman
deferredDeferred until a stated triggerhttps://github.com/ExaDev/github-runner/issues?q=state%3Aopen%20label%3A%22deferred%22
on Nov 16, 2026https://github.com/ExaDev/github-runner/issues/256#event-36234812870
Mearmanhttps://github.com/Mearman
on Nov 16, 2026https://github.com/ExaDev/github-runner/issues/256#event-36234820501
https://github.com/Mearman
Mearmanhttps://github.com/Mearman
on Nov 17, 2026https://github.com/ExaDev/github-runner/issues/256#issuecomment-6935991290
https://github.com/Mearman
Mearmanhttps://github.com/Mearman
on Nov 20, 2026https://github.com/ExaDev/github-runner/issues/256#issuecomment-7031245079
Sign up for freehttps://github.com/signup?return_to=https://github.com/ExaDev/github-runner/issues/256
Sign in to commenthttps://github.com/login?return_to=https://github.com/ExaDev/github-runner/issues/256
deferredDeferred until a stated triggerhttps://github.com/ExaDev/github-runner/issues?q=state%3Aopen%20label%3A%22deferred%22
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.