Title: feat: add ServerTailnet logs and metrics for unreachable agents by BobbyHo · Pull Request #30133 · coder/coder · GitHub
Open Graph Title: feat: add ServerTailnet logs and metrics for unreachable agents by BobbyHo · Pull Request #30133 · coder/coder
X Title: feat: add ServerTailnet logs and metrics for unreachable agents by BobbyHo · Pull Request #30133 · coder/coder
Description: Part of PLAT-680, a sub-issue of PLAT-360. TL;DR PLAT-360: after coderd has been up for a long time, its ServerTailnet loses reachability to a subset of workspace agents. Workspace app and web terminal requests to those agents fail with 502 while other agents on the same coderd keep working. A restart clears it, and the customer restarts once the 502 rate crosses their threshold, so the state is gone before anyone can look at it. Today the default-level logs and the existing metrics only say that an agent was unreachable. They do not say where along the path the connection broke: coordinator, WireGuard tunnel, embedded DERP relay, or the agent's own API connection. The Tailscale internals that would tell us only log at debug, and coderd's log level cannot be changed at runtime. This PR adds metrics and request log fields at the default level so the next incident can be narrowed down from a normal Prometheus scrape and warn-level logs, without restarting coderd and without running with debug logging on all the time. chatd also dials agents through the ServerTailnet, so it needs two small changes. It has no request log line, so its failures get their own warn line. It also dials new agents every few seconds while they boot, and abandons dials to agents that a rebuild replaced. Those failures are expected, so chatd marks them and they are not counted. Change When an agent doesn't answer, the 502 request log line gets the agent ID, a reason, how long the request waited, and peer state (node received, lost flag, handshake time, DERP regions, byte counts). The PTY route and callers with no request log, such as chatd, log one warn line with the same fields. Reverse proxy log lines also record which dial step failed, so a failure after the agent answered a ping is not counted as unreachable. New metrics: coder_servertailnet_agent_unreachable_total{reason} coder_servertailnet_await_reachable_seconds coder_servertailnet_derp_connects_total coder_workspaceapps_agent_offline_total{status,access_method}, since the offline page moved from 502 to 404 in #23090. Peer diagnostics gain TX and RX byte counts. A peer with no handshake reports zero time, not 1970, which also fixes coder ping. Reverse proxy dials stop after one minute. Chat readiness polls, which dial while a new agent boots, are not counted. New log fields on unreachable-agent lines Field What it is What it tells us agent_id The agent the request tried to reach. Lets us group failures per agent and check whether the same agents fail every time or the set moves. reason Why the attempt was classified as unreachable. Values are in the next table. The single field that says which layer to look at first. unreachable_after How long the request waited for the agent's first ping before giving up. Separates a slow agent from one that never answers, and shows whether the wait hit the request timeout or the client left. dial_phase Reverse proxy only. await_reachable means waiting for the first ping. tcp_dial means the ping succeeded and the TCP dial to the app port failed. tcp_dial failures are above the tunnel (the app is not listening) and are not counted as unreachable. Only await_reachable points at the network path. tcp_dial_after Reverse proxy only, with dial_phase=tcp_dial. Time until the TCP dial failed. Distinguishes an immediate connection refused from a dial that hung inside the tunnel. peer_node_received Whether the coordinator delivered the agent's node to this coderd. false means the failure is in the control plane: the coordinator never told us where the agent is, so WireGuard was never configured for it. peer_lost Whether the coordinator marked the peer lost, or this coderd's own coordination stream ended. true means the agent's coordinator connection dropped, or ours did. Many agents lost at once points at this coderd, not the agents. server_preferred_derp The DERP region this coderd uses as its home relay. Confirms which relay our side of the tunnel goes through. A change over time means coderd re-homed. peer_preferred_derp The DERP region the agent reported as its home relay. Compared with the server region, shows whether both sides agree on a relay and whether the failing agents share one. peer_last_handshake Time of the last completed WireGuard handshake with this agent. Omitted when there has never been one. An old handshake with recent traffic means the tunnel was working and then stopped, which is the PLAT-360 shape. Missing means the tunnel never came up. peer_tx_bytes / peer_rx_bytes Bytes sent to and received from the agent over WireGuard since the peer was added. RX at zero means nothing ever came back from the agent. Both at zero means our packets never left coderd. Large counts on a failing peer mean it worked earlier. Compare two lines for the same agent to see whether the counts still move. Values of reason Reason Condition Where to look next no_node The coordinator never sent the agent's node, or it was removed after a disconnect or lost timeout. Control plane: coordinator, agent API connection, pubsub. WireGuard was never configured for this agent. peer_lost The coordinator reported the peer lost, or this coderd's coordination stream ended. The agent's coordinator connection or this coderd's. A spike across many agents means this coderd lost its coordinator stream. no_handshake The node arrived but WireGuard never completed a handshake. The path between coderd and the agent: DERP relay, the in-memory pipe to the embedded relay, or the agent's DERP connection. Check the byte counts to see which direction is silent. handshake_stale The last handshake is older than 180 seconds, WireGuard's session limit. The tunnel worked and then our handshakes went unanswered. This is the PLAT-360 symptom. Byte counts show whether anything still flows. handshake_ok The last handshake is under 180 seconds old. The agent answered a handshake recently. It may have stopped answering since then, or the failure may be above WireGuard. Compare the byte counts on two lines for the same agent. diagnostics_timeout The peer state could not be read within one second. The WireGuard engine lock was busy. On its own it is a sign of a stuck engine, which is itself evidence. client_canceled The client went away within five seconds. Not a network failure. Counted separately so it does not inflate the other reasons. New metrics Metric What it is What it tells us coder_servertailnet_agent_unreachable_total{reason} Counter of connection attempts where the agent did not answer its first ping, labeled by the reason above. The rate is the alert signal. The reason split says whether an incident is a control plane failure, a tunnel failure, or something above the tunnel, without reading logs. No per-agent label, so cardinality is fixed. coder_servertailnet_await_reachable_seconds Histogram of time until an agent answered the first ping on a new connection. A drift toward the upper buckets before agents become unreachable is the early warning of a slow loss of reachability, and shows whether the failure is sudden or gradual. coder_servertailnet_derp_connects_total Counter of times the ServerTailnet connected to the embedded DERP relay over the in-memory pipe. Reads 1 after startup, or stays 0 when the embedded relay is disabled. A rise means the relay connection was rebuilt. If it rises alongside unreachable failures, the in-memory pipe to the embedded relay is the suspect. If it stays flat, the relay side is stable. coder_workspaceapps_agent_offline_total{status,access_method} Counter of app and terminal requests that got the offline page because the agent's database status was not connected. These requests never reach the tunnel. A rise means agents are dropping their API connection to coderd, which is a different failure from the tunnel losing them. Separates the two 502 writers the customer sees. Where each signal sits on the network path The diagram shows the DERP-only path, which is the customer's setup. With direct connections enabled, WireGuard traffic can bypass the relay, but the coordinator, offline check, and handshake fields are the same. browser / CLI | | HTTPS: workspace app, web terminal, chat v +-------------------------------------------------------------------------------+ | coderd | | | | app proxy / PTY route / chatd | | agent DB status check | | [M4] agent_offline_total ........ agent not "connected" in the database, | | | request stops here (agent API is down) | | v | | ServerTailnet.AgentConn / ReverseProxy | | wait for first ping | | [M2] await_reachable_seconds .... time to first answer on success | | [M1] agent_unreachable_total .... no answer; request log line gets | | [L] reason, unreachable_after, agent_id, reason, dial_phase, peer_* | | dial_phase, tcp_dial_after | | | | | | control plane: coordinator (node exchange) | | | [L] peer_node_received, peer_lost | | v | | WireGuard / magicsock | | [L] peer_last_handshake, peer_tx_bytes, peer_rx_bytes, | | server_preferred_derp, peer_preferred_derp | | | | | | in-memory pipe | | v | | embedded DERP relay | | [M3] derp_connects_total ........ times this pipe was (re)built | +---------|----------------------------------------------------------------------+ | ^ | DERP over HTTPS / WebSocket | agent API over WebSocket: v | coordination, stats, status workspace agent -------------------------------+ (WireGuard peer, netstack, apps, PTY) M1..M4 new Prometheus metrics L new fields on the warn-level log line Reading top to bottom: a rise in M4 means agents are losing their API connection before any tunnel is involved. M1 with no_node or peer_lost means the coordinator step failed. M1 with no_handshake or handshake_stale means the WireGuard step failed, and the byte counts say in which direction. A matching rise in M3 puts the fault at the relay pipe. M1 with handshake_ok means the last handshake was recent; it does not show whether the tunnel still works.
Open Graph Description: Part of PLAT-680, a sub-issue of PLAT-360. TL;DR PLAT-360: after coderd has been up for a long time, its ServerTailnet loses reachability to a subset of workspace agents. Workspace app and web term...
X Description: Part of PLAT-680, a sub-issue of PLAT-360. TL;DR PLAT-360: after coderd has been up for a long time, its ServerTailnet loses reachability to a subset of workspace agents. Workspace app and web term...
Opengraph URL: https://github.com/coder/coder/pull/30133
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:24131fe9-1cfa-133f-02c4-4cf5ee815c07 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | A5B4:38C5CF:3193685:3ED9C21:6B01B186 |
| html-safe-nonce | 283803f0fbf8882520b33a8f5664300c5f5bbd681d1c1e891a7a773e2a13caa8 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBNUI0OjM4QzVDRjozMTkzNjg1OjNFRDlDMjE6NkIwMUIxODYiLCJ2aXNpdG9yX2lkIjoiMzYzMTQyMDMzNjg5OTIwNzU1OCIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 8fd8e0c90cd7ee1e0c88dfa13c7ea2cd91184539d0765d3d367a9e088f0b7c14 |
| hovercard-subject-tag | pull_request:4683467942 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/coder/coder/pull/30133/files |
| twitter:image | https://avatars.githubusercontent.com/u/23092290?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/23092290?s=400&v=4 |
| og:image:alt | Part of PLAT-680, a sub-issue of PLAT-360. TL;DR PLAT-360: after coderd has been up for a long time, its ServerTailnet loses reachability to a subset of workspace agents. Workspace app and web term... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | cc5dfdd32f62ebd3223cab3af3aa2b268d3605218237326a48a99d1efd89495c |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/coder/coder git https://github.com/coder/coder.git |
| octolytics-dimension-user_id | 95932066 |
| octolytics-dimension-user_login | coder |
| octolytics-dimension-repository_id | 440752086 |
| octolytics-dimension-repository_nwo | coder/coder |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 440752086 |
| octolytics-dimension-repository_network_root_nwo | coder/coder |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 8c77c9e8439031ad54c75a250694842892f693e1 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width