René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Intermittent Console Proxy Failure in 4.22.1.0","articleBody":"### problem\n\nOn a 4.22.1.0 deployment with two management servers behind a load balancer VM and system-VM console access fails intermittently. The browser receives a generic Apache-style \"Internal Server Error\" page. Reopening the console sometimes works, sometimes fails, with no change to the environment.\n\nThe failure has been isolated to the console session token validation path. Every surrounding component has been verified healthy. Pinning the browser directly to a single management server (bypassing the load balancer) does not resolve it, so this is not simply LB source-IP rewriting.\n\n- Browser shows stock \"Internal Server Error\" (Apache-style) page intermittently when opening a console.\n- CPVM (/var/log/cloud/cloud.out) on failure:\n    Session \u003cuuid\u003e has already been used, cannot connect\n    External authenticator failed authentication request for vm \u003cvm-uuid\u003e with sid \u003csid\u003e\n    ERROR ConsoleProxyNoVNCHandler ... Failed to create viewer ...\n      com.cloud.consoleproxy.AuthenticationException: External authenticator failed request\n- Also observed earlier: org.eclipse.jetty.websocket.api.CloseException: TimeoutException: Idle timeout expired: 300000/300000 ms.\n\nKEY DIAGNOSTIC FINDING\n \nThe cloud.console_session table records the load balancer IP (10.125.128.38) as\nconsole_endpoint_creator_address for sessions, and a subset of rows never reach acquired/removed\n(these correlate with the failures):\n \n    id    uuid        instance  host  acquired             removed              creator_address  client_address\n    1414  94403a87..  6         14    NULL                 NULL                 172.31.1.204     NULL\n    1413  91842ef1..  85        14    2026-06-15 13:40:54  2026-06-15 13:41:00  172.31.1.204     172.31.1.204\n    1412  ade60441..  1902      27    2026-06-15 13:39:10  2026-06-15 13:39:20  10.125.128.38    172.31.1.204\n    1411  135aaa62..  1902      27    2026-06-15 13:37:16  2026-06-15 13:39:10  10.125.128.38    172.31.1.204\n    1410  6cd4ac53..  1881      27    NULL                 NULL                 10.125.128.38    NULL\n \nconsole_endpoint_creator_address is being recorded as the load balancer (10.125.128.38) and in\nsome rows as the client IP (172.31.1.204) — never as a real management server IP (.37/.39).\nNeither .38 nor the client is a valid validation target, which appears to be why those sessions\nare never acquired.\n \nOn failure, NOTHING is logged on either management server (management-server.log filtered for\nconsole|authentication|failed|console_session shows only cluster heartbeat). On success, the\nminting MS logs the full createConsoleEndpoint -\u003e Compose console url -\u003e Adding allowed session\n-\u003e ConsoleAccessAuthenticationCommand flow. So failing validations are being rejected at the\nCPVM and never reach a real MS.\n \n \nRULED OUT (verified, not assumed)\n \n- CPVM health: up 3+ days, ports 80/8001/8080 bound by the cloud Java process (no stray Apache),\n  actively serving noVNC. curl of /resource/noVNC/vnc.html returns 200.\n- Network/MTU: DF ping at 1472 bytes succeeds 0% loss between client and CPVM; path MTU is full 1500.\n- Management servers: both Up in mshost; both reachable on 8250 from CPVM; clocks within ~1s\n  (verified date -u).\n- KVM host clock: within ~1s of MS.\n- VNC port: virsh vncdisplay v-1902-VM = :4 (5904), exactly matching the port the MS hands out —\n  no stale-port mismatch.\n- Agent version: host cloudstack-agent = 4.22.1.0-shapeblue0, matches MS.\n- cluster.node.IP: correctly set per node; no rogue/extra management instance;\n- CPVM rebuild: destroyed/recreated multiple times — no effect.\n- Pinning browser directly to a single MS (https://\u003cmgmt-server-ip\u003e:8080/client), bypassing the LB:\n  still fails. (So this is NOT solely LB source-IP rewriting / cross-MS in-memory token, despite\n  PR #7094 being present and the console_session table being populated.)\n \n \nWHY THIS LOOKS LIKE A BUG, NOT MISCONFIGURATION\n \n- PR #7094 (DB-backed console sessions for multi-MS) is present — the console_session table exists\n  and is written.\n- Yet console_endpoint_creator_address is being populated with the load balancer IP and sometimes\n  the client IP, rather than the processing management server's own IP. Those values are not valid\n  validation targets.\n- Bypassing the load balancer (direct-to-MS) does not fix it, so source-IP rewriting by the LB is\n  not a complete explanation.\n- The result is intermittent, single-use-token \"already used / external authenticator failed\"\n  rejections at the CPVM, with no corresponding log on any management server.\n\n### versions\n\n- CloudStack version: 4.22.1.0-shapeblue0\n- Hypervisor: KVM (host agent cloudstack-agent 4.22.1.0-shapeblue0, matched to MS)\n- Management servers: 2 nodes\n    mshost table: both nodes Up\n- Haproxy load balancers (fronts the management/UI tier; NOT a management server)\n- SSL: disabled (consoleproxy.sslEnabled=false, no consoleproxy.url.domain)\n- CPVM: v-1902-VM, agent up, serving noVNC\n- Client: Windows workstation\n- Console settings present: console.session.cleanup.interval=180,\n  console.session.cleanup.retention.hours=240,\n  consoleproxy.session.timeout=300000, consoleproxy.session.max=50,\n  novnc.console.default=true, novnc.console.sourceip.check.enabled=false\n\n### The steps to reproduce the bug\n\n1. Deploy CloudStack 4.22.1.0 with two management servers behind a load balancer fronting the management/UI tier. KVM hypervisor, SSL disabled.\n2. From a client on a different subnet, open the CloudStack UI through the load balancer and click \"View Console\" on a running VM or system VM.\n3. Repeat opening the console several times.\n4. Observe that console access succeeds intermittently — some attempts load the noVNC console, others return a generic \"Internal Server Error\" page in the browser.\n5. On a failing attempt, check the CPVM log (/var/log/cloud/cloud.out):\n     Session \u003cuuid\u003e has already been used, cannot connect\n     External authenticator failed authentication request for vm \u003cvm-uuid\u003e with sid \u003csid\u003e\n     com.cloud.consoleproxy.AuthenticationException: External authenticator failed request\n6. On a failing attempt, check both management-server logs — NOTHING is logged on either MS (no createConsoleEndpoint, no ConsoleAccessAuthenticationCommand). On a succeeding attempt, the full flow IS logged on the minting MS.\n7. Inspect the cloud.console_session table:\n     SELECT id,uuid,acquired,removed,console_endpoint_creator_address,client_address FROM cloud.console_session ORDER BY created DESC LIMIT 10;\n   Note that console_endpoint_creator_address is recorded as the load balancer IP, and in some rows the client IP — never a real management server IP. Rows with these creator addresses are the ones that never reach 'acquired'/'removed', and these correlate with the failures.\n\n### What to do about it?\n\nconsole_endpoint_creator_address is being populated with the load balancer IP (and sometimes the client IP) instead of the processing management server's own management IP. Since the CPVM validates the one-time console token against the recorded creator address, and neither the LB nor the client can service that validation callback, those sessions are never acquired and the CPVM rejects the connection — producing the intermittent HTTP 500.\n\nExpected: console_endpoint_creator_address should be the management server that processed the createConsoleEndpoint call (a real MS IP from the `host` setting), regardless of whether the request arrived via a load balancer.","author":{"url":"https://github.com/tverkade","@type":"Person","name":"tverkade"},"datePublished":"2026-06-15T14:05:27.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/13422/cloudstack/issues/13422"}