René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Delay in determining a KVM host is down","articleBody":"### problem\n\nWith the default configurations, CloudStack determines a KVM host is down in 15-20 minutes. The HA-enabled instances will be started on another host only after this process. While reviewing the delay for the host state investigation followed by a ping timeout I see one command that takes 10 minutes **'com.cloud.agent.api.CheckOnHostCommand** printing in the logs the following message 'timed out after 3600'. Later the host is determined as down via the neighbouring host quickly. \n\nI suspect there is some issue in this specific implementation and if fixed the VM HA delay in KVM could be reduced by 10 minutes. \n\n```\n2025-01-28 06:22:30,041 DEBUG [c.c.a.t.Request] (AgentTaskPool-4:ctx-1ece8add) (logid:49d74f9a) Seq 2-4979573812988215360: Sending  { Cmd , MgmtId: 32988184186020, via: 2(ref-trl-5786-k-Mu22-jithin-raju-kvm2), Ver: v1, Flags: 100011, [{\"com.cloud.agent.api.CheckOnHostCommand\":{\"host\":{\"guid\":\"439751ba-a6eb-3103-b60d-8321f53224fb-LibvirtComputingResource\",\"privateNetwork\":{\"ip\":\"10.1.33.180\",\"netmask\":\"255.255.240.0\",\"mac\":\"1e:00:a9:00:0a:72\",\"isSecurityGroupEnabled\":\"false\"},\"storageNetwork1\":{\"ip\":\"10.1.33.180\",\"netmask\":\"255.255.240.0\",\"mac\":\"1e:00:a9:00:0a:72\",\"isSecurityGroupEnabled\":\"false\"}},\"reportCheckFailureIfOneStorageIsDown\":\"false\",\"wait\":\"0\",\"bypassHostMaintenance\":\"false\"}}] }\n2025-01-28 06:32:14,792 DEBUG [c.c.a.m.AgentAttache] (AgentTaskPool-4:ctx-1ece8add) (logid:49d74f9a) Seq 2-4979573812988215360: Waiting some more time because this is the current command\n2025-01-28 06:32:14,792 DEBUG [c.c.a.m.AgentAttache] (AgentTaskPool-4:ctx-1ece8add) (logid:49d74f9a) Seq 2-4979573812988215360: Waiting some more time because this is the current command\n2025-01-28 06:32:14,792 WARN  [c.c.a.m.AgentAttache] (AgentTaskPool-4:ctx-1ece8add) (logid:49d74f9a) Seq 2-4979573812988215360: Timed out on Seq 2-4979573812988215360:  { Cmd , MgmtId: 32988184186020, via: 2(ref-trl-5786-k-Mu22-jithin-raju-kvm2), Ver: v1, Flags: 100011, [{\"com.cloud.agent.api.CheckOnHostCommand\":{\"host\":{\"guid\":\"439751ba-a6eb-3103-b60d-8321f53224fb-LibvirtComputingResource\",\"privateNetwork\":{\"ip\":\"10.1.33.180\",\"netmask\":\"255.255.240.0\",\"mac\":\"1e:00:a9:00:0a:72\",\"isSecurityGroupEnabled\":\"false\"},\"storageNetwork1\":{\"ip\":\"10.1.33.180\",\"netmask\":\"255.255.240.0\",\"mac\":\"1e:00:a9:00:0a:72\",\"isSecurityGroupEnabled\":\"false\"}},\"reportCheckFailureIfOneStorageIsDown\":\"false\",\"wait\":\"0\",\"bypassHostMaintenance\":\"false\"}}] }\n2025-01-28 06:32:14,793 DEBUG [c.c.a.m.AgentAttache] (AgentTaskPool-4:ctx-1ece8add) (logid:49d74f9a) Seq 2-4979573812988215360: Cancelling.\n2025-01-28 06:32:14,793 WARN  [c.c.a.m.AgentManagerImpl] (AgentTaskPool-4:ctx-1ece8add) (logid:49d74f9a) Operation timed out: Commands 4979573812988215360 to Host 2 timed out after 3600\n```\n\nhttps://gist.github.com/rajujith/9a51c52163eb4862b497057a40e8b812#file-acs-kvm-vm-ha-host-down\n\n### versions\n\n4.19.1.3\n\n### The steps to reproduce the bug\n\n1. Power off the host via ILO/IDRAC or power off the nested hypervisor through the base hypervisor.\n2. Observe the delay in VM HA and review the logs\n\n...\n\n\n### What to do about it?\n\nReduce the delay in the VM HA on KVM. ","author":{"url":"https://github.com/rajujith","@type":"Person","name":"rajujith"},"datePublished":"2025-02-27T06:05:28.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":4},"url":"https://github.com/10477/cloudstack/issues/10477"}