René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"KVM Host HA: host reaches Fenced but VMs remain Running on failed host and HA work is marked Done without restart","articleBody":"##### ISSUE TYPE\n\n* Bug Report\n\n##### COMPONENT NAME\n\nKVM, HA, Orchestration\n\n##### CLOUDSTACK VERSION\n\n4.22.x\n\n##### CONFIGURATION\n\n- KVM cluster\n- Host HA enabled\n- VM HA enabled\n- Shared storage - netapp NFS 4.1\n- 2 management servers\n- Host HA provider: `kvmhaprovider`\n- OBM reconfigured to IPMI for this test\n- Relevant tuning applied before this test:\n  - `wait = 20`\n  - `kvm.ha.activity.check.interval = 20`\n  - `kvm.ha.activity.check.max.attempts = 5`\n  - `kvm.ha.degraded.max.period = 120`\n\n##### OS / ENVIRONMENT\n\nACS 4.22\nNetApp NFS 4.1\nUbuntu 24.04\nmysql  Ver 8.0.45-0ubuntu0.24.04.1\nlibvirt0:amd64   10.0.0-2ubuntu8.11 \n\n##### SUMMARY\n\nWhen a KVM host is powered off, CloudStack eventually fences the host, but some VMs that were on the failed host remain in `Running` state on that host and are not restarted elsewhere.\n\nIn this case:\n- the failed host reaches `ha_state = Fenced`\n- HA work items are created for the affected VMs with reason `HostDown`\n- those HA work items are marked `Done`\n- but the affected VMs still remain:\n  - `state = Running`\n  - `host_id = failed host`\n  - `power_state = PowerOn`\n  - `power_host = failed host`\n\nSo CloudStack appears to consider HA recovery complete even though the VMs are still recorded as running on a fenced/offline host.\n\n##### STEPS TO REPRODUCE\n\n1. Configure a KVM cluster with Host HA and VM HA enabled. (ipmi oob)\n2. Ensure there are user/system VMs running on host.\n3. Tune the HA settings as follows and restart management:\n   - `wait = 20`\n   - `kvm.ha.activity.check.interval = 20`\n   - `kvm.ha.activity.check.max.attempts = 5`\n   - `kvm.ha.degraded.max.period = 120`\n4. Power off host.\n5. Observe management logs and database state (`ha_config`, `op_ha_work`, `vm_instance`).\n\n##### EXPECTED RESULTS\n\nAfter the failed host is confirmed down and fenced:\n\n1. HA should identify all VMs that were on the failed host as unavailable.\n2. Those VMs should no longer remain `Running` on the failed host.\n3. CloudStack should either:\n   - transition them to `Stopped` and restart them on an eligible host, or\n   - at minimum correct their VM/power state to reflect that they are no longer running on the fenced host.\n4. HA work items should only be marked `Done` after the VM state and placement are consistent with the actual recovery result.\n\n##### ACTUAL RESULTS\n\nThe failed host reaches `Fenced` in `ha_config`:\n\n```sql\nselect * from ha_config;\n+----+-------------+---------------+---------+------------+---------------+--------------+---------------------+-----------------+\n| id | resource_id | resource_type | enabled | ha_state   | provider      | update_count | update_time         | mgmt_server_id  |\n+----+-------------+---------------+---------+------------+---------------+--------------+---------------------+-----------------+\n|  1 |           6 | Host          |       1 | Fenced     | kvmhaprovider |          131 | 2026-03-31 06:42:57 | 248902281439561 |\n|  2 |           5 | Host          |       1 | Ineligible | kvmhaprovider |           19 | 2026-03-31 06:31:21 | 248902281439561 |\n+----+-------------+---------------+---------+------------+---------------+--------------+---------------------+-----------------+\n\n```\n\n[logs.txt](https://github.com/user-attachments/files/26370096/logs.txt)\n\nHowever, multiple VMs still remain Running on the fenced host 6:\n\n```\nSELECT id, instance_name, state, host_id, last_host_id, power_state, power_host, update_time, power_state_update_time\nFROM vm_instance\nWHERE host_id = 6\nORDER BY id;\n\n+----+---------------+---------+---------+--------------+-------------+------------+---------------------+-------------------------+\n| id | instance_name | state   | host_id | last_host_id | power_state | power_host | update_time         | power_state_update_time |\n+----+---------------+---------+---------+--------------+-------------+------------+---------------------+-------------------------+\n| 78 | r-78-VM       | Running |       6 |            6 | PowerOn     |          6 | 2026-03-31 06:19:55 | 2026-03-31 06:26:12     |\n| 80 | v-80-VM       | Running |       6 |            6 | PowerOn     |          6 | 2026-03-31 06:19:15 | 2026-03-31 06:26:12     |\n| 84 | s-84-VM       | Running |       6 |            6 | PowerOn     |          6 | 2026-03-31 06:19:14 | 2026-03-31 06:26:12     |\n| 87 | i-2-87-VM     | Running |       6 |            6 | PowerOn     |          6 | 2026-03-31 06:21:56 | 2026-03-31 06:26:12     |\n| 88 | i-2-88-VM     | Running |       6 |            6 | PowerOn     |          6 | 2026-03-31 06:21:56 | 2026-03-31 06:26:12     |\n+----+---------------+---------+---------+--------------+-------------+------------+---------------------+-------------------------+\n```\n\nAt the same time, HA work is created for those affected VMs and marked Done with reason HostDown:\n\n```\nSELECT *\nFROM op_ha_work\nORDER BY id DESC\nLIMIT 50;\n\n+----+-------------+------+--------------------+---------+-----------------+---------+---------------------+-------+---------------------+------+-------------+---------+----------+\n| id | instance_id | type | vm_type            | state   | mgmt_server_id  | host_id | created             | tried | taken               | step | time_to_try | updated | reason   |\n+----+-------------+------+--------------------+---------+-----------------+---------+---------------------+-------+---------------------+------+-------------+---------+----------+\n| 85 |          88 | HA   | User               | Running | 248902281439561 |       6 | 2026-03-31 06:31:18 |     0 | 2026-03-31 06:31:18 | Done | 1733338553  |      18 | HostDown |\n| 84 |          87 | HA   | User               | Running | 248902281439561 |       6 | 2026-03-31 06:31:18 |     0 | 2026-03-31 06:31:18 | Done | 1733338553  |      13 | HostDown |\n| 83 |          78 | HA   | DomainRouter       | Running | 248902281439561 |       6 | 2026-03-31 06:31:18 |     3 | 2026-03-31 06:31:18 | Done | 1733338553  |     100 | HostDown |\n| 82 |          80 | HA   | ConsoleProxy       | Running | 248902281439561 |       6 | 2026-03-31 06:31:18 |     0 | 2026-03-31 06:31:18 | Done | 1733338553  |      35 | HostDown |\n| 81 |          84 | HA   | SecondaryStorageVm | Running | 248902281439561 |       6 | 2026-03-31 06:31:18 |     0 | 2026-03-31 06:31:18 | Done | 1733338553  |      30 | HostDown |\n+----+-------------+------+--------------------+---------+-----------------+---------+---------------------+-------+---------------------+------+-------------+---------+----------+\n\n```\n\nThis leaves CloudStack in an inconsistent state:\n\nhost is fenced\nHA work is complete\nbut VMs are still shown as running on the failed host\nand no failover/restart occurs for those VMs\n","author":{"url":"https://github.com/TimServers","@type":"Person","name":"TimServers"},"datePublished":"2026-03-31T06:50:43.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/12922/cloudstack/issues/12922"}