René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"KVM: kvmheartbeat.sh / kvmspheartbeat.sh hardcoded sysrq reboot causes false-positive host fencing on LINSTOR/DRBD primary storage","articleBody":"### Description\n\nThe KVM agent's storage heartbeat scripts (`kvmheartbeat.sh`, `kvmspheartbeat.sh`) hard-code an immediate kernel-level reboot via `echo b \u003e /proc/sysrq-trigger` when a heartbeat write to primary storage times out. This:\n\n1. Bypasses all OS-level shutdown protections (no clean filesystem unmount, no graceful VM stop)\n2. Drops ALL running VMs on the host instantly\n3. Triggers HA cascade — surviving hosts get flooded with VM restart requests\n4. Cannot be replaced with a gentler action (restart agent, graceful reboot, log-only) without forking the script\n\n### Affected files\n\n- `scripts/vm/hypervisor/kvm/kvmheartbeat.sh` (around line 162)\n- `scripts/vm/hypervisor/kvm/kvmspheartbeat.sh` (around line 64)\n\n```sh\n# Both scripts contain:\n/usr/bin/logger -t heartbeat \"...will reboot system...\"\nsync \u0026\nsleep 5\necho b \u003e /proc/sysrq-trigger     # \u003c-- hardcoded panic-reboot\n```\n\n### Reproduction\n\n1. CloudStack 4.22.0.0 KVM hypervisor running Ubuntu 24.04\n2. Primary storage: **LINSTOR with DRBD** replication (Linstor storage pool)\n3. Trigger parallel DRBD resyncs across many resources (e.g. after a peer-node reboot, or `linstor resource resume-sync` after a maintenance pause)\n4. While resync I/O contends with normal I/O on the same disks, the heartbeat write to its `hb` file occasionally takes longer than the hardcoded timeout\n5. The host immediately force-reboots itself via sysrq, even though no actual fault exists\n\n### Real-world impact\n\nWe hit this 5+ times within 4 hours during recovery from an unrelated incident. Cascade pattern:\n\n- Heartbeat times out on host A → sysrq reboot → 90+ VMs dropped\n- HA worker reschedules those VMs onto host B\n- Host B's I/O spikes → its heartbeat times out → sysrq reboot → all its VMs dropped\n- Host A comes back, repeat\n\nEach reboot took ~3 minutes; total customer-visible outage was several hours. Patching the script line out (`echo b \u003e /proc/sysrq-trigger` → log-only) immediately stopped the cascade.\n\nThe exact log line preceding each reboot:\n\n```\nheartbeat[68685]: kvmspheartbeat.sh will reboot system because it was unable to write the heartbeat to the storage.\n```\n\n### Affected versions\n\n- CloudStack: **4.22.0.0** (cloudstack-agent / cloudstack-common 4.22.0.0)\n- Hypervisor: Ubuntu 24.04 LTS / KVM, libvirt 10.x\n- Storage: LINSTOR 1.33.2 / DRBD 9.3.1 (LINBIT public PPA)\n- Reproduces on both single-cluster and multi-cluster zones\n\n### Why this matters\n\nThe assumption \"heartbeat write timeout = host is dead\" was reasonable for **NFS shared storage** where transient I/O latency is rare and the only failure mode of concern is split-brain during a real network partition.\n\nWith **LINSTOR/DRBD** (or any local storage doing replication), the same disk serves application I/O, replication I/O, and heartbeat I/O — heartbeat can be transiently delayed without the host being dead. A fence-on-failure mechanism shouldn't be a binary panic-button: it should be **configurable**, and ideally **graceful** by default.\n\n### Existing partial workaround (poorly documented)\n\nThere is already an agent property `reboot.host.and.alert.management.on.heartbeat.timeout` (default `true`) that, when set to `false`, prevents the Java-side `KVMHAMonitor` from invoking the shell script in fence mode (cflag=1). Setting:\n\n```properties\nreboot.host.and.alert.management.on.heartbeat.timeout=false\n```\n\navoids the reboot entirely. This is an undocumented but effective workaround for the described issue.\n\nHowever, this is a binary on/off — it doesn't support intermediate fence actions (restart agent, graceful reboot, log-with-alert). Operators want a middle ground: detect the failure, *react* in a less destructive way.\n\n### Proposed enhancement\n\nAdd a finer-grained `kvm.heartbeat.fence.action` property that **supersedes** the binary boolean when set:\n\n```properties\n# Action when storage heartbeat write fails persistently\n# Values: reboot | graceful-reboot | restart-agent | log-only\n# Default: reboot (preserves current behavior for backward compatibility)\nkvm.heartbeat.fence.action=graceful-reboot\n```\n\nAction semantics:\n\n- `reboot` — current behavior (sysrq-trigger), kept as fallback for backward compat\n- `graceful-reboot` — `systemctl reboot` instead of sysrq, lets VMs stop cleanly\n- `restart-agent` — restart `cloudstack-agent` only; running VMs survive\n- `log-only` — log + alert, take no automatic action (admin investigates)\n\nExisting `reboot.host.and.alert.management.on.heartbeat.timeout=false` continues to work as a complete bypass (Java-side, never invokes the shell script in fence mode).\n\n### Workarounds available today\n\n1. **Set `reboot.host.and.alert.management.on.heartbeat.timeout=false`** in `/etc/cloudstack/agent/agent.properties` (Java-side flag — best official option until a proper fix lands)\n2. **Patch both shell scripts** to replace `echo b \u003e /proc/sysrq-trigger` with a `logger` line — survives until next package upgrade\n3. **Move heartbeat target to host-local storage** that's not under DRBD I/O contention\n\n### Willingness to contribute\n\nA follow-up PR implementing the configurable `kvm.heartbeat.fence.action` design above is in progress.\n","author":{"url":"https://github.com/jmsperu","@type":"Person","name":"jmsperu"},"datePublished":"2026-04-30T23:56:33.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/13089/cloudstack/issues/13089"}