René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"KVM NAS backup: VM remains paused indefinitely when backup job fails","articleBody":"## Description\n\nWhen using the NAS backup plugin on KVM, if a backup job fails (e.g. due to backup storage being full or I/O errors on the NFS target), the VM remains **indefinitely paused** at the hypervisor level. CloudStack marks the backup as `Error` but does not resume the VM, leaving it unresponsive until manually resumed via `virsh resume`.\n\n## Steps to Reproduce\n\n1. Configure NAS backup with NFS storage for a running KVM VM\n2. Fill up the NFS backup storage to 100% capacity\n3. Wait for the scheduled backup to trigger\n4. Observe the VM becomes paused and never resumes\n\n## Expected Behavior\n\nThe VM should be automatically resumed after a backup failure. The backup should be marked as failed, but the VM should continue running normally.\n\n## Actual Behavior\n\nThe VM remains in a `paused` state indefinitely. The backup monitoring loop in `nasbackup.sh` enters an infinite cycle:\n1. `virsh backup-begin` pauses the QEMU domain for consistent snapshot\n2. Backup write fails (storage full)\n3. `domjobinfo` reports `Failed` status\n4. `cleanup()` is called but **does not resume the VM**\n5. No `exit` statement after cleanup — loop continues, repeatedly detecting the failed job\n\n## Root Cause Analysis\n\nThree bugs in `scripts/vm/hypervisor/kvm/nasbackup.sh`:\n\n### Bug 1: Missing exit after failed backup cleanup (line 144)\n```bash\ncase \"$status\" in\n  Failed)\n    echo \"Virsh backup job failed\"\n    cleanup ;;   # \u003c-- no exit, falls through to sleep and loops forever\nesac\n```\n\n### Bug 2: cleanup() never resumes the VM (line 222)\nThe `cleanup()` function only removes files and unmounts storage. It never checks if the VM is paused or attempts to resume it, even though `virsh backup-begin` may have paused the domain.\n\n### Bug 3: Missing exit in backup_stopped_vm() (line 181)\nSimilar to Bug 1, `backup_stopped_vm()` calls `cleanup()` on `qemu-img convert` failure but does not exit, allowing the loop to continue processing subsequent disks.\n\n## Impact\n\n- **Production outage**: All services on the affected VM become unresponsive\n- **Cascading failures**: When backup storage fills up, ALL VMs being backed up get paused simultaneously\n- **Silent failure**: CloudStack UI shows the VM as \"Running\" while it is actually paused at the KVM level\n- **No automatic recovery**: Manual intervention (`virsh resume`) is required per VM\n\nIn our environment, NFS backup storage filling to 100% caused **8 production VMs** to become paused simultaneously across 3 KVM hosts, with some VMs remaining paused for over 6 hours before detection.\n\n## Environment\n\n- CloudStack 4.19/4.20/main (code is unchanged across versions)\n- KVM hypervisor\n- NAS backup plugin with NFS storage\n- File: `scripts/vm/hypervisor/kvm/nasbackup.sh`\n\n## Proposed Fix\n\nPR forthcoming with the following changes:\n1. Add VM state check and `virsh resume` to `cleanup()` function\n2. Add missing `exit 1` after `cleanup()` in the `Failed` backup job case\n3. Add missing `exit 1` after `cleanup()` in `backup_stopped_vm()` on `qemu-img convert` failure","author":{"url":"https://github.com/jmsperu","@type":"Person","name":"jmsperu"},"datePublished":"2026-03-17T03:56:12.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/12821/cloudstack/issues/12821"}