René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"CIFS NAS Backups failing \"randomly\"","articleBody":"### problem\n\nIt seems that in my current conditions, the backups are failing because of a CIFS warning that is failing the Backup even if the backup was indeed successful\n\n```\n2025-11-24 11:05:32,938 ERROR [o.a.c.b.NASBackupProvider] (API-Job-Executor-11:[ctx-673f730c, job-3216, ctx-2492b04d]) (logid:2a7df0d3) Failed to take backup for VM i-12-606-VM: Job type:         Completed   \nOperation:        Backup      \nTime elapsed:     29235        ms\nFile processed:   23.000 GiB\nFile remaining:   0.000 B\nFile total:       23.000 GiB\n\n2769033951\n2025-11-24 11:05:32,946 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-11:[ctx-673f730c, job-3216]) (logid:2a7df0d3) Complete async job-3216, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{\"uuidList\":[],\"errorcode\":\"530\",\"errortext\":\"Failed to create VM backup\"}\n```\n\nI tried downgrading from smb 3.1.1 to 3.0, but it didn't fixed the issue, here is my full mount options :\n`//XXXX.XXXX/xxxxxx on /tmp/csbackup.YDH0M type cifs (rw,relatime,vers=3.0,cache=strict,upcall_target=app,username=xxxxxxx,uid=0,noforceuid,gid=0,noforcegid,addr=78.46.12.119,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,nobrl,reparse=nfs,nativesocket,symlink=native,rsize=4194304,wsize=4194304,bsize=1048576,retrans=1,echo_interval=60,actimeo=1,closetimeo=1)`\n\n\n### versions\n\n- CloudStack 4.22.0.0\n```\n[root@compute01 ~]# uname -a\nLinux compute01 5.14.0-570.17.1.el9_6.x86_64 #1 SMP PREEMPT_DYNAMIC Fri May 23 22:47:01 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux\n```\n```\n[root@compute01 ~]# cat /etc/os-release \nNAME=\"Rocky Linux\"\nVERSION=\"9.6 (Blue Onyx)\"\n```\n```\n[root@compute01 ~]# ls -lah /tmp/csbackup.YDH0M/i-12-606-VM/2025.11.24.11.04.56/\ntotal 1.4G\ndrwxr-xr-x. 2 root root    0 Nov 24 11:04 .\ndrwxr-xr-x. 2 root root    0 Nov 24 11:03 ..\n-rwxr-xr-x. 1 root root 2.8M Nov 24 11:03 datadisk.96c4484b-2b51-46bd-9476-c7c9e0cedb87.qcow2\n-rwxr-xr-x. 1 root root 5.7K Nov 24 11:04 domain-config.xml\n-rwxr-xr-x. 1 root root  299 Nov 24 11:05 domblklist.xml\n-rwxr-xr-x. 1 root root  166 Nov 24 11:04 domiflist.xml\n-rwxr-xr-x. 1 root root  620 Nov 24 11:04 dominfo.xml\n-rwxr-xr-x. 1 root root 2.6G Nov 24 11:04 root.98ee2a4e-8c87-4a2e-8b61-aac596fffebd.qcow2\n```\n\n### The steps to reproduce the bug\n\n1. Configure the NAS Backup plugin\n2. Add a CIFS (SMB) backup repository\n3. Try to do multiple backups (Sometimes it fails, sometimes it doesnt)\n\n\n### What to do about it?\n\nhttps://github.com/apache/cloudstack/blob/8171d9568c941877e0290f77051ed45f31a73544/scripts/vm/hypervisor/kvm/nasbackup.sh#L151\n\nIt seems that the CIFS mount doesn't really treat `sync` as it should in my current setup, and stays busy even after syncing :\n```\n[root@compute01 ~]# /usr/bin/bash /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/nasbackup.sh -o backup -v i-12-606-VM -t cifs -s '//XXXXXXXX.xxxxxxxx/xxxxxxxx' -m 'vers=3.0,username=xxxxxxx,password=xxxxxxxx' -p 'i-12-606-VM/2025.11.24.12.02.09' -q false -d ''\nJob type:         Completed   \nOperation:        Backup      \nTime elapsed:     27079        ms\nFile processed:   23.000 GiB\nFile remaining:   0.000 B\nFile total:       23.000 GiB\n\n2769033951\numount: /tmp/csbackup.rnf14: target is busy.\n\n```\n\nWe could avoid this issue by checking if the mountpoint is still busy, for example with fuser (more available than lsof) :\n```bash\n  # Print statistics\n  virsh -c qemu:///system domjobinfo $VM --completed\n  du -sb $dest | cut -f1\n\n  elapsed=0\n  while fuser -m \"$mount_point\" \u003e/dev/null 2\u003e\u00261; do\n    if (( elapsed \u003e= 10 )); then\n      echo \"Timeout for unmounting reached: still busy\"\n      exit 1\n    fi\n    sleep 1\n    elapsed=$((elapsed + 1))\n  done\n\n  umount $mount_point\n  rmdir $mount_point\n```\n\nAlso when i checked the code for the script, it seems that the backup_stopped_vm is not umounting and removing the mount point, i don't know if that's on purpose.","author":{"url":"https://github.com/Hanarion","@type":"Person","name":"Hanarion"},"datePublished":"2025-11-24T11:23:05.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":7},"url":"https://github.com/12122/cloudstack/issues/12122"}