René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"XenServer / XCP-ng hypervisor resource, Storage (secondary storage mount / system VM deployment)","articleBody":"### problem\n\nOn a fresh deployment with XCP-ng 8.3 hosts, the secondary storage VM and console proxy VM never deploy. They fail and retry in a loop with ascending VM ids. The management log reports `Could not mount secondary storage \u003cnfs\u003e:/\u003cexport\u003e`, which is misleading: NFS connectivity, exports, and permissions are all fine (the share mounts manually from both the hosts and the management server without issue).\n\nThe real cause is that the host-side XAPI plugin `/etc/xapi.d/plugins/cloud-plugin-storage` fails to load on XCP-ng 8.3 because it does a top-level `import lvhdutil`, and `lvhdutil` no longer exists as an importable module on XCP-ng 8.3. Because the plugin crashes at import time, every storage operation that goes through it fails, which CloudStack surfaces as a mount failure and a \"storage unreachable\" condition.\n\nThe plugin deployed to XCP-ng 8.3 hosts is sourced from the `xenserver84` directory:\n`.../xenserver/xcpserver83/../xenserver84/cloud-plugin-storage`.\n\n### versions\n\n~~~\n4.22.0.0\nFully patched XCP-ng 8.3 servers\n~~~\n\n\n### The steps to reproduce the bug\n\nSystem VMs cycle endlessly with ascending ids. The management log shows, per attempt:\n\n~~~\nWARN  [c.c.h.x.r.XcpServer83Resource] callHostPlugin failed for cmd: mountNfsSecondaryStorage\n      with args remoteDir: \u003cnfs\u003e:/\u003cexport\u003e, localDir: /var/cloud_mount/\u003cuuid\u003e, nfsVersion: null,\n      due to Task failed!\nWARN  [c.c.h.x.r.Xenserver625StorageProcessor] ... Could not mount secondary storage \u003cnfs\u003e:/\u003cexport\u003e ...\nERROR [o.a.c.e.o.VolumeOrchestrator] Unable to create volume [ROOT-xxx] ...\nWARN  [c.c.v.ClusteredVirtualMachineManagerImpl] Resource [StoragePool:x] is unreachable ...\n~~~\n\nThe underlying XAPI task error reveals the actual failure (host-side Python traceback):\n\n~~~\nerrorInfo: [XENAPI_PLUGIN_FAILURE, non-zero exit, , Traceback (most recent call last):\n  File \"/etc/xapi.d/plugins/cloud-plugin-storage\", line 34, in \u003cmodule\u003e\n    import lvhdutil\nModuleNotFoundError: No module named 'lvhdutil'\n]\n~~~\n\nOn the XCP-ng 8.3 host, the `lvhdutil` source module is absent — only a stale orphan bytecode file remains, which Python 3 will not import:\n\n~~~\n# find /opt/xensource/sm /usr/lib/python* -name 'lvhdutil*'\n/opt/xensource/sm/__pycache__/lvhdutil.cpython-36.pyc\n\n# ls /opt/xensource/sm/ | grep -i lvhd\n(no source .py present)\n~~~\n\nNote the other imports just above `import lvhdutil` (`SR`, `VDI`, `SRCommand`, `util`, `lvutil`, `vhdutil`) succeed, which confirms `/opt/xensource/sm` is on the path and that `lvhdutil` specifically has been removed/refactored out in XCP-ng 8.3.\n\n\n### What to do about it?\n\nMake the import non-fatal in\n`/usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver84/cloud-plugin-storage`\non the management server, then push the patched plugin to the hosts (the management server only copies host plugins on host add, not on reconnect/restart, so a manual push or host re-add is required):\n\n~~~python\nimport vhdutil\nimport shutil\ntry:\n    import lvhdutil\nexcept ImportError:\n    lvhdutil = None\nimport errno\n~~~\n\nAfter this, the NFS mount succeeds and the system VMs deploy normally.\n\n##### OPEN QUESTION / SUGGESTED FIX\nThe guard above unblocks NFS and basic operation, but it does not restore any plugin code path that genuinely needs `lvhdutil` (relevant to LVM/iSCSI primary storage operations such as snapshots and template-from-volume). The proper fix is likely to import `lvhdutil` from its new location in XCP-ng 8.3's refactored `sm` stack (or call its replacement), rather than guarding it to `None`. Guidance on the correct module/path for 8.3 would be appreciated.\n\n##### RELATED\nPossibly related to discussion #11346 (4.20.1 + XCP-ng 8.3 system VMs failing to deploy), though that case was resolved as an NFSv4 negotiation issue on the storage server — this is a distinct root cause (plugin import failure).\n","author":{"url":"https://github.com/BannerSmash","@type":"Person","name":"BannerSmash"},"datePublished":"2026-05-29T20:22:15.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":7},"url":"https://github.com/13292/cloudstack/issues/13292"}