René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Migrating VM instances fails using arm64 hosts","articleBody":"### problem\n\nMigrating systemvms (CPVM/SSVM) from arm64 local-storage based host throws this error using 4.22/RC3:\n\n```\n2025-11-02 00:24:01,001 DEBUG [c.c.a.t.Request] (AgentManager-Handler-1:[]) (logid:) Seq 233-8103101629546364942: Processing:  { Ans: , MgmtId: 104339268210986, via: 233, Ver: v1, Flags: 110, [{\"com.cloud.agent.api.Answer\":{\"result\":\"false\",\"details\":\"com.cloud.utils.exception.CloudRuntimeException: Could not fetch storage pool 63229d40-f348-4636-8707-74ab46ab28d0 from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid '63229d40-f348-4636-8707-74ab46ab28d0'\n        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:286)\n        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:272)\n        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.disconnectPhysicalDisksViaVmSpec(KVMStoragePoolManager.java:247)\n        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtPrepareForMigrationCommandWrapper.handleRollback(LibvirtPrepareForMigrationCommandWrapper.java:160)\n        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtPrepareForMigrationCommandWrapper.execute(LibvirtPrepareForMigrationCommandWrapper.java:59)\n        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtPrepareForMigrationCommandWrapper.execute(LibvirtPrepareForMigrationCommandWrapper.java:50)\n        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)\n        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2280)\n        at com.cloud.agent.Agent.processRequest(Agent.java:813)\n        at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295)\n        at com.cloud.utils.nio.Task.call(Task.java:83)\n        at com.cloud.utils.nio.Task.call(Task.java:29)\n        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n        at java.base/java.lang.Thread.run(Thread.java:840)\n\",\"wait\":\"0\",\"bypassHostMaintenance\":\"false\"}}] }\n\n```\n\nLater also causes NPE:\n```\n2025-11-02 00:24:03,278 INFO  [o.a.c.s.m.KvmNonManagedStorageDataMotionStrategy] (Work-Job-Executor-19:[ctx-b7d81f9a, job-15822/job-15823, ctx-a65e4c89]) (logid:95b78c81) Expunging dest volume [id: 1547, state: Ready] as part of failed VM migration with volumes command for VM [1161].\n2025-11-02 00:24:03,279 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-19:[ctx-b7d81f9a, job-15822/job-15823, ctx-a65e4c89]) (logid:95b78c81) Failed to copy volume java.lang.NullPointerException: Cannot invoke \"com.cloud.storage.VolumeVO.getId()\" because \"this.volumeVO\" is null\n        at org.apache.cloudstack.storage.volume.VolumeObject.stateTransit(VolumeObject.java:243)\n        at org.apache.cloudstack.storage.volume.VolumeObject.processEvent(VolumeObject.java:413)\n        at org.apache.cloudstack.storage.motion.StorageSystemDataMotionStrategy.copyAsync(StorageSystemDataMotionStrategy.java:2199)\n        at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:136)\n        at org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:2356)\n        at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1514)\n        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)\n        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n        at java.base/java.lang.reflect.Method.invoke(Method.java:569)\n```\n\n```\n2025-11-02 00:24:03,364 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-19:[ctx-b7d81f9a, job-15822/job-15823]) (logid:95b78c81) Unable to complete AsyncJob {\"accountId\":2,\"cmd\":\"com.cloud.vm.VmWorkMigrateWithStorage\",\"cmdInfo\":\"rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZVdpdGhTdG9yYWdlsew9z6UxtXMCAANKAApkZXN0SG9zdElkSgAJc3JjSG9zdElkTAAMdm9sdW1lVG9Qb29sdAAPTGphdmEvdXRpbC9NYXA7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZEoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hwAAAAAAAAAAIAAAAAAAAAAgAAAAAAAASJdAAZVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbAAAAAAAAADpAAAAAAAAALNzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYWRGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAAdwgAAAAQAAAAAHg\",\"cmdVersion\":0,\"completeMsid\":null,\"created\":\"Sun Nov 02 00:23:46 IST 2025\",\"id\":15823,\"initMsid\":104339268210986,\"instanceId\":null,\"instanceType\":null,\"lastPolled\":null,\"lastUpdated\":null,\"processStatus\":0,\"removed\":null,\"result\":null,\"resultCode\":0,\"status\":\"IN_PROGRESS\",\"userId\":2,\"uuid\":\"f4950b59-4f53-47dc-856a-5abda1a15964\"}, job origin: 15822 com.cloud.utils.exception.CloudRuntimeException: Failed to migrate VM [VM instance {\"id\":1161,\"instanceName\":\"s-1161-VM\",\"state\":\"Migrating\",\"type\":\"SecondaryStorageVm\",\"uuid\":\"ff0368d4-e71c-4e49-ad70-c7dab8ef7ea9\"}] along with its volumes due to [java.lang.NullPointerException: Cannot invoke \"com.cloud.storage.VolumeVO.getId()\" because \"this.volumeVO\" is null].\n        at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1520)\n        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)\n        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n        at java.base/java.lang.reflect.Method.invoke(Method.java:569)\n        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)\n        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)\n        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)\n        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)\n        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)\n\n```\n\nOn the target host, this exists:\n```\nroot@cloudpi:/home/rohit# virsh pool-dumpxml cc155507-f2ba-4a96-a9b9-2dac9dc50a24\n\u003cpool type='dir'\u003e\n  \u003cname\u003ecc155507-f2ba-4a96-a9b9-2dac9dc50a24\u003c/name\u003e\n  \u003cuuid\u003ecc155507-f2ba-4a96-a9b9-2dac9dc50a24\u003c/uuid\u003e\n  \u003ccapacity unit='bytes'\u003e1969186340864\u003c/capacity\u003e\n  \u003callocation unit='bytes'\u003e1526301487104\u003c/allocation\u003e\n  \u003cavailable unit='bytes'\u003e442884853760\u003c/available\u003e\n  \u003csource\u003e\n  \u003c/source\u003e\n  \u003ctarget\u003e\n    \u003cpath\u003e/var/lib/libvirt/images\u003c/path\u003e\n    \u003cpermissions\u003e\n      \u003cmode\u003e0711\u003c/mode\u003e\n      \u003cowner\u003e0\u003c/owner\u003e\n      \u003cgroup\u003e0\u003c/group\u003e\n    \u003c/permissions\u003e\n  \u003c/target\u003e\n\u003c/pool\u003e\n\n```\n\nOn the source host, this exists:\n```\nroot@kvmpi:/home/rohit# virsh pool-dumpxml 63229d40-f348-4636-8707-74ab46ab28d0\n\u003cpool type='dir'\u003e\n  \u003cname\u003e63229d40-f348-4636-8707-74ab46ab28d0\u003c/name\u003e\n  \u003cuuid\u003e63229d40-f348-4636-8707-74ab46ab28d0\u003c/uuid\u003e\n  \u003ccapacity unit='bytes'\u003e1969186340864\u003c/capacity\u003e\n  \u003callocation unit='bytes'\u003e939565473792\u003c/allocation\u003e\n  \u003cavailable unit='bytes'\u003e1029620867072\u003c/available\u003e\n  \u003csource\u003e\n  \u003c/source\u003e\n  \u003ctarget\u003e\n    \u003cpath\u003e/var/lib/libvirt/images\u003c/path\u003e\n    \u003cpermissions\u003e\n      \u003cmode\u003e0711\u003c/mode\u003e\n      \u003cowner\u003e0\u003c/owner\u003e\n      \u003cgroup\u003e0\u003c/group\u003e\n    \u003c/permissions\u003e\n  \u003c/target\u003e\n\u003c/pool\u003e\n\n```\n\n\nIn ACS DB: storage pool are defined (source is pikvm, dest is cloudpi)\n```\n*************************** 2. row ***************************\n                   id: 44\n                 name: local-pikvm-arm64\n                 uuid: 63229d40-f348-4636-8707-74ab46ab28d0\n            pool_type: Filesystem\n                 port: 0\n       data_center_id: 1\n               pod_id: 1\n           cluster_id: 13\n           used_bytes: 939565735936\n       capacity_bytes: 1969186340864\n         host_address: 10.10.1.10\n            user_info: NULL\n                 path: /var/lib/libvirt/images\n              created: 2024-07-29 15:40:15\n              removed: NULL\n          update_time: 2025-11-01 19:00:44\n               status: Up\nstorage_provider_name: DefaultPrimary\n                scope: HOST\n           hypervisor: KVM\n              managed: 0\n        capacity_iops: NULL\n               parent: 0\n            used_iops: 24\n*************************** 3. row ***************************\n                   id: 47\n                 name: local-cloudpi-arm64\n                 uuid: cc155507-f2ba-4a96-a9b9-2dac9dc50a24\n            pool_type: Filesystem\n                 port: 0\n       data_center_id: 1\n               pod_id: 1\n           cluster_id: 13\n           used_bytes: 1527497064448\n       capacity_bytes: 1969186340864\n         host_address: 10.10.1.5\n            user_info: NULL\n                 path: /var/lib/libvirt/images\n              created: 2025-08-10 11:13:27\n              removed: NULL\n          update_time: 2025-11-01 19:06:01\n               status: Up\nstorage_provider_name: DefaultPrimary\n                scope: HOST\n           hypervisor: KVM\n              managed: 0\n        capacity_iops: NULL\n               parent: 0\n            used_iops: 49\n\n```\n\n### versions\n\nACS 4.22 RC3, adv zone, arm64+x86 mix host env. Preferred arch set to arm64.\n\n### The steps to reproduce the bug\n\nSee description above\n\n### What to do about it?\n\nIdeally the local storage pool of the target host should be used in the domain/xml ","author":{"url":"https://github.com/yadvr","@type":"Person","name":"yadvr"},"datePublished":"2025-11-01T19:00:02.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":19},"url":"https://github.com/11956/cloudstack/issues/11956"}