René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Cannot cancel maintenance mode of a primary storage v4.21","articleBody":"### problem\n\nZone-wide storage:\nIf there are virtual machines on specific primary storage and that storage is set to Maintenance mode it is not possible to cancel the maintenance mode if the agents in the zone were restarted.\nCluster-wide storage pool:\nIf there are virtual machines on such pool, and the pool is set to maintenance and after that if we try to cancel the maintenance mode it's not possible\n\n### versions\n\n4.21\nZone-wide primary storage - StorPool and NFS\nCluster-wide primary storage - NFS\n\n### The steps to reproduce the bug\n\nAll VMs are created with service offering that have storage tags. I haven't tested it without tags\nZone-wide pool:\n- deploy a VM\n- enable maintenance mode of the storage pool\n- force reconnect all host in the zone from the UI (or restart the cloudstack-agent service)\n- cancle maintenance mode of the storage pool\n\nCluster-wide\n- deploy a VM\n- enable maintenance mode\n- cancle maintenance mode\n\n```\n2025-09-30 14:09:42,578 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) Found pools [[]] that match with tags [[nfs-c\nluster2]].\n2025-09-30 14:09:42,578 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) No storage pools available for [shared] volum\ne allocation.\n2025-09-30 14:09:42,578 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) Using volume allocation algorithm random to r\neorder pools.\n2025-09-30 14:09:42,578 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) Trying to find suitable pools to allocate volume [Vo\nlume {\"id\":562,\"instanceId\":380,\"name\":\"ROOT-380\",\"uuid\":\"bb7f3c0b-19f8-4be1-a6ce-bf29cff9a0ff\",\"volumeType\":\"ROOT\"}] necessary to deploy VM [VM instance {\"id\":380,\"instanceName\":\"i-2-380-VM\",\"state\":\"Startin\ng\",\"type\":\"User\",\"uuid\":\"f4f66160-a169-4b4a-8729-df528c516c33\"}], using StoragePoolAllocator: [ZoneWideStoragePoolAllocator].\n2025-09-30 14:09:42,580 DEBUG [o.a.c.s.a.ZoneWideStoragePoolAllocator] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) Could not find any zone wide storage pool that ma\ntched with any of the following tags [[nfs-cluster2]].\n2025-09-30 14:09:42,581 DEBUG [o.a.c.s.a.ZoneWideStoragePoolAllocator] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) ZoneWideStoragePoolAllocator is returning [0] sui\ntable storage pools [[]].\n2025-09-30 14:09:42,581 DEBUG [o.a.c.s.a.ZoneWideStoragePoolAllocator] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) Using volume allocation algorithm random to reord\ner pools.\n2025-09-30 14:09:42,581 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) No suitable pools found for volume [Volume {\"id\":562\n,\"instanceId\":380,\"name\":\"ROOT-380\",\"uuid\":\"bb7f3c0b-19f8-4be1-a6ce-bf29cff9a0ff\",\"volumeType\":\"ROOT\"}] used by VM [VM instance {\"id\":380,\"instanceName\":\"i-2-380-VM\",\"state\":\"Starting\",\"type\":\"User\",\"uuid\":\"f\n4f66160-a169-4b4a-8729-df528c516c33\"}] under cluster: [3].\n2025-09-30 14:09:42,581 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) No suitable pools found\n2025-09-30 14:09:42,581 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) No suitable storagePools found under this Cluster: C\nluster {id: \"3\", name: \"Cluster-A-2\", uuid: \"bd8f0dce-1699-4879-8804-04713a1a87ee\"}\n2025-09-30 14:09:42,583 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) Could not find suitable Deployment Destination for t\nhis VM under any clusters, returning. \n2025-09-30 14:09:42,584 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) Searching resources only under specified Pod: HostPod {\"id\":1,\"nam\ne\":\"Pod-A\",\"uuid\":\"f1cb86a7-ea4f-4211-a104-dd501578d7e4\"}\n2025-09-30 14:09:42,584 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) Listing clusters in order of aggregate capacity, that have (at lea\nst one host with) enough CPU and RAM capacity under this Pod: 1\n2025-09-30 14:09:42,585 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) CapacityType: CPU is used for Cluster ordering\n2025-09-30 14:09:42,586 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) Removing from the clusterId list these clusters from avoid set: [1\n, 3]\n2025-09-30 14:09:42,587 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) No clusters found after removing disabled clusters and clusters in\n avoid list, returning.\n2025-09-30 14:09:42,594 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) VM instance {\"id\":380,\"instanceName\":\"i-2-380-VM\",\"state\":\"Sto\npped\",\"type\":\"User\",\"uuid\":\"f4f66160-a169-4b4a-8729-df528c516c33\"} state transited from [Starting] to [Stopped] with event [OperationFailed]. VM's original host: Host {\"id\":7,\"name\":\"cs-dev-kvm-3-4\",\"type\":\"R\nouting\",\"uuid\":\"e078cdb7-8819-43b2-a2c9-262788ad6099\"}, new host: null, host before state transition: null\n2025-09-30 14:09:42,598 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) Invocation exception, caused by: com.cloud.exception.Insuffi\ncientServerCapacityException: Unable to create a deployment for VM instance {\"id\":380,\"instanceName\":\"i-2-380-VM\",\"state\":\"Starting\",\"type\":\"User\",\"uuid\":\"f4f66160-a169-4b4a-8729-df528c516c33\"}Scope=interface com.cloud.dc.DataCenter; id=1\n2025-09-30 14:09:42,599 INFO  [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445, ctx-59178917]) (logid:51ee9166) Rethrow exception com.cloud.exception.InsufficientServerCapa\ncityException: Unable to create a deployment for VM instance {\"id\":380,\"instanceName\":\"i-2-380-VM\",\"state\":\"Starting\",\"type\":\"User\",\"uuid\":\"f4f66160-a169-4b4a-8729-df528c516c33\"}Scope=interface com.cloud.dc.D\nataCenter; id=1\n2025-09-30 14:09:42,599 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445]) (logid:51ee9166) Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 380, job origi\nn: 5444\n2025-09-30 14:09:42,599 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-15:[ctx-6b4bcd04, job-5444/job-5445]) (logid:51ee9166) Unable to complete AsyncJob {\"accountId\":2,\"cmd\":\"com.cloud.vm.VmWorkStart\",\n\"cmdInfo\":\"rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zd\nElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkc\nQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAABfHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwc\nHA\",\"cmdVersion\":0,\"completeMsid\":null,\"created\":\"Tue Sep 30 14:09:41 UTC 2025\",\"id\":5445,\"initMsid\":2199191191555,\"instanceId\":null,\"instanceType\":null,\"lastPolled\":null,\"lastUpdated\":null,\"processStatus\":0,\n\"removed\":null,\"result\":null,\"resultCode\":0,\"status\":\"IN_PROGRESS\",\"userId\":2,\"uuid\":\"68e14887-3529-4f40-a73d-1960007af720\"}, job origin: 5444 com.cloud.exception.InsufficientServerCapacityException: Unable t\no create a deployment for VM instance {\"id\":380,\"instanceName\":\"i-2-380-VM\",\"state\":\"Starting\",\"type\":\"User\",\"uuid\":\"f4f66160-a169-4b4a-8729-df528c516c33\"}Scope=interface com.cloud.dc.DataCenter; id=1\n        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1451)\n```\n\n\n### What to do about it?\n\n_No response_","author":{"url":"https://github.com/slavkap","@type":"Person","name":"slavkap"},"datePublished":"2025-09-30T14:38:08.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":14},"url":"https://github.com/11764/cloudstack/issues/11764"}