René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"FirstFitRouting Allocator only tries the first cluster it finds then fails before attempting others","articleBody":"Our CS environment has two clusters, one windows and one linux. We have host tags that allocate where VM's are placed. We are facing an issue when setting host in maintenance that sometimes it will just fall to error state. Upon checking the logs I found that while it checked the first cluster (ID 19, Windows) however it never tried any hosts in the Linux cluster (ID 16).\n\nHere is the log from the management server. As we can see, the allocator tried to find resources in cluster 19, failed to find any with the Linux tag, then failed out before trying the other cluster.\n\n```\n2025-03-20 20:04:13,017 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388]) (logid:30d64f64) Run VM work job: com.cloud.vm.VmWorkMigrateAway for VM 8127, job origin: 3900296\n2025-03-20 20:04:13,018 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975]) (logid:30d64f64) Execute VM work job: com.cloud.vm.VmWorkMigrateAway{\"srcHostId\":982,\"userId\":1,\"accountId\":1,\"vmId\":8127,\"handlerName\":\"VirtualMachineManagerImpl\"}\n2025-03-20 20:04:13,031 INFO  [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975]) (logid:30d64f64) Searching for hosts in the zone for vm migration\n2025-03-20 20:04:13,034 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975]) (logid:30d64f64) Trying to deploy VM [error decoding VM instance {\"id\":8127,\"instanceName\":\"i-10989-8127-VM\",\"type\":\"User\",\"uuid\":\"ce0c1fb6-65ef-4942-97de-05b255fdcd7d\"}] and details: Plan [{\"_dcId\":14,\"_poolId\":637,\"_recreateDisks\":false,\"preferredHostIds\":[],\"migrationPlan\":true,\"hostPriorities\":{}}]; avoid list [{\"_clusterIds\":[24],\"_hostIds\":[982]}] and planner: [].\n2025-03-20 20:04:13,037 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975]) (logid:30d64f64) Checking non dedicated resources to deploy VM [{\"instanceName\":\"i-10989-8127-VM\",\"type\":\"User\",\"uuid\":\"ce0c1fb6-65ef-4942-97de-05b255fdcd7d\"}].\n2025-03-20 20:04:13,046 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975]) (logid:30d64f64) Adding pods [[]], clusters [[]] and hosts [[]] to the avoid list in the deploy process of user VM [\"ce0c1fb6-65ef-4942-97de-05b255fdcd7d\"], because this VM is not explicitly dedicated to these components.\n2025-03-20 20:04:13,048 DEBUG [o.a.c.u.r.ReflectionToStringBuilderUtils] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975]) (logid:30d64f64) Object is null, not reflecting it.\n2025-03-20 20:04:13,049 DEBUG [o.a.c.u.r.ReflectionToStringBuilderUtils] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975]) (logid:30d64f64) Object is null, not reflecting it.\n2025-03-20 20:04:13,049 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975]) (logid:30d64f64) Trying to allocate a host and storage pools from datacenter [{\"name\":\"CS12\",\"uuid\":\"81e07cf1-e395-4f5e-aa7e-4e9d80aa3cfb\"}], pod [null], cluster [null], to deploy VM [{\"instanceName\":\"i-10989-8127-VM\",\"type\":\"User\",\"uuid\":\"ce0c1fb6-65ef-4942-97de-05b255fdcd7d\"}] with requested CPU [24000] and requested RAM [(32.00 GB) 34359738368].\n2025-03-20 20:04:13,050 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975]) (logid:30d64f64) ROOT volume [69d114a8-5673-4ad7-a387-2ac0d11cf3a1] is ready to deploy VM [ce0c1fb6-65ef-4942-97de-05b255fdcd7d].\n2025-03-20 20:04:13,054 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975]) (logid:30d64f64) Adding pods [] to the avoid set because these pods are in the Disabled state.\n2025-03-20 20:04:13,057 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975]) (logid:30d64f64) Adding clusters [] of pod [5] to the void set because these clusters are in the Disabled state.\n2025-03-20 20:04:13,060 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975]) (logid:30d64f64) Adding hosts [] of datacenter [81e07cf1-e395-4f5e-aa7e-4e9d80aa3cfb] to the avoid set, because these hosts are in the Disabled state.\n2025-03-20 20:04:13,065 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975]) (logid:30d64f64) DeploymentPlan [DataCenterDeployment] has not specified host. Trying to find another destination to deploy VM [ce0c1fb6-65ef-4942-97de-05b255fdcd7d], avoiding pods [], clusters [24] and hosts [982].\n2025-03-20 20:04:13,065 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975]) (logid:30d64f64) Deploy avoids pods: [], clusters: [24], hosts: [982].\n2025-03-20 20:04:13,065 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975]) (logid:30d64f64) Deploy hosts with priorities {}, hosts have NORMAL priority by default\n2025-03-20 20:04:13,066 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975]) (logid:30d64f64) This VM has last host_id specified, trying to choose the same host: 982\n2025-03-20 20:04:13,069 WARN  [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975]) (logid:30d64f64) The last host [bf145201-bcab-4634-8750-36b064574bc7] of VM [ce0c1fb6-65ef-4942-97de-05b255fdcd7d] is in the avoid set. Skipping this and trying other available hosts.\n2025-03-20 20:04:13,069 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975]) (logid:30d64f64) Cannot choose the last host to deploy this VM VM instance {\"id\":8127,\"instanceName\":\"i-10989-8127-VM\",\"type\":\"User\",\"uuid\":\"ce0c1fb6-65ef-4942-97de-05b255fdcd7d\"}.\n2025-03-20 20:04:13,072 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975]) (logid:30d64f64) Searching all possible resources under this Zone: 14\n2025-03-20 20:04:13,076 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975]) (logid:30d64f64) Listing clusters in order of aggregate capacity, that have (at least one host with) enough CPU and RAM capacity under this Zone: 14\n2025-03-20 20:04:13,084 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975]) (logid:30d64f64) Removing from the clusterId list these clusters from avoid set: [24]\n2025-03-20 20:04:13,102 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975]) (logid:30d64f64) The clusterId list for the given offering tag: [19, 16]\n2025-03-20 20:04:13,106 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975]) (logid:30d64f64) Checking resources in Cluster: 19 under Pod: 5\n2025-03-20 20:04:13,111 INFO  [c.c.a.m.a.i.FirstFitRoutingAllocator] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975, FirstFitRoutingAllocator]) (logid:30d64f64)  Guest VM is requested with Custom[UEFI] Boot Type false\n2025-03-20 20:04:13,111 DEBUG [c.c.a.m.a.i.FirstFitRoutingAllocator] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975, FirstFitRoutingAllocator]) (logid:30d64f64) Looking for hosts in zone [14], pod [5], cluster [19]\n2025-03-20 20:04:13,111 DEBUG [c.c.a.m.a.i.FirstFitRoutingAllocator] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975, FirstFitRoutingAllocator]) (logid:30d64f64) Looking for hosts having tag specified on SvcOffering:generic\n2025-03-20 20:04:13,118 DEBUG [c.c.a.m.a.i.FirstFitRoutingAllocator] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975, FirstFitRoutingAllocator]) (logid:30d64f64) Hosts with tag 'generic' are:[Host {\"id\":919,\"name\":\"node87\",\"type\":\"Routing\",\"uuid\":\"e5668da2-862f-4cea-bda8-20707525ecda\"}, Host {\"id\":1126,\"name\":\"node51\",\"type\":\"Routing\",\"uuid\":\"be3eb6a2-29b3-4a1e-a125-af8f9135d3a6\"}, Host {\"id\":1129,\"name\":\"node52\",\"type\":\"Routing\",\"uuid\":\"a803eaea-82a1-4c10-a4d2-b0db6c8c15c6\"}, Host {\"id\":1350,\"name\":\"node84\",\"type\":\"Routing\",\"uuid\":\"1a006517-3a4b-4a98-85a2-f693fa4f4501\"}, Host {\"id\":1368,\"name\":\"node50\",\"type\":\"Routing\",\"uuid\":\"bc93e3f6-3751-46fd-8dee-c24f6e8fc922\"}]\n2025-03-20 20:04:13,118 DEBUG [c.c.a.m.a.i.FirstFitRoutingAllocator] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975, FirstFitRoutingAllocator]) (logid:30d64f64) Looking for hosts having tag specified on Template:linux\n2025-03-20 20:04:13,123 DEBUG [c.c.a.m.a.i.FirstFitRoutingAllocator] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975, FirstFitRoutingAllocator]) (logid:30d64f64) Hosts with tag 'linux' are:[]\n2025-03-20 20:04:13,124 DEBUG [c.c.a.m.a.i.FirstFitRoutingAllocator] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975, FirstFitRoutingAllocator]) (logid:30d64f64) Found 0 Hosts satisfying both tags, host ids are:[]\n2025-03-20 20:04:13,126 ERROR [c.c.a.m.a.i.FirstFitRoutingAllocator] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975, FirstFitRoutingAllocator]) (logid:30d64f64) No suitable host found for vm [VM instance {\"id\":8127,\"instanceName\":\"i-10989-8127-VM\",\"type\":\"User\",\"uuid\":\"ce0c1fb6-65ef-4942-97de-05b255fdcd7d\"}] with tags [generic].\n2025-03-20 20:04:13,127 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975]) (logid:30d64f64) Invocation exception, caused by: com.cloud.utils.exception.CloudRuntimeException: No suitable host found for vm [VM instance {\"id\":8127,\"instanceName\":\"i-10989-8127-VM\",\"type\":\"User\",\"uuid\":\"ce0c1fb6-65ef-4942-97de-05b255fdcd7d\"}].\n2025-03-20 20:04:13,127 INFO  [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388, ctx-e309f975]) (logid:30d64f64) Rethrow exception com.cloud.utils.exception.CloudRuntimeException: No suitable host found for vm [VM instance {\"id\":8127,\"instanceName\":\"i-10989-8127-VM\",\"type\":\"User\",\"uuid\":\"ce0c1fb6-65ef-4942-97de-05b255fdcd7d\"}].\n2025-03-20 20:04:13,127 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388]) (logid:30d64f64) Done with run of VM work job: com.cloud.vm.VmWorkMigrateAway for VM 8127, job origin: 3900296\n2025-03-20 20:04:13,127 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388]) (logid:30d64f64) Unable to complete AsyncJobVO: {id:3901388, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkMigrateAway, cmdInfo: rO0ABXNyAB5jb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZUF3YXmt4MX4jtcEmwIAAUoACXNyY0hvc3RJZHhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAfv3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAD1g, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 95539959339, completeMsid: null, lastUpdated: null, lastPolled: null, created: Thu Mar 20 20:04:12 UTC 2025, removed: null}, job origin: 3900296 com.cloud.utils.exception.CloudRuntimeException: No suitable host found for vm [VM instance {\"id\":8127,\"instanceName\":\"i-10989-8127-VM\",\"type\":\"User\",\"uuid\":\"ce0c1fb6-65ef-4942-97de-05b255fdcd7d\"}].\n2025-03-20 20:04:13,128 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-145:[ctx-2ec22277, job-3900296/job-3901388]) (logid:30d64f64) Complete async job-3901388, jobStatus: FAILED, resultCode: 0\n```\n\nI also checked the UI upon doing a manual migration, all windows hosts are invalid destinations in the options to select a host (which I believe findHostsForMigration uses, so that should be working fine)\n\n### versions\n\nACS 4.20, KVM with NFS Primary/Secondary storage\n\n### The steps to reproduce the bug\n\nSetup two clusters, with host tags to identify where VM's should go (windows/linux).\nTry to set maintenance on one of the hosts.\nCheck for clusters in the log to see if it attempts others.\n\n\n### What to do about it?\n\nCorrect the code to try all clusters instead of just the first.","author":{"url":"https://github.com/loth","@type":"Person","name":"loth"},"datePublished":"2025-03-20T21:16:30.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":9},"url":"https://github.com/10598/cloudstack/issues/10598"}