René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"systemvms do not come up on vmware","articleBody":"When deploying cloudstack on VMware, systemvms are unable to come up and are continually redeployed. This appears to be due to a null pointer exception.\r\nWhen it is build on commit `dc225de8115a5f03b7eb8959c7e726943dfb2be8`, it all works fine and the systemvms come up as usual\r\n\r\n\r\n##### ISSUE TYPE\r\n * Bug Report\r\n\r\n##### COMPONENT NAME\r\nVmwareResource\r\n\r\n##### CLOUDSTACK VERSION\r\nmaster (4.14)\r\n\r\n##### CONFIGURATION\r\nJust on bringing up the management server for the first time\r\n\r\n\r\n##### OS / ENVIRONMENT\r\nIdentified on vmware 65\r\n\r\n\r\n##### SUMMARY\r\nThe systemvms do not come up but get endlessly recreated, \r\n\r\n\r\n##### STEPS TO REPRODUCE\r\nJust build off master and deploy on VMware\r\n\r\n##### EXPECTED RESULTS\r\nThe systemvms come up successfully\r\n\r\n\r\n##### ACTUAL RESULTS\r\nThe systemvms fail to come up and the following log errors\r\n~~~\r\n2020-03-18 09:23:49,203 INFO  [c.c.h.v.r.VmwareResource] (DirectAgent-22:ctx-4d5ca7b7 10.2.3.170, job-38/job-44, cmd: StartCommand) (logid:69db3762) Executing resource StartCommand: {\"vm\":{\"id\":6,\"name\":\"v-6-VM\",\"bootloader\":\"HVM\",\"state\":\"Starting\",\"type\":\"ConsoleProxy\",\"cpus\":1,\"minSpeed\":250,\"maxSpeed\":500,\"minRam\":1073741824,\"maxRam\":1073741824,\"hostName\":\"v-6-VM\",\"arch\":\"x86_64\",\"os\":\"Other Linux (64-bit)\",\"platformEmulator\":\"otherLinux64Guest\",\"bootArgs\":\" template=domP type=consoleproxy host=10.2.3.243 port=8250 name=v-6-VM premium=true zone=1 pod=1 guid=Proxy.6 proxy_vm=6 disable_rp_filter=true eth2ip=10.1.36.162 eth2mask=255.255.224.0 gateway=10.1.63.254 eth0ip=0.0.0.0 eth0mask=0.0.0.0 eth1ip=10.2.8.128 eth1mask=255.255.0.0 mgmtcidr=10.2.0.0/16 localgw=10.2.254.254 internaldns1=8.8.8.8 internaldns2=8.8.4.4 dns1=8.8.8.8 dns2=8.8.4.4\",\"enableHA\":false,\"limitCpuUse\":false,\"enableDynamicallyScaleVm\":false,\"vncPassword\":\"z2udCq7b4hpXI87OGEwccA\",\"params\":{\"cpuOvercommitRatio\":\"2.0\",\"vmware.reserve.mem\":\"false\",\"vmware.reserve.cpu\":\"false\",\"memoryOvercommitRatio\":\"1.0\",\"nicAdapter\":\"E1000\"},\"uuid\":\"38badf91-b9cf-4fdd-88a8-3fd94add7a5e\",\"disks\":[{\"data\":{\"org.apache.cloudstack.storage.to.VolumeObjectTO\":{\"uuid\":\"5aac8e4b-156c-4382-97bb-7e832c2d5a46\",\"volumeType\":\"ROOT\",\"dataStore\":{\"org.apache.cloudstack.storage.to.PrimaryDataStoreTO\":{\"uuid\":\"d8d96070-6f33-3574-ac36-ed3f3349b304\",\"id\":2,\"poolType\":\"NetworkFilesystem\",\"host\":\"10.2.0.16\",\"path\":\"/acs/primary/trl-221-v-M7-djumani/trl-221-v-M7-djumani-esxi-pri2\",\"port\":2049,\"url\":\"NetworkFilesystem://10.2.0.16/acs/primary/trl-221-v-M7-djumani/trl-221-v-M7-djumani-esxi-pri2/?ROLE=Primary\u0026STOREUUID=d8d96070-6f33-3574-ac36-ed3f3349b304\",\"isManaged\":false}},\"name\":\"ROOT-6\",\"size\":0,\"path\":\"ROOT-6\",\"volumeId\":6,\"vmName\":\"v-6-VM\",\"accountId\":1,\"format\":\"OVA\",\"provisioningType\":\"THIN\",\"id\":6,\"deviceId\":0,\"bytesReadRate\":0,\"bytesWriteRate\":0,\"iopsReadRate\":0,\"iopsWriteRate\":0,\"hypervisorType\":\"VMware\",\"directDownload\":false}},\"diskSeq\":0,\"path\":\"ROOT-6\",\"type\":\"ROOT\",\"_details\":{\"storageHost\":\"10.2.0.16\",\"managed\":\"false\",\"storagePort\":\"2049\",\"volumeSize\":\"0\"}}],\"nics\":[{\"deviceId\":2,\"networkRateMbps\":-1,\"defaultNic\":true,\"pxeDisable\":true,\"nicUuid\":\"f22aeb91-8721-4b3d-80c7-6812041717dd\",\"details\":{\"MacAddressChanges\":\"true\",\"PromiscuousMode\":\"false\",\"ForgedTransmits\":\"true\"},\"dpdkEnabled\":false,\"uuid\":\"f1e62679-7d86-43d5-95e5-12e2ffcead7f\",\"ip\":\"10.1.36.162\",\"netmask\":\"255.255.224.0\",\"gateway\":\"10.1.63.254\",\"mac\":\"1e:00:36:00:00:02\",\"dns1\":\"8.8.8.8\",\"dns2\":\"8.8.4.4\",\"broadcastType\":\"Vlan\",\"type\":\"Public\",\"broadcastUri\":\"vlan://7\",\"isolationUri\":\"vlan://7\",\"isSecurityGroupEnabled\":false,\"name\":\"vSwitch1,,vmwaresvs\"},{\"deviceId\":0,\"networkRateMbps\":-1,\"defaultNic\":false,\"pxeDisable\":true,\"nicUuid\":\"1094b7d3-4be5-424a-ae88-1c4da8b8e39d\",\"details\":{\"MacAddressChanges\":\"true\",\"PromiscuousMode\":\"false\",\"ForgedTransmits\":\"true\"},\"dpdkEnabled\":false,\"uuid\":\"b6614297-1f5b-4030-869e-60284abf0ce3\",\"mac\":\"02:00:5a:ad:00:06\",\"broadcastType\":\"LinkLocal\",\"type\":\"Control\",\"isSecurityGroupEnabled\":false,\"name\":\"vSwitch0,,vmwaresvs\"},{\"deviceId\":1,\"networkRateMbps\":-1,\"defaultNic\":false,\"pxeDisable\":true,\"nicUuid\":\"f978470b-9f73-4f7e-ba9f-5aa3477b1fba\",\"details\":{\"MacAddressChanges\":\"true\",\"PromiscuousMode\":\"false\",\"ForgedTransmits\":\"true\"},\"dpdkEnabled\":false,\"uuid\":\"b55693c5-f807-464e-80d3-a0224f1dc930\",\"ip\":\"10.2.8.128\",\"netmask\":\"255.255.0.0\",\"gateway\":\"10.2.254.254\",\"mac\":\"1e:00:05:00:00:1c\",\"broadcastType\":\"Native\",\"type\":\"Management\",\"isSecurityGroupEnabled\":false,\"name\":\"vSwitch0,,vmwaresvs\"}],\"guestOsDetails\":{},\"extraConfig\":{}},\"hostIp\":\"10.2.3.170\",\"executeInSequence\":false,\"wait\":0}\r\n2020-03-18 09:23:49,342 DEBUG [c.c.h.v.r.VmwareResource] (DirectAgent-22:ctx-4d5ca7b7 10.2.3.170, job-38/job-44, cmd: StartCommand) (logid:69db3762) Using mapping name : OTHER_LINUX_64_GUEST\r\n2020-03-18 09:23:49,420 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-22:ctx-4d5ca7b7 10.2.3.170, job-38/job-44, cmd: StartCommand) (logid:69db3762) find VM v-6-VM on host\r\n2020-03-18 09:23:49,420 INFO  [c.c.h.v.m.HostMO] (DirectAgent-22:ctx-4d5ca7b7 10.2.3.170, job-38/job-44, cmd: StartCommand) (logid:69db3762) VM v-6-VM not found in host cache\r\n2020-03-18 09:23:49,420 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-22:ctx-4d5ca7b7 10.2.3.170, job-38/job-44, cmd: StartCommand) (logid:69db3762) load VM cache on host\r\n2020-03-18 09:23:49,613 INFO  [c.c.h.v.r.VmwareResource] (DirectAgent-22:ctx-4d5ca7b7 10.2.3.170, job-38/job-44, cmd: StartCommand) (logid:69db3762) Found vm v-6-VM at other host, relocate to 10.2.3.170\r\n2020-03-18 09:23:50,712 DEBUG [c.c.h.v.m.HypervisorHostHelper] (DirectAgent-37:ctx-0d022993 10.2.3.147, job-37/job-43, cmd: StartCommand) (logid:bd8aa2d5) Prepare network on other host, vlan: 7, host: 10.2.3.170\r\n2020-03-18 09:23:50,936 INFO  [c.c.h.v.m.HypervisorHostHelper] (DirectAgent-37:ctx-0d022993 10.2.3.147, job-37/job-43, cmd: StartCommand) (logid:bd8aa2d5) Network cloud.public.7.0.1-vSwitch1 is ready on vSwitch vSwitch1\r\n2020-03-18 09:23:50,936 INFO  [c.c.h.v.m.HypervisorHostHelper] (DirectAgent-37:ctx-0d022993 10.2.3.147, job-37/job-43, cmd: StartCommand) (logid:bd8aa2d5) Network cloud.public.7.0.1-vSwitch1 is ready on vSwitch vSwitch1\r\n2020-03-18 09:23:50,936 INFO  [c.c.h.v.r.VmwareResource] (DirectAgent-37:ctx-0d022993 10.2.3.147, job-37/job-43, cmd: StartCommand) (logid:bd8aa2d5) Preparing NIC device on network cloud.public.7.0.1-vSwitch1\r\n2020-03-18 09:23:50,936 DEBUG [c.c.h.v.r.VmwareResource] (DirectAgent-37:ctx-0d022993 10.2.3.147, job-37/job-43, cmd: StartCommand) (logid:bd8aa2d5) Prepare NIC at new device {\"operation\":\"ADD\",\"device\":{\"addressType\":\"Manual\",\"macAddress\":\"1e:00:4f:00:00:01\",\"key\":-5,\"backing\":{\"network\":{\"value\":\"network-63\",\"type\":\"Network\"},\"deviceName\":\"cloud.public.7.0.1-vSwitch1\"},\"connectable\":{\"startConnected\":true,\"allowGuestControl\":true,\"connected\":true}}}\r\n2020-03-18 09:23:50,936 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-37:ctx-0d022993 10.2.3.147, job-37/job-43, cmd: StartCommand) (logid:bd8aa2d5) find VM s-5-VM on host\r\n2020-03-18 09:23:50,936 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-37:ctx-0d022993 10.2.3.147, job-37/job-43, cmd: StartCommand) (logid:bd8aa2d5) VM s-5-VM found in host cache\r\n2020-03-18 09:23:50,968 INFO  [c.c.h.v.r.VmwareResource] (DirectAgent-37:ctx-0d022993 10.2.3.147, job-37/job-43, cmd: StartCommand) (logid:bd8aa2d5) Configure VNC port for VM s-5-VM, port: 5947, host: 10.2.3.147\r\n2020-03-18 09:23:50,969 INFO  [c.c.h.v.u.VmwareHelper] (DirectAgent-37:ctx-0d022993 10.2.3.147, job-37/job-43, cmd: StartCommand) (logid:bd8aa2d5) [ignored]failed to get message for exception: null\r\n2020-03-18 09:23:50,969 WARN  [c.c.h.v.r.VmwareResource] (DirectAgent-37:ctx-0d022993 10.2.3.147, job-37/job-43, cmd: StartCommand) (logid:bd8aa2d5) StartCommand failed due to Exception: java.lang.NullPointerException\r\nMessage: null\r\n\r\njava.lang.NullPointerException\r\n    at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:2289)\r\n    at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:531)\r\n    at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)\r\n    at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)\r\n    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)\r\n    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)\r\n    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)\r\n    at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)\r\n    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\r\n    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\r\n    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)\r\n    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\r\n    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\r\n    at java.base/java.lang.Thread.run(Thread.java:834)\r\n2020-03-18 09:23:50,970 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-37:ctx-0d022993) (logid:bd8aa2d5) Seq 2-7490612080223977492: Cancelling because one of the answers is false and it is stop on error.\r\n2020-03-18 09:23:50,970 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-37:ctx-0d022993) (logid:bd8aa2d5) Seq 2-7490612080223977492: Response Received: \r\n\r\n~~~\r\nguessing this is because the bootmode is null, from the VirtualMachineTO object\r\n\r\nThis is soon followed by logs about how there are insufficient resources to bring up the vm","author":{"url":"https://github.com/davidjumani","@type":"Person","name":"davidjumani"},"datePublished":"2020-03-18T11:34:56.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":3},"url":"https://github.com/3978/cloudstack/issues/3978"}