René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"CloudStack fails to start more VMs","articleBody":"\n### Discussed in https://github.com/apache/cloudstack/discussions/10184\n\n\u003cdiv type='discussions-op-text'\u003e\n\n\u003csup\u003eOriginally posted by **akrasnov-drv** January 14, 2025\u003c/sup\u003e\nHi,\n\nI'm struggling to make CloudStack 4.20.0.0 properly start KVM VMs on Ubuntu 22.\n\nWe have isolated network over VLAN.\nCloudStack manages to start single VM and to add several more. But when I ask to start more (e.g. 10-30), Cloudstack starts behaving weird. \nNew VMs produce different errors, then Cloudstack becomes slow, does not clean resources, and at the end stays with number of VMs in Starting state.\n\nI have 5 KVM servers connected, each able to handle 30 VMs alone (in KVM without Cloudstack). VMs use local server storage. I do not see any resource problem.\nI tried to debug the issue, and looks like virtual router stops working properly. I found in its log that it restarts managing script at some point, still part of VMs do not get proper network config. Static NAT enable also returns errors.\n`Error while enabling static nat. Ip Id: 14`\nExpunge for VMs then also hangs.\nIn addition sometimes I see KVM hosts stop communicating with management, and stop writing to their local logs.\n\nTo recover I need to restart management, delete virtual router and clean stuck resources, sometimes directly in mysql db. Agent restart is also sometimes needed.\nAny help to understand and fix the problem is highly appreciated.\nI'll provide logs or other info on request.\n\nThanks,\nAlex.\n\u003c/div\u003e\n\nTo summon\n\n    - under some load part of VMs stays in Starting state, and UI becomes unresponsive, libvirt restart revives UI and expunging of VMs\n    - part of VMs that manage to start do not get IPs\n    - most fail to get static nat configured (I have enough free public IPs)\n    - at the end primary VR fails but backup one is not promoted to primary for some 30-60 min\n","author":{"url":"https://github.com/akrasnov-drv","@type":"Person","name":"akrasnov-drv"},"datePublished":"2025-01-19T09:41:26.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":10},"url":"https://github.com/10205/cloudstack/issues/10205"}