René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Live scaling for VMs with fixed service offerings on KVM","articleBody":"# Live scaling for VMs with fixed service offerings on KVM\n\nThis specification introduces a new feature enabling users to live scale VMs created with fixed service offerings on the KVM hypervisor.\n\n## Table of contents\n- [Live scaling for VMs with fixed service offerings on KVM](#live-scaling-for-vms-with-fixed-service-offerings-on-kvm)\n  - [Table of contents](#table-of-contents)\n  - [1. Problem description](#1-problem-description)\n  - [2. Proposed changes](#2-proposed-changes)\n    - [2.1. Apache CloudStack definition of domain XMLs with KVM](#21-apache-cloudstack-definition-of-domain-xmls-with-kvm)\n    - [2.2. New cluster-wide settings to control the memory and vCPUs maximum capacity for live scaling](#22-new-cluster-wide-settings-to-control-the-memory-and-vcpus-maximum-capacity-for-live-scaling)\n    - [2.3. `scaleVirtualMachine` API](#23-scalevirtualmachine-api)\n    - [2.4. New cluster-wide setting to control automatic VM migration on insufficient capacity during live scaling](#24-new-cluster-wide-setting-to-control-automatic-vm-migration-on-insufficient-capacity-during-live-scaling)\n  - [3. Conclusion and Limitations](#3-conclusion-and-limitations)\n\n## \u003csection id=\"1-problem-description\"\u003e1. Problem description\u003c/section\u003e\n\nCurrently, Apache CloudStack supports three types of compute offerings: fixed, custom constrained and custom unconstrained offerings. Fixed offerings have a fixed number of vCPUs, vCPU speed, and RAM memory. Custom constrained offerings accept a fixed vCPU speed, and a range of vCPUs and RAM memory in which users can select a value from. Lastly, custom unconstrained offerings accept an arbitrary amount of vCPUs, vCPU speed and RAM memory.\n\nWhen using KVM as hypervisor, Apache CloudStack supports scaling `Stopped` instances by executing the `scaleVirtualMachine` API. During this process, since the VMs are stopped, their metadata is updated in the database and, thus, when the VMs are later started again, their respective domain XMLs are created with the updated attributes.\n\nFor running VMs, Apache CloudStack only supports live scaling VMs with custom constrained and custom unconstrained compute offerings[^enable-scale-vm-setting]. For VMs with fixed compute offerings, users necessarily need to stop the VMs, change their service offerings through the `scaleVirtualMachine` API and start them again. However, depending on the criticality of the applications running on the VMs, the downtime caused by the scaling process is highly undesirable.\n\n## \u003csection id=\"2-proposed-changes\"\u003e2. Proposed changes\u003c/section\u003e\n\nTo address the described problem, this specification introduces the feature of live scaling VMs with fixed service offerings when using the KVM hypervisor on Apache CloudStack cloud environments. A high-level design of the feature is presented, briefly describing the proposed changes to the generation of guest VM domain XMLs, the `scaleVirtualMachine` API workflows and the global settings controlling the scaling process.\n\n### \u003csection id=\"21-apache-cloudstack-definition-of-domain-xmls-with-kvm\"\u003e2.1. Apache CloudStack definition of domain XMLs with KVM\u003c/section\u003e\n\nDuring the process of deploying virtual machines with KVM, a VM transfer object (`VirtualMachineTO`) is implemented given a `VirtualMachineProfile`, which is an object that stores the attributes of the VM. The implementation of VMs transfer objects is illustrated in the following activity diagram:\n\n\u003cimg width=\"361\" height=\"921\" alt=\"Image\" src=\"https://github.com/user-attachments/assets/f6dce360-5f80-4c6f-ab8c-ab8bc634a2e9\" /\u003e\n\nWhen configuring the memory and vCPU attributes for the VMs, Apache CloudStack implements the following workflow:\n\n\u003cimg width=\"849\" height=\"657\" alt=\"Image\" src=\"https://github.com/user-attachments/assets/aad3adc2-758a-417a-aa73-34e97d8c2827\" /\u003e\n\nRegarding the definition of domain XMLs for guest VMs, the following boilerplate is used by the Apache CloudStack Agent:\n\n```xml\n\u003cdomain type='kvm' id='4'\u003e\n    \u003c!--(...)--\u003e\n    \u003cmaxMemory slots='16' unit='KiB'\u003e1930240\u003c/maxMemory\u003e\n    \u003cmemory unit='KiB'\u003e1048576\u003c/memory\u003e\n    \u003ccurrentMemory unit='KiB'\u003e1048576\u003c/currentMemory\u003e\n    \u003cvcpu placement='static' current='1'\u003e2\u003c/vcpu\u003e\n    \u003c!--(...)--\u003e\n    \u003ccpu mode='custom' match='exact' check='full'\u003e\n        \u003c!--(...)--\u003e\n        \u003cnuma\u003e\n            \u003ccell id='0' cpus='0-1' memory='1048576' unit='KiB'/\u003e\n        \u003c/numa\u003e\n        \u003c!--(...)--\u003e\n    \u003c/cpu\u003e\n\u003c/domain\u003e\n```\n\nRegarding vCPU configuration, the content of the `vcpu` element defines the maximum number of vCPUs that can be allocated for the guest VM. Its `current` property defines the number of vCPUs that are effectively active[^cpu-configuration-libvirt-docs]. As can be analyzed from the above activity diagrams, for custom constrained compute offerings, the content of the `vcpu` element is defined as the offering's maximum number of vCPUs. For custom unconstrained offerings, the element's content is defined as the `vm.serviceoffering.cpu.cores.max` global setting value, defaulting to the host maximum CPU capacity in scenarios that the global setting is equal to zero.\n\nAs for guest VMs memory configuration, the `maxMemory` element defines the maximum memory allocation for the VM. The `memory` element represents the maximum amount of memory available for the VM at boot time. The `currentMemory` represents the actual allocation of memory for the VM[^memory-configuration-libvirt-docs]. Lastly, Libvirt currently requires the specification of NUMA nodes to enable memory hotplug. Each `cell` element represents a NUMA node. The `cpus` attribute define the range of CPUs that are part of the node. The `memory` attribute represents the amount of memory in use by the VM[^numa-configuration-libvirt-docs].\n\nAs can be noticed from the VM configuration workflow, for custom constrained compute offerings, the `maxMemory` element content is defined as the offering's maximum number of memory. For custom unconstrained offerings, the element's content is defined as the `vm.serviceoffering.ram.size.max` global setting value, defaulting to the host maximum memory capacity in scenarios in which the global setting is equal to zero.\n\nHence, to address the live scaling of VMs with fixed offerings, the first validation on the configuration of memory and vCPU for a guest VM, that checks whether the VM is dynamically scalable, will be modified. Currently, it considers a VM to be dynamically scalable if its offering is dynamic (that is, either the amount of vCPUs, vCPU speed or RAM memory is not specified); the VM's `dynamically_scalable` property is `true`; and the global setting `enable.dynamic.scale.vm` is true. \n\nTherefore, the check for dynamic offerings will be removed from the above mentioned validation. As a consequence of that, when the `dynamically_scalable` property of VMs is `true` and the `enable.dynamic.scale.vm` global setting is `true`, then the domain XMLs of guest VMs will always have a range of memory and vCPUs to be scaled up to, even when the VMs are created from fixed compute offerings.\n\nTo define the upper limit of the memory and vCPU live scaling range, the `kvm.cpu.dynamic.scaling.capacity` and `kvm.memory.dynamic.scaling.capacity` global settings will be considered (see [new cluster-wide settings](#22-new-cluster-wide-settings-to-control-the-memory-and-vcpus-maximum-capacity-for-live-scaling)). Therefore, the maximum number of vCPUs will be retrieved from the `kvm.cpu.dynamic.scaling.capacity` setting value and the maximum number of memory will be retrieved from the `kvm.memory.dynamic.scaling.capacity` value. When these settings are equal to zero, the host maximum capacity of CPU and memory will be considered. \n\nThis procedure will also be applied for custom constrained offerings to enable seamless live scaling between all types of compute offerings existing in Apache CloudStack. Thus, at the KVM level, all types of VMs hosted in a given host will be homogeneous regarding the CPU and memory upper limits. At the Apache CloudStack level, on the other hand, the Management Server will be responsible for validating the computing resources ranges defined in the constrained offerings. \n\n### \u003csection id=\"22-new-cluster-wide-settings-to-control-the-memory-and-vcpus-maximum-capacity-for-live-scaling\"\u003e2.2. New cluster-wide settings to control the memory and vCPUs maximum capacity for live scaling\u003c/section\u003e \n\nAs mentioned on the [domain XMLs definition section](#21-apache-cloudstack-definition-of-domain-xmls-with-kvm), the `vm.serviceoffering.cpu.cores.max` and `vm.serviceoffering.ram.size.max` global settings are used to control de maximum amount of vCPUs and memory to which VMs with custom unconstrained offerings can be live scaled to. However, both settings are also currently used to limit the maximum amount of CPU and RAM that can be defined for compute offerings and allocated to VMs.\n\nTo segregate goals and responsibilities, two new cluster-wide settings will be introduced:\n\n| Name   | Type   | Scope  | Description |\n| ------ | ------ | ------ | ----------- |\n| `kvm.memory.dynamic.scaling.capacity` | Integer | Cluster | Defines the maximum memory capacity in MiB for which VMs can be dynamically scaled to with KVM. The `kvm.memory.dynamic.scaling.capacity` setting's value will be used to define the value of the `\u003cmaxMemory /\u003e` element of domain XMLs. If it is set to a value less than or equal to `0`, then the host's memory capacity will be considered. |\n| `kvm.cpu.dynamic.scaling.capacity` | Integer | Cluster | Defines the maximum vCPU capacity for which VMs can be dynamically scaled to with KVM. The `kvm.cpu.dynamic.scaling.capacity` setting's value will be used to define the value of the `\u003cvcpu /\u003e` element of domain XMLs. If it is set to a value less than or equal to  `0`, then the host's CPU cores capacity will be considered. |\n\nTherefore, both settings will be used to exclusively control the maximum live scaling capacity for memory and vCPUs. To maintain compatibility with the current behavior, the values of the `vm.serviceoffering.cpu.cores.max` and `vm.serviceoffering.ram.size.max` global settings will be used to populate the initial values of the new cluster-wide settings.\n\n### \u003csection id=\"23-scalevirtualmachine-api\"\u003e2.3. `scaleVirtualMachine` API\u003c/section\u003e\n\nThe following activity diagram illustrates the current implementation of the VM scaling process:\n\n\u003cimg width=\"951\" height=\"982\" alt=\"Image\" src=\"https://github.com/user-attachments/assets/13c8e18c-9ec3-4f42-b334-d4f9cf3331a1\" /\u003e\n\nAs can be noticed, when upgrading running VMs, after performing some general validations, the Management Server checks if the VM is running on KVM, and if its offering is not dynamic (that is, vCPU, vCPU speed and RAM memory are specified for the offering). If the VM meets these conditions, then an exception is thrown, informing the end user that KVM does not support live scaling VMs with fixed compute offerings.\n\nTherefore, the above-mentioned check for dynamic offerings will be removed from the scaling workflow. This is possible because the domain XMLs of guest VMs will be prepared to support live scaling when the global setting `enable.dynamic.scale.vm` and the VM's `dynamically_scalable` property are set to `true`.\n\nAdditionally, it is relevant to note that the current workflow does not update the VM's CPU quota percentage based on its new CPU frequency. To address this, the current implementation will be extended so that, before building the `ScaleVmCommand`, the new CPU quota percentage is calculated and included in the command payload, along with a flag indicating whether the CPU cap has changed between the old and new service offerings.\n\nAt the Agent side, the following workflow will be executed:\n\n\u003cimg width=\"1059\" height=\"651\" alt=\"Image\" src=\"https://github.com/user-attachments/assets/1b3c67c1-002b-4edc-9d32-a1ec38478a22\" /\u003e\n\nThus, if both the old and the new service offerings have CPU limitation enabled, the domain's CPU quota will be updated on the fly. If the old service offering does not have CPU limitation enabled but the new one does, the domain's CPU quota and period parameters will be set. Conversely, it the old offering has CPU limitation enabled and the new one does not, these parameters will be removed from the domain.  \n\nTo update the CPU scheduling parameters of a domain on the fly, the `virDomainSetSchedulerParameters` Libvirt API will be used. To remove CPU limitation from a domain, the API will be called with the `vcpu_quota` parameter set to `17,592,186,044,415`. Due to a Libvirt regression (see [Libvirt regression description](https://gitlab.com/libvirt/libvirt/-/work_items/324)), it is not possible to set this value to `-1`. However, setting it to `17,592,186,044,415` has the same effect, ensuring that CPU limitations are effectively removed from running domains despite the Libvirt's constraint.\n\n### \u003csection id=\"24-new-cluster-wide-setting-to-control-automatic-vm-migration-on-insufficient-capacity-during-live-scaling\"\u003e2.4. New cluster-wide setting to control automatic VM migration on insufficient capacity during live scaling\u003c/section\u003e \n\nThe current live scaling workflow, regardless of the hypervisor in use (KVM, VMware or XenServer), includes handling for cases where there is insufficient compute capacity to perform the operation. When the Management Server identifies that the VM's current host lacks sufficient capacity to scale the instance, it automatically migrates the VM to a suitable host.\n\nHowever, this behavior may be undesirable in certain cloud production environments. Live scaling operations triggered by end users can lead to multiple unpredictable migrations, potentially impacting other aspects of the infrastructure. To address this, the following cluster-wide setting will be introduced to allow operators to control whether VMs should be automatically migrated in such scenarios:\n\n| Name   | Type   | Scope  | Description |\n| ------ | ------ | ------ | ----------- |\n| `auto.migrate.vm.on.live.scale.insufficient.capacity` | Boolean | Cluster | Defines whether a VM should be automatically migrated to a suitable host when the current host lacks sufficient compute capacity to live scale the instance. Defaults to true. |\n\nThe default value is set to true to preserve the existing behavior.\n\n## \u003csection id=\"3-conclusion-and-limitations\"\u003e3. Conclusion and Limitations\u003c/section\u003e\n\nThis proposal introduces support for live scaling of VMs with fixed service offerings when using the KVM hypervisor in Apache CloudStack environments. As a limitation, it is important to highlight that live scaling will not be supported for existing domains created prior to upgrading to the Apache CloudStack release in which this patch is included. This is because their domain XMLs are not prepared to support live CPU and memory scaling.\n\nTo enable live scaling for such VMs, the following steps are required:\n\n1. Ensure that the `enable.dynamic.scale.vm` global setting is enabled.\n2. Ensure that the VM's template is marked as dynamically scalable.\n3. Ensure that the VM itself is marked as dynamically scalable.\n4. Stop and start the VM so that its domain is recreated with the required structure for live scaling.\n\n[^enable-scale-vm-setting]: The global setting `enable.dynamic.scale.vm` controls whether the live scaling feature is enabled or not.\n\n[^cpu-configuration-libvirt-docs]: More information about the configuration of guest VMs CPU attributes can be found at [the Libvirt documentation](https://libvirt.org/formatdomain.html#cpu-allocation).\n\n[^memory-configuration-libvirt-docs]: More information about the configuration of guest VMs memory attributes can be found at [the Libvirt documentation](https://libvirt.org/formatdomain.html#memory-allocation).\n\n[^numa-configuration-libvirt-docs]: More information about the configuration of guest NUMA topology can be found at [the Libvirt documentation](https://www.libvirt.org/formatdomain.html#cpu-model-and-topology).\n","author":{"url":"https://github.com/bernardodemarco","@type":"Person","name":"bernardodemarco"},"datePublished":"2026-03-27T22:34:55.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/12908/cloudstack/issues/12908"}