René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"KVM Disk-only VM Snapshots","articleBody":"##### ISSUE TYPE\r\n * Feature Idea\r\n\r\n##### COMPONENT NAME\r\n~~~\r\nVM Snapshot\r\n~~~\r\n\r\n##### CLOUDSTACK VERSION\r\n\r\n~~~\r\n4.20/main\r\n~~~\r\n\r\n##### CONFIGURATION\r\n\r\n##### OS / ENVIRONMENT\r\n\r\nKVM, file storage (NFS, Shared mountpoint, local storage)\r\n\r\n##### SUMMARY\r\n\r\nThis spec addresses an update to the disk-only VM snapshot feature on the KVM\r\n\r\n# 1. Problem Description\r\n\r\nCurrently, using KVM as the hypervisor, CloudStack does not support disk-only snapshots of VMs with volumes in NFS or local storage, CloudStack also does not support VM snapshots for stopped VMs; this means that if the user needs some sort of snapshot of their volumes, they must use the volume snapshot/backup feature. Furthermore, the current implementation relies on the same workflows as volume snapshots/backups: \r\n\r\n1. The VM will be frozen (ignoring the quiesce parameter);\r\n2. Each volume will be processed individually using the volume snapshot workflow;\r\n3. Once all the snapshots are done, the VM will be resumed. \r\n\r\nHowever, this approach is flawed: as we not only create the snapshots, but also copy all of them to another directory, there will be a lot of downtime, as the VM is frozen during this whole process. This downtime might be extremely long if the volumes are big.\r\n\r\nMoreover, as the snapshots will be copied to another directory in the primary storage, the revert takes some time as we need to copy the snapshot back.\r\n\r\n### 1.1 Basic Definitions\r\n\r\nHere are some basic definitions that will be used throughout this spec:\r\n\r\n- Backing file: a read-only file that will be read when data is not found on the top file.\r\n- Delta: a file that stores data that was changed in comparison to its backing file(s). When a snapshot is taken, the current delta/file will become the backing file of a new delta, thus preserving the data on the file being snapshotted.\r\n- Backing chain: a chain of backing files.\r\n- Current delta/snapshot: the current delta that is being written to. It may have a parent snapshot and siblings.\r\n- Parent snapshot: the immediate predecessor of a given snapshot/delta in its backing chain. A snapshot/delta can only have one parent.\r\n- Child snapshot: the immediate successor of a given snapshot in the backing chain. A snapshot might have multiple children. The current delta cannot have children.\r\n- Sibling snapshot: a snapshot that is the child of a given snapshot/delta's parent.\r\n\r\n\r\n# 2. Proposed Changes\r\n\r\nTo address the described problems, we propose to extend the VM snapshot feature on KVM to allow disk-only VM snapshots for NFS and local storage; other types of storage, such as shared-mount-point, already support disk-only VM snapshot. Furthermore, we intend to change the disk-only VM snapshot process for all other file-based storages (local, NFS and shared-mount-point): \r\n\r\n1. We will take all the snapshots at the same time, instead of one at a time.\r\n2. Unlike volume snapshots, the disk-only VM snapshots will not be copied to another directory, they will stay as is after taken and be part of the volumes' backing chains. This makes reverting a snapshot much faster as we only have to change the paths that will be pointed to in the VM's DOM.\r\n3. The VM will only be frozen if the `quiesceVM` parameter is `true`.\r\n\r\n### 2.0.2. Limitations\r\n\r\n- This proposal will not change the process for non-file based storages, such as LVM and RBD (Ceph). \r\n- The snapshots will be external, that is, they will generate a new file when created, instead of internal, where the snapshots are inside the single volume file. This is a limitation with Qemu: active Qemu domains require external disk snapshots when not saving the state of the VM.\r\n- As Libvirt does not support reverting external snapshots, the reversion will have to be done by ACS while the VM is stopped. \r\n- To avoid adding complexity to the volume migration, we will not allow migrating a volume when they have disk-only VM snapshots; this limitation is already present in the current implementation for VM snapshots with memory, although for different reasons.\r\n- After taking a disk-only VM snapshot, attaching/detaching volumes will not be permitted. This limitation already exists for other VM snapshot types.\r\n- This feature will be mutually exclusive with volume snapshots. When taking a volume snapshot of a volume that is attached to a VM with disk-only VM snapshots, the volume snapshot will fail, furthermore, the user will not be able to create snapshot policies for the volume as well. Conversely, if one of the VM's volumes has a snapshot or snapshot policy, the user will not be able to create a disk-only VM snapshot. There are multiple reasons to limit the interaction between these features:\r\n    1. When reverting volume snapshots we have two choices: create a new tree for that volume that will have the reverted snapshot as base; invalidate the VM snapshots. The first option adds complexity to the implementation by adding lots of trees that have to be taken care of, while the second option will make the user lose data.\r\n    2. The KVM incremental volume snapshot feature uses an API that is not compatible with `virDomainSnapshotCreateXML` (the API used for this feature). With this in mind, allowing volume and disk-only VM snapshots to coexist would create edge cases for failure, for example: if the user has a snapshot policy, and the `kvm.incremental.snapshot` is changed to true, the volume snapshots will suddenly begin to fail.\r\n- This feature will also be mutually exclusive with VM snapshots with memory. When reverting an external snapshot, all internal that were taken after the creation of the snapshot being reverted will be lost, causing data loss. As the VM snapshot with memory uses internal snapshots and this feature uses external snapshots, we will not allow both to coexist to prevent data loss.\r\n\r\n# 2.1. Disk-only VM Snapshot Creation\r\n\u003csection id=\"snap-creation\"\u003e\u003c/section\u003e\r\n\r\nThe proposed disk-only VM snapshot creation workflow is summarized in the following diagram.\r\n\r\n\u003cimg src=\"https://res.cloudinary.com/sc-clouds/image/upload/v1715878023/specs/cloudstack/disk-only-vm-snapshot/vm_snapshot_creation_2_tftl1d.png\"\r\n    alt=\"create-snapshot\"\r\n    style=\"width: 100%;\r\n    height: auto;\"\u003e\r\n\r\n* If the VM is running, we will call the `virDomainSnapshotCreateXML` API, informing all the VM's volumes with the `snapshot` key and the `external` value, and using the flags:\r\n        1. `VIR_DOMAIN_SNAPSHOT_CREATE_ATOMIC`: to make the snapshot atomic across all the volumes;\r\n        2. `VIR_DOMAIN_SNAPSHOT_CREATE_DISK_ONLY`: to make the snapshot disk-only;\r\n        3. `VIR_DOMAIN_SNAPSHOT_CREATE_NO_METADATA`: to tell Libvirt not to save any metadata for the snapshot. This flag will be informed because we do not need Libvirt to save any metadata, all the other processes regarding the VM snapshots will be done manually using qemu-img.\r\n        4. `VIR_DOMAIN_SNAPSHOT_CREATE_QUIESCE`: if `quiesceVM` is `true`, this flag will be informed as well to keep the VM frozen during the snapshot process, once the snapshot is done it will be already thawed.\r\n* Otherwise, we will call `qemu-img create` for every volume of the VM, to create a delta on top of the current file, which will become our snapshot.\r\n\r\nUnlike the volume snapshots, the disk-only VM snapshots are not designed to be backups; thus, we will not copy the disk-only VM snapshots to another directory or storage. We want the disk-only snapshots to be fast to revert whenever needed, and keeping them in the volumes backing-chain is the best way to achieve this.\r\n\r\nCurrently, the VM is always frozen and resumed during the snapshot process, regardless of what is informed in the `quiesceVM` parameter. This process will be changed, the VM will only be frozen if the `quiesceVM` is informed. Furthermore, the downtime of the proposed process will be orders of magnitude smaller then the current implementation, as there will not be any copy while the VM is frozen.\r\n\r\nDuring the VM snapshot process, the snapshot job is queued alongside the other VM jobs; therefore, we do not have to worry about the VM being stopped/started during the snapshot, as each job is processed sequentially for each given VM. Furthermore, after creating the VM snapshot, ACS already forbids detaching volumes from the VM, so we do not need to worry about this case as well.\r\n\r\n# 2.2. VM Snapshot Reversion\r\n\u003csection id=\"snap-reversion\"\u003e\u003c/section\u003e\r\n\r\nThe proposed disk-only VM snapshot restore process is summarized in the diagram below. The process will be repeated for all the VM's volumes.\r\n\r\n\u003cimg src=\"https://res.cloudinary.com/sc-clouds/image/upload/v1715707430/specs/cloudstack/disk-only-vm-snapshot/vm_snapshot_reversion_1_iw5dej.png\"\r\n    alt=\"revert-snapshot\"\r\n    style=\"width: 100%;\r\n    height: auto;\"\u003e\r\n\r\n1. If the VM is running, we throw an exception, otherwise, we continue;\r\n2. If the current delta's parent is dead, we:\r\n    1. Merge our sibling and our parent;\r\n    2. Rebase our sibling's children (if any) to point to the merged snapshot;\r\n    3. Remove our sibling's old file, as it is now empty;\r\n    4. Change our sibling's path in the DB so that it points to the merged file;\r\n3. Delete the current delta that is being written to. It only contains changes on the disk that will be reverted as part of the process;\r\n4. Create a new delta on top of the snapshot that is being reverted to, so that we do not write directly into it and are able to return to it again later; \r\n5. Update the volume path to point to the newly created delta.\r\n\r\nThe proposed process will allow us to go back and forth on snapshots if need be. Furthermore, this process will be much faster than reverting a volume snapshot, as the bottleneck here is deleting the top delta that will not be used anymore; which should be much faster than copying a volume snapshot from another storage and replacing the old volume. \r\n\r\nThe process done in step 2 was added to cover an edge case where dead snapshots would be left in the storage until the VM was expunged. Here's a simple example of why it's needed:\r\n\r\n- Let's imagine the below image represents the current state of the snapshot tree: with `Current` being the current delta that is being written to, `Snap 1` the parent of `Current` and `Snap 2`. If we delete `Snap 1`, following the diagram on the \u003ca href=\"#snap-deletion\" class=\"internal-link\"\u003esnapshot deletion section\u003c/a\u003e, we can see that it will be marked as destroyed, but will not be deleted nor merged, as none of these operations can be done in this situation.\r\n\r\n\u003cimg src=\"https://res.cloudinary.com/sc-clouds/image/upload/v1715784150/specs/cloudstack/disk-only-vm-snapshot/Drawing_1_gsu9vr.png\"\r\n    style=\"width: 10%;\r\n    display: block;\r\n    margin-left: auto;\r\n    margin-right: auto;\r\n    height: auto;\"\u003e\r\n\r\n- Then, we decide to revert to `Snap 2`, thus the old `Current` will be deleted and a new delta will be created on top of `Snap 2`. The problem is that now we have a dead snapshot, that will not be removed by any other process, as the user will not see it, and none of the processes of the snapshot deletion will delete it:\r\n\r\n\u003cimg src=\"https://res.cloudinary.com/sc-clouds/image/upload/v1715784167/specs/cloudstack/disk-only-vm-snapshot/Drawing_2_vmmjhl.png\"\r\n    style=\"width: 5%;\r\n    display: block;\r\n    margin-left: auto;\r\n    margin-right: auto;\r\n    height: auto;\"\u003e\r\n\r\n- Now adding step 2 of the proposed reversion workflow: during reversion we will merge `Snap 1` and `Snap 2` (using `qemu-img commit`) and be left with only `Snap 2`, and our current delta:\r\n\r\n\u003cimg src=\"https://res.cloudinary.com/sc-clouds/image/upload/v1715784171/specs/cloudstack/disk-only-vm-snapshot/Drawing_3_jlf7xx.png\"\r\n    alt=\"revert-snapshot-ex3\"\r\n    style=\"width: 5%;\r\n    display: block;\r\n    margin-left: auto;\r\n    margin-right: auto;\r\n    height: auto;\"\u003e\r\n    \r\n# 2.3. VM Snapshot Deletion\r\n\u003csection id=\"snap-deletion\"\u003e\u003c/section\u003e\r\n\r\nIn order to keep the snapshot tree consistent and with the least amount of dead nodes, the snapshot deletion process will always try to manipulate the snapshot tree to remove any unneeded nodes while keeping the ones that are still needed; even if they were removed by the user, in these cases, they'll be marked as deleted on the DB, but will remain on the storage primary until they can be merged with another snapshot. The diagram below summarizes the snapshot deletion process, this process will be repeated for all the VM's volumes:\r\n\r\n\u003cimg src=\"https://res.cloudinary.com/sc-clouds/image/upload/v1715869488/specs/cloudstack/disk-only-vm-snapshot/vm_snapshot_deletion_6_iobpxf.png\"\r\n    alt=\"snapshot-deletion\"\r\n    style=\"width: 100%;\r\n    height: auto;\"\u003e\r\n\r\nAs this diagram has several branches, each branch will be explained separately:\r\n\r\n* If our snapshot has a single child: \r\n    1. we will merge it with its child. If the VM is running we use Libvirt's `virDomainBlockCommit`; else use `qemu-img commit`, to commit the child to it;\r\n    2. rebase its grandchildren to the merged file, if they exist. If the VM is running, the `virDomainBlockCommit` API already does this for us;\r\n    3. remove the child's old file;\r\n    4. change the child's path on the DB to the merged file;\r\n* If our snapshot has more than one child:\r\n    1. we mark it as removed on the DB, but keep it in storage until it can be deleted.\r\n* If our snapshot has no children:\r\n    1. we delete the snapshot file\r\n    2. if the snapshot has a dead parent and only one sibling:\r\n        1. we merge the sibling with its parent. If the VM is running we use Libvirt's `virDomainBlockCommit`; else use `qemu-img commit`, to commit the sibling to the parent;\r\n        2. rebase the sibling's children to the merged file. If the VM is running, the `virDomainBlockCommit` API already does this for us;\r\n        3. remove the sibling's old file;\r\n        4. change the sibling's path on the DB to the merged file;\r\n\r\nThe proposed deletion process leaves room for one edge case, which can lead to a dead node that would only be removed when the volume was deleted: If we revert to a snapshot that has one other child and then delete it, using the above algorithm, the deleted snapshot will end up only marked as removed on the DB. If we revert to another snapshot, this will leave a dead node on the tree that would not be removed (the snapshot that was previously deleted). To solve this edge case, when this specific situation happens, we will do as explained in the \u003ca href=\"#snap-reversion\" class=\"internal-link\"\u003esnapshot reversion section\u003c/a\u003e and merge the dead node with its child.\r\n\r\n# 2.4. Template Creation from Volume\r\n\r\nThe current process of creating a template from a volume does not need to be changed. We already convert the volume when creating a template, so the volume's backing chain will be merged when creating a template.","author":{"url":"https://github.com/JoaoJandre","@type":"Person","name":"JoaoJandre"},"datePublished":"2024-08-13T18:31:06.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/9524/cloudstack/issues/9524"}