René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Feature: Dynamic Secondary Storage Selectors","articleBody":"##### ISSUE TYPE\r\n * Feature Idea\r\n\r\n##### COMPONENT NAME\r\n~~~\r\nAPI\r\n~~~\r\n\r\n##### CLOUDSTACK VERSION\r\n\u003c!--\r\nNew line separated list of affected versions, commit ID for issues on main branch.\r\n--\u003e\r\n\r\n~~~\r\n4.18/Main\r\n~~~\r\n\r\n##### SUMMARY\r\n\r\nCurrently, ACS does not allow resources such as ISOs, Volumes, Snapshots, and Templates to be directed to a specific purpose secondary storage. Thus, operators are unable to determine to which secondary storage a given resource will be allocated. This specification aims to extend ACS for operators to be able to direct these resources to a specific secondary storage using a user-defined rule.\r\n\r\n## Table of contents\r\n- [1. Proposed changes](#proposed-changes)\r\n- [2. New APIs](#new-api-proposal)\r\n    - [2.1. `createSecondaryStorageSelector` API](#createSecondaryStorageSelector-API)\r\n    - [2.2. `updateSecondaryStorageSelector` API](#updateSecondaryStorageSelector-API)\r\n    - [2.3. `removeSecondaryStorageSelector` API](#removeSecondaryStorageSelector-API)\r\n    - [2.4. `listSecondaryStorageSelectors` API](#listSecondaryStorageSelectors-API)\r\n    - [2.5. Variables available for the heuristic rules](#variables)\r\n    - [2.6. Examples](#examples)\r\n\r\n## \u003csection id=\"proposed-changes\"\u003e1. Proposed changes\u003c/section\u003e\r\n\r\nThis specification proposes to change the way ACS defines which secondary storage will be used to store some resources. The intent is to make the Management Server select a secondary storage based on rules written in JavaScript (JS) that will only affect new resources, i.e. resources previously allocated on secondary storage will not be changed. The proposed changes are described in the following subsections.\r\n\r\n\r\n## \u003csection id=\"new-api-proposal\"\u003e2. New API proposal\u003c/section\u003e\r\n\r\nCreate a new API (`createSecondaryStorageSelector`) which will have as mandatory parameters: (`zoneid: String`, `purpose: String`, `heuristic_rule: String`), the created rule will determine, based on the return of the `heuristic_rule`, in which secondary storage a given resource will be allocated.\r\n  \r\nRegarding the API parameters:\r\n\r\n- `zoneid`: Required, since all configurations will only be in the zone scope;\r\n- `purpose`: Required, this parameter will determine the purpose for which the heuristic was created, the values accepted by this parameter will be: `VOLUME`, `SNAPSHOT`, `ISO`, `TEMPLATE`;\r\n- `heuristic_rule`: A script, written in JS, to determine the secondary storage where the resource will be allocated. This script will take the following attributes from the secondary storage as input: `id`, `disksizeused`, `protocol`. For Volumes, we will take the `size` and `format` attributes, for Snapshots we will take the `hypervisor_type` and `size` attributes and, for Templates we will take the `format` and `hypervisor_type`.\r\n\r\nFurthermore, we propose three additional APIs, one to update a created heuristic (`updateSecondaryStorageSelector`), that will allow operators to edit the defined `heuristic_rule`; the second API (`removeSecondaryStorageSelector`) to disable a previous created heuristic, and the third to list heuristics created by operators which were defined in a zone (`listSecondaryStorageSelectors`). The image below presents an overview of the proposed changes.\r\n\r\n\u003cimg src=\"https://res.cloudinary.com/sc-clouds/image/upload/v1686935719/specs/dynamic-secondary-storage-selection/dsss_c62ebu.png\"\r\n    alt=\"Overview of the proposed changes\"\r\n    style=\"width: 70%;\r\n    height: auto;\r\n    display: block;\r\n    margin-left: auto;\r\n    margin-right: auto;\"\u003e\r\n\r\nTo persist the information about the heuristic, a new table (`heuristics`) will be created in the `cloud` database. Which will have the following columns:\r\n\r\n| **Field**            | **Type**     | **Required** | **Use**                                                                                                                                                                                  |\r\n|:--------------------:|:------------:|:------------:|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|\r\n| uuid                 | varchar(40)  | Yes          | A unique identifier of the heuristic.                                                                                                                                                    |\r\n| name                 | text         | Yes          | A unique name for the heuristic.                                                                                                                                                         |\r\n| description          | text         | No           | Description of the heuristic.                                                                                                                                                            |\r\n| zone_id              | bigint(20)   | Yes          | Foreign key to the `cloud.data_center` table.                                                                                                                                            |\r\n| purpose              | varchar(255) | Yes          | Whether the heuristic referenced was created with the purpose to manage resources of type: `VOLUME`, `TEMPLATE`, `ISO`, or `SNAPSHOT` to be directed to a specific secondary storage.    |\r\n| heuristic_rule       | text         | Yes          | Script implemented in JS that will determine to which secondary storage a given resource will be allocated.                                                                              |\r\n| created              | datetime     | Yes          | Any date from the current date. If this parameter is not informed, the default value will be the current date.                                                                           |\r\n| removed              | datetime     | No           | When the heuristic was removed.                                                                                                                                                          |\r\n\r\n\r\n## \u003csection id=\"createSecondaryStorageSelector-API\"\u003e2.1 `createSecondaryStorageSelector` API\u003c/section\u003e\r\nThe API `createSecondaryStorageSelector` will have three parameters: `zoneid`, `purpose` and `heuristic_rule`. The return of the script `heuristic_rule` will determine where the resource defined in the `purpose` parameter will be allocated for download or upload resources. The image below presents the flowchart for this API.\r\n\r\n\u003cimg src=\"https://res.cloudinary.com/sc-clouds/image/upload/v1679941242/specs/dynamic-secondary-storage-selection/createSecondaryStorageSelector_lqrvht.png\"\r\n    alt=\"Flow chart createSecondaryStorageSelector API\"\r\n    style=\"width: 70%;\r\n    height: auto;\r\n    display: block;\r\n    margin-left: auto;\r\n    margin-right: auto;\"\u003e\r\n\r\n## \u003csection id=\"updateSecondaryStorageSelector-API\"\u003e2.2 `updateSecondaryStorageSelector` API\u003c/section\u003e\r\nThe API `updateSecondaryStorageSelector` will have two parameters: `uuid` and `heuristic_rule`. This API will update the heuristic defined by the given `uuid` with the new attribute `heuristic_rule`. The image below presents the flowchart for this API.\r\n\r\n\u003cimg src=\"https://res.cloudinary.com/sc-clouds/image/upload/v1679941242/specs/dynamic-secondary-storage-selection/updateSecondaryStorageSelector_jk7mq6.png\"\r\n    alt=\"Flow chart updateSecondaryStorageSelector API\"\r\n    style=\"width: 70%;\r\n    height: auto;\r\n    display: block;\r\n    margin-left: auto;\r\n    margin-right: auto;\"\u003e\r\n\r\n## \u003csection id=\"removeSecondaryStorageSelector-API\"\u003e2.3 `removeSecondaryStorageSelector` API\u003c/section\u003e\r\nThe API `removeSecondaryStorageSelector` will have 1 parameter: `uuid`. The image below presents the flowchart for this API.\r\n\r\n\u003cimg src=\"https://res.cloudinary.com/sc-clouds/image/upload/v1679941242/specs/dynamic-secondary-storage-selection/removeSecondaryStorageSelector_xzt9xx.png\"\r\n    alt=\"Flow chart removeSecondaryStorageSelector API\"\r\n    style=\"width: 70%;\r\n    height: auto;\r\n    display: block;\r\n    margin-left: auto;\r\n    margin-right: auto;\"\u003e\r\n\r\n## \u003csection id=\"listSecondaryStorageSelectors-API\"\u003e2.4 `listSecondaryStorageSelectors` API\u003c/section\u003e\r\nThe API `listSecondaryStorageSelectors` will have 2 parameters: `zone_uuid` and `purpose`. The operator can give a valid purpose and a `zone_uuid`, so that the list of active heuristics will be filtered with the given purpose and zone. The image below presents the flowchart for this API.\r\n\r\n\u003cimg src=\"https://res.cloudinary.com/sc-clouds/image/upload/v1679941242/specs/dynamic-secondary-storage-selection/listSecondaryStorageSelectors_x8ryd3.png\"\r\n    alt=\"Flow chart listSecondaryStorageSelectors API\"\r\n    style=\"width: 70%;\r\n    height: auto;\r\n    display: block;\r\n    margin-left: auto;\r\n    margin-right: auto;\"\u003e\r\n\r\n## \u003csection id=\"variables\"\u003e 2.5 Variables available for the heuristic rules\u003c/section\u003e \r\n\r\nThe table below presents the possible values for each resource when executing the JS rule. The variables of the secondary storage are available to all types of resources; whereas, the variables for snapshots, ISOs, templates and volumes are only available for each respective resource types, according to the `purpose` parameter of the API `createSecondaryStorageSelector`.\r\n\r\n\u003ctable\u003e\r\n\u003cthead\u003e\r\n  \u003ctr\u003e\r\n    \u003cth\u003eResource\u003c/th\u003e\r\n    \u003cth\u003eVariables\u003c/th\u003e\r\n  \u003c/tr\u003e\r\n\u003c/thead\u003e\r\n\u003ctbody\u003e\r\n  \u003ctr\u003e\r\n    \u003ctd rowspan=\"5\"\u003eSecondary Storage\u003c/td\u003e\r\n    \u003ctd\u003eid\u003c/td\u003e\r\n  \u003c/tr\u003e\r\n  \u003ctr\u003e\r\n    \u003ctd\u003ename\u003c/td\u003e\r\n  \u003c/tr\u003e\r\n  \u003ctr\u003e\r\n    \u003ctd\u003eusedDiskSize\u003c/td\u003e\r\n  \u003c/tr\u003e\r\n  \u003ctr\u003e\r\n    \u003ctd\u003etotalDiskSize\u003cbr\u003e\u003c/td\u003e\r\n  \u003c/tr\u003e\r\n  \u003ctr\u003e\r\n    \u003ctd\u003eprotocol\u003c/td\u003e\r\n  \u003c/tr\u003e\r\n  \u003ctr\u003e\r\n    \u003ctd rowspan=\"3\"\u003eSnapshot\u003c/td\u003e\r\n    \u003ctd\u003esize\u003c/td\u003e\r\n  \u003c/tr\u003e\r\n  \u003ctr\u003e\r\n    \u003ctd\u003ehypervisorType\u003c/td\u003e\r\n  \u003c/tr\u003e\r\n  \u003ctr\u003e\r\n    \u003ctd\u003ename\u003c/td\u003e\r\n  \u003c/tr\u003e\r\n  \u003ctr\u003e\r\n    \u003ctd rowspan=\"4\"\u003eISO/Template\u003c/td\u003e\r\n    \u003ctd\u003eformat\u003c/td\u003e\r\n  \u003c/tr\u003e\r\n  \u003ctr\u003e\r\n    \u003ctd\u003ehypervisorType\u003c/td\u003e\r\n  \u003c/tr\u003e\r\n  \u003ctr\u003e\r\n    \u003ctd\u003etemplateType\u003c/td\u003e\r\n  \u003c/tr\u003e\r\n  \u003ctr\u003e\r\n    \u003ctd\u003ename\u003c/td\u003e\r\n  \u003c/tr\u003e\r\n  \u003ctr\u003e\r\n    \u003ctd rowspan=\"2\"\u003eVolume\u003c/td\u003e\r\n    \u003ctd\u003esize\u003c/td\u003e\r\n  \u003c/tr\u003e\r\n  \u003ctr\u003e\r\n    \u003ctd\u003eformat\u003c/td\u003e\r\n  \u003c/tr\u003e\r\n  \u003ctr\u003e\r\n    \u003ctd rowspan=\"4\"\u003eAccount\u003c/td\u003e\r\n    \u003ctd\u003eid\u003c/td\u003e\r\n  \u003c/tr\u003e\r\n  \u003ctr\u003e\r\n    \u003ctd\u003ename\u003c/td\u003e\r\n  \u003c/tr\u003e\r\n  \u003ctr\u003e\r\n    \u003ctd\u003edomain.id\u003c/td\u003e\r\n  \u003c/tr\u003e\r\n  \u003ctr\u003e\r\n    \u003ctd\u003edomain.name\u003c/td\u003e\r\n  \u003c/tr\u003e\r\n\u003c/tbody\u003e\r\n\u003c/table\u003e\r\n\r\n## \u003csection id=\"examples\"\u003e 2.6 Examples\u003c/section\u003e \r\n\r\n1. Allocate a resource type to a specific secondary storage.\r\n```js\r\nfunction findStorageWithSpecificId(pool) {\r\n\treturn pool.id === '7432f961-c602-4e8e-8580-2496ffbbc45d';\r\n}\r\n\r\nsecondaryStorages.filter(findStorageWithSpecificId)[0].id\r\n```\r\n2. Dedicate storage pools for a type of template format.\r\n```js\r\nfunction directToDedicatedQCOW2Pool(pool) {\r\n  return pool.id === '7432f961-c602-4e8e-8580-2496ffbbc45d';\r\n}\r\n\r\nfunction directToDedicatedVHDPool(pool) {\r\n  return pool.id === '1ea0109a-299d-4e37-8460-3e9823f9f25c';\r\n}\r\n\r\nif (template.format === 'QCOW2') {\r\n  secondaryStorages.filter(directToDedicatedQCOW2Pool)[0].id\r\n} else if (template.format === 'VHD') {\r\n  secondaryStorages.filter(directToDedicatedVHDPool)[0].id\r\n}\r\n```\r\n3. Direct snapshot of volumes with the KVM hypervisor to a specific secondary storage.\r\n```js\r\nif (snapshot.hypervisorType === 'KVM') {\r\n  '7432f961-c602-4e8e-8580-2496ffbbc45d';\r\n}\r\n```\r\n4. Direct resources to a specific domain:\r\n```js\r\nif (account.domain.id == '52d83793-26de-11ec-8dcf-5254005dcdac') {\r\n  '1ea0109a-299d-4e37-8460-3e9823f9f25c'\r\n} else if (account.domain.id == 'c1186146-5ceb-4901-94a1-dd1d24bd849d') {\r\n  '7432f961-c602-4e8e-8580-2496ffbbc45d'\r\n}\r\n```","author":{"url":"https://github.com/BryanMLima","@type":"Person","name":"BryanMLima"},"datePublished":"2023-06-20T13:01:14.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/7654/cloudstack/issues/7654"}