René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"network: L3 routed networking without shared L2 domain","articleBody":"### The required feature described as a wish\n\n# Networking without Layer 2\nThis proposal is to add a new networking feature to CloudStack where Instances are directly assigned /32 (IPv4) and/or /128 (IPv6) addresses without a shared Layer 2 domain.\n\nA shared Layer 2 domain in this case would be a VLAN or VXLAN VNI where Instances shared the same Broadcast/Multicast domain and where they use a shared IP-gateway for their routing.\n\n# Layer 3\nBy leveraging various features of the Linux kernel, making this a KVM-only feature, we can directly route an IPv4 and/or IPv6 address to a virtual machine by using a dynamic routing protocol like BGP, but this could also work with OSPF(v3).\n\nBy eliminating the need for Layer 2 we can create a routed network where no Instance has a \"network relationship\" with another Instance. Every Instance has one or more routes installed in the routing table of the network and can be routed to any host at any time.\n\nIn the examples below I will use two IP-addresses:\n- 2.57.57.30\n- 2001:678:3a4:100::80\n\n# Hypervisor host as gateway\nOn the hypervisor the cloudbr0 bridge will be created and assigned an IPv4 and IPv6 address:\n\n```\nauto cloudbr0\niface cloudbr0 inet static\n    address 169.254.0.1/32\n    address fe80::1/64\n    bridge-ports none\n    bridge-stp off\n    bridge-fd 0\n```\n\nAll Instances will be connected to this bridge and they will be configured to use the following IP-gateways:\n- 169.254.0.1\n- fe80::1\n\n# Inside the Instance\nAs there is no Layer 2 available the IP-configuration within the VM has to be done using ConfigDrive for cloud-init, a Virtual Router handing out DHCP and cloud-init data is not possible in this design. For the VM there is no way of detecting the cloud-init source over the network as our current CloudStack provider within cloud-init relies on the DHCP server as a source.\n\nAfter the Instance has used cloud-init to fetch the networking information from ConfigDrive the Netplan (Ubuntu Linux) configuration would look like this:\n\n```\nnetwork:\n  ethernets:\n    ens18:\n      accept-ra: no\n      nameservers:\n          addresses:\n              - 2620:fe::fe\n              - 2620:fe::9\n      addresses:\n              - 2.57.57.30/32\n              - 2001:678:3a4:100::80/128\n      routes:\n      - to: default\n        via: fe80::1\n      - to: default\n        via: 169.254.0.1\n        on-link: true\n  version: 2\n```\n\nIn this configuration the network inside the Instance is configured to the the addresses configured on **cloudbr0** as the gateway, meaning that the hypervisor will act as the gateway and route the IP-traffic.\n\nThis results in the interface being configured:\n\n```\n2: ens18: \u003cBROADCAST,MULTICAST,UP,LOWER_UP\u003e mtu 1500 qdisc fq_codel state UP group default qlen 1000\n    link/ether 52:02:45:76:d2:35 brd ff:ff:ff:ff:ff:ff\n    altname enp0s18\n    inet 2.57.57.30/32 scope global ens18\n       valid_lft forever preferred_lft forever\n    inet6 2001:678:3a4:100::80/128 scope global \n       valid_lft forever preferred_lft forever\n```\n\nAll hypervisors will use an identical configuration for cloudbr0, this allows all Instances to have the same routes in their route table:\n\n```\nroot@web01:~# ip -6 route show\n::1 dev lo proto kernel metric 256 pref medium\n2001:678:3a4:100::80 dev ens18 proto kernel metric 256 pref medium\nfe80::/64 dev ens18 proto kernel metric 256 pref medium\ndefault via fe80::1 dev ens18 proto static metric 1024 pref medium\nroot@web01:~# ip -4 route show\ndefault via 169.254.0.1 dev ens18 proto static onlink \nroot@web01:~#\n```\n\n# ARP and NDP neighbor configuration\nCloudStack is aware of the IPv4 and/or IPv6 addresses assigned to an Instance as well as the MAC address. On the hypervisor these entries have to be installed into the kernel's routing table and neighbor table. In this example the commands would be:\n\n```\nip -6 route add 2001:678:3a4:100::80/128 dev cloudbr0\nip -6 neigh add 2001:678:3a4:100::80 lladdr 52:02:45:76:d2:35 dev cloudbr0 nud permanent\nip -4 route add 2.57.57.30/32 dev cloudbr0\nip -4 neigh add 2.57.57.30 lladdr 52:02:45:76:d2:35 dev cloudbr0 nud permanent\n```\n\nThese entries would need to be added upon Instance start on that host and removed on Instance stop/migrate. The KVM Agent should handle the orchestration of these entries.\n\n# Dynamic Routing\nConfiguring these entries in the routing table is not sufficient, these need to be advertised to the upstream network. For this the hypervisor host would need to use some form of dynamic routing. BGP is the most commonly used, while others would like to use OSPF(v3).\n\nIn both cases the hypervisor will announce these /32 (IPv4) and /128 (IPv6) addresses to the upstream network while receiving a default route (0.0.0.0/0 and ::/0) from the network to be able to route traffic.\n\nA very simple piece of configuration for FRRouting (BGP or OSPF) could be:\n\n## BGP\n```\nrouter bgp\n redistribute kernel route-map only-cloud\n!\nroute-map only-cloud permit 10\n match interface cloudbr0\n```\n\n## OSPF\n```router ospf\n redistribute kernel route-map only-cloud\n network YOUR_NETWORK/XX area 0.0.0.0\n!\nroute-map only-cloud permit 10\n match interface cloudbr0\n```\n\n# IP address pools\nAs each Instance is assigned a IPv4 and/or IPv6 address there is no need to create a \"network\" inside CloudStack. The concept would be that CloudStack simply has a pool of addresses to choose from and allocates them to an Instance\n\nA pool could be:\n\n- 2.57.57.80\n- 145.31.53.21\n- 90.78.37.15\n- 88.17.11.53\n\n- 2001:db8::100\n- 2001:678:3a4:100::80\n- 2a00:f10:415:27::100\n\nThese addresses have no relationship with eachother, but they don't have to as each individual address is assigned the a VM\n\nThis networking setup also allows for very easy single stack IPv6-only Virtual Machines where IPv4 can be added or removed when needed. There is no dependency on either of the two protocols.\n\n# Summary\nThis networking design completely eliminates the use of Layer 2 broadcast/multicast domains. Each Instance becomes a full L3 routed part of the network where CloudStack's orchestration will make sure the addresses are routed towards the host where the Instance is on.\n\nUsing this setup it's very easy to create a massively scalable and reliable network spanning multiple datacenters as there is no shared L2 or VXLAN overlay.\n\nThe most common use-case for this feature will probably be public cloud providers which need to assign public IPv4/IPv6 addresses to Instance and want to share nothing between the VMs.\n","author":{"url":"https://github.com/wido","@type":"Person","name":"wido"},"datePublished":"2025-12-08T21:04:26.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":25},"url":"https://github.com/12210/cloudstack/issues/12210"}