René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"IPv6 ROUTED Filtered networks: VR's ip6_firewall fw_input chain missing 'ct state established,related accept' rule — IPv6 BGP cannot establish","articleBody":"# IPv6 BGP-routed Isolated network: missing `ct state established,related` INPUT rule on VR's IPv6 firewall\n\n## Summary\n\nWhen creating a tenant network using an IPv6-only ROUTED + Filtered offering (`internetprotocol=ipv6`, `networkmode=ROUTED`, services including Firewall), the Virtual Router's nftables `ip6 ip6_firewall fw_input` chain has `policy drop` and only ICMPv6 accept rules. There is **no `ct state established,related accept` rule** on the public NIC.\n\nBecause the VR initiates BGP outbound to upstream PE peers, the **return SYN-ACK is silently dropped at the v6 INPUT hook**, before TCP's MD5 verification ever runs. BGP IPv6 sessions cannot reach `Established`.\n\nThe equivalent IPv4 INPUT chain on the same VR DOES have `iifname \"eth2\" ct state related,established counter accept`, and IPv4 BGP works correctly.\n\n## Environment\n\n- **Apache CloudStack 4.22.0.0** (live install on staging mgmt host)\n- Source analysis cross-checked against `4.20` branch HEAD `a7c2a05` — same bug visible in source on both branches\n- Hypervisor: KVM on Ubuntu 24.04\n- Hosts: 2-node staging cluster\n- VR systemvm template: ACS 4.20 stock\n- FRR on VR: `8.4.4`\n- Network offering: `IsolatedV6RoutedFiltered` (`internetprotocol=ipv6`, `routingmode=Dynamic`, `networkmode=ROUTED`, services `[UserData, Firewall, Dhcp, Dns]`, `egressdefaultpolicy=true`)\n- BGP peer ASN: `999999` (external)\n- ACS ASN range: `4200000001-4200000099` (32-bit private)\n- IPv6 guest prefix: `/48`\n- Reproduced on **two independent VRs** (`r-276-VM` ASN 4200000052, `r-278-VM` ASN 4200000081) — identical symptom, identical fix.\n\n## Steps to reproduce\n\n1. Configure zone with IPv6 BGP routing: ASN range, BGP peers (dual-stack), IPv6 guest prefix `/48`.\n2. Create a network offering matching the above shape, then enable it.\n3. `createNetwork` using the offering.\n4. Deploy a VM into the network — VR is provisioned.\n5. SSH into the VR via its link-local IP (`port 3922`, systemvm key from `/root/.ssh/id_rsa.cloud`).\n6. Check BGP state.\n\n## Expected\n\n```\n$ vtysh -c \"show bgp ipv6 unicast summary\"\nNeighbor                         State/PfxRcd\n2400:88e0:ffff:258::2  Established     1\n2400:88e0:ffff:258::3  Established     1\n```\n\nVR advertises tenant `/64` upstream; VMs in the network are reachable from the IPv6 internet.\n\n## Actual\n\n```\n$ vtysh -c \"show bgp ipv6 unicast summary\"\nNeighbor                         State/PfxRcd\n2400:88e0:ffff:258::2  Connect          0\n2400:88e0:ffff:258::3  Connect          0\n```\n\nThe IPv4 sessions on the SAME VR work normally:\n```\n10.25.12.2  Established  PfxRcd=1\n10.25.12.3  Established  PfxRcd=1\n```\n\n## Diagnostic\n\nPacket capture on the hypervisor's underlay (`bond0`, VLAN 258):\n\n```\nVR → PE: TCP SYN (port 179) with MD5\nPE → VR: TCP SYN-ACK with MD5\nVR → PE: TCP SYN retransmit (VR never sent ACK)\nPE → VR: TCP SYN-ACK retransmit\n... cycle repeats until VR's connect timeout ...\n```\n\nPE responds correctly. Return packet reaches the VR's `eth2`. But VR's nftables drops it before TCP processes it.\n\nInside the VR, the v6 firewall table:\n\n```\n$ nft list table ip6 ip6_firewall\ntable ip6 ip6_firewall {\n    chain fw_input {\n        type filter hook input priority filter; policy drop;\n        icmpv6 type { echo-request, echo-reply, nd-router-advert,\n                       nd-neighbor-solicit, nd-neighbor-advert } accept\n    }\n    chain fw_forward {\n        type filter hook forward priority filter; policy accept;\n        ct state established,related accept\n        ip6 saddr \u003ctenant-/64\u003e jump fw_chain_egress\n        ip6 daddr \u003ctenant-/64\u003e jump fw_chain_ingress\n    }\n    chain fw_chain_egress { counter accept }\n    chain fw_chain_ingress {\n        # tenant-configured ingress rules\n        ip6 saddr ::/0 ip6 daddr ::/0 icmpv6 type { ... } accept\n        ip6 saddr ::/0 ip6 daddr ::/0 tcp dport 22 accept\n        counter drop\n    }\n}\n```\n\nFor comparison, the IPv4 table on the same VR:\n\n```\n$ nft list table ip ip4_firewall\ntable ip ip4_firewall {\n    chain INPUT {\n        type filter hook input priority filter; policy drop;\n        ...\n        iifname \"eth2\" ct state established,related counter packets ... accept\n        ...\n    }\n    ...\n}\n```\n\nThe IPv4 INPUT chain has the rule on `eth2`; the IPv6 `fw_input` chain does not.\n\nKernel TCPMD5 counters are all zero, confirming the drop happens before TCP state machine — i.e., at netfilter.\n\n## Source code root cause\n\nIn `systemvm/debian/opt/cloud/bin/cs/CsAddress.py`, `fw_router_routing()` writes the default INPUT and FORWARD rules for **IPv4 only**:\n\n```python\ndef fw_router_routing(self):\n    if self.config.is_vpc() or not self.config.is_routed():\n        return\n\n    # Add default rules for INPUT chain\n    self.nft_ipv4_fw.append({'type': \"\", 'chain': 'INPUT',\n                             'rule': \"iifname lo counter accept\"})\n    self.nft_ipv4_fw.append({'type': \"\", 'chain': 'INPUT',\n                             'rule': \"iifname eth2 ct state related,established counter accept\"})  # \u003c-- this rule\n    # Add default rules for FORWARD chain\n    self.nft_ipv4_fw.append({'type': \"\", 'chain': 'FORWARD',\n               'rule': 'iifname \"eth2\" oifname \"eth0\" ct state related,established counter accept'})\n    # ... more v4-only rules ...\n```\n\nThere is **no IPv6 equivalent** of this function — `nft_ipv6_fw` is not appended-to anywhere. The IPv6 firewall's INPUT chain default rules are entirely missing for ROUTED-mode Isolated networks.\n\n`CsNetfilter.py:add_ip6_chain()` adds the `ct state established,related accept` rule **only** to FORWARD-hooked chains, not INPUT:\n\n```python\ndef add_ip6_chain(self, address_family, table, chain, hook, action):\n    ...\n    if hook == \"input\" or hook == \"output\":\n        CsHelper.execute(\"nft add rule %s %s %s icmpv6 type { ... } accept\" % ...)\n    elif hook == \"forward\":\n        CsHelper.execute(\"nft add rule %s %s %s ct state established,related accept\" % ...)\n```\n\nSo for v6 INPUT (`fw_input` chain), only ICMPv6 is allowed and the chain inherits `policy drop`. The return BGP traffic never matches anything → dropped.\n\n## Reproduction confirmed across multiple VRs\n\nTested independently on two fresh VRs in two different tenant networks. Both showed:\n- IPv4 BGP works (Established)\n- IPv6 BGP stuck at Connect (PfxRcd=0)\n- Same fw_input chain layout with same missing rule\n- Same fix applies\n\n## Workaround\n\nOn the running VR, apply the missing rule and restart FRR:\n\n```bash\nnft 'add rule ip6 ip6_firewall fw_input iifname \"eth2\" ct state established,related counter accept'\nsystemctl restart frr\n```\n\nWithin seconds, both IPv6 BGP sessions reach `Established`, tenant /64 is advertised, VMs become reachable from IPv6 internet. Verified end-to-end with SSH from public IPv6 internet to VM inside the v6-only routed network.\n\n**Caveat**: the workaround is in-memory only. Lost on:\n- VR reboot\n- Any subsequent `cmk createIpv6FirewallRule` / `cmk deleteIpv6FirewallRule` call (ACS regenerates the chain from its own config DB, wiping the manually-added rule)\n- Any other event that triggers a v6 firewall reconfiguration on the VR\n\nEach tenant FW rule change wipes the workaround. The operator has to SSH back into the VR and re-apply the nft rule after every FW change. This makes the offering effectively unusable as a customer product without the upstream fix.\n\n## Proposed fix — VALIDATED on a live VR\n\nAdd a v6 equivalent of `fw_router_routing()` in `systemvm/debian/opt/cloud/bin/cs/CsAddress.py` plus expose `nft_ipv6_fw` on `CsIP`. `nft_ipv6_fw` already exists on `CsConfig` (line 43); we just need to plumb it through CsIP and write into it.\n\nThree changes in `CsAddress.py`:\n\n**1. Add reference in `CsIP.__init__` (around line 312):**\n```diff\n         self.nft_ipv4_fw = config.get_nft_ipv4_fw()\n         self.nft_ipv4_acl = config.get_nft_ipv4_acl()\n+        self.nft_ipv6_fw = config.get_ipv6_fw()\n```\n\n**2. Add new `fw_router_routing_v6()` method (immediately before `fw_vpcrouter_routing` at line 674):**\n```python\ndef fw_router_routing_v6(self):\n    if self.config.is_vpc() or not self.config.is_routed():\n        return\n    # IPv6 INPUT chain defaults — mirror of fw_router_routing() for v4.\n    # Without these, return traffic for VR-initiated v6 connections (BGP etc) \n    # is silently dropped by the default-DROP policy.\n    self.nft_ipv6_fw.append({'type': \"\", 'chain': 'fw_input',\n                             'rule': \"iifname lo counter accept\"})\n    self.nft_ipv6_fw.append({'type': \"\", 'chain': 'fw_input',\n                             'rule': \"iifname eth2 ct state established,related counter accept\"})\n    if self.get_type() in [\"guest\"]:\n        self.nft_ipv6_fw.append({'type': \"\", 'chain': 'fw_input',\n                                 'rule': \"iifname %s ct state established,related counter accept\" % self.dev})\n```\n\n**3. Call it from `CsIP.configure()` (line 756-757):**\n```diff\n         self.fw_router_routing()\n         self.fw_vpcrouter_routing()\n+        self.fw_router_routing_v6()\n```\n\nNote: `eth2` is hardcoded matching the v4 convention (and `PUBLIC_INTERFACES[\"router\"]` in `CsHelper.py`). A more robust fix could reference that constant.\n\n### Validation\n\nApplied this patch in-place on a running VR (`r-278-VM`, ACS 4.22.0.0) on 2026-05-16:\n\n1. Pre-patch: v6 BGP stuck in Connect; v6 fw_input chain had only ICMPv6 accept\n2. Patch applied; `/opt/cloud/bin/configure.py cmd_line.json` triggered re-process\n3. fw_input chain now includes `iifname \"eth2\" ct state established,related counter accept`\n4. v6 BGP sessions Established within seconds, PfxRcd=1, PfxSnt=2\n\n**Survival test (the key one)**: After patch, ran `cmk createIpv6FirewallRule networkid=\u003cnet\u003e traffictype=Ingress protocol=tcp startport=80 endport=80` — this pushes `ipv6_firewall_rules.json` to the VR and triggers the full IpTablesExecutor flush+rebuild path that previously wiped the manual nft workaround. **After the FW change:**\n- `iifname \"eth2\" ct state established,related accept` rule **persists** in fw_input (with active counters)\n- Both v6 BGP sessions **still Established**\n- End-to-end SSH from public IPv6 internet to VM in the network **still works**\n\nThis confirms the fix is correct and durable. The bug is in CsAddress.py / `nft_ipv6_fw` not being populated; the rest of the pipeline handles the v6 list correctly once it has content.\n\n### VPC equivalent\n\nThe same gap likely exists in the VPC routed path (`fw_vpcrouter_routing` at line 674). Not tested here (our setup is non-VPC Isolated) but worth a symmetric audit.\n\n## Affected versions\n\n**Verified on Apache CloudStack 4.22.0.0** (latest LTS at time of filing). PR #10970, which added the equivalent FORWARD-chain rule, is present and active in this build — but the INPUT-chain rule was deliberately removed in the PR's second commit (\"Remove rule from input chain\"), leaving this regression.\n\nAffected versions (by code inspection + PR #10970 history):\n- 4.20.2, 4.20.3, 4.21.x, 4.22.0.0, 4.22.0.1 — all affected\n- 4.20.0, 4.20.1 — also affected, but for a different reason (PR #10970 itself wasn't yet merged, so both FORWARD and INPUT chains were missing the rule)\n\n## Severity\n\n**High** for anyone wanting to deploy IPv6-only ROUTED Isolated networks at scale. The feature appears to work (offering enables, network creates, VR provisions, BGP-v4 establishes) but tenant v6 traffic doesn't route because BGP-v6 silently fails. Diagnosis requires packet captures on the underlay — not obvious from the VR's own view.\n\n## Related\n\n- **PR #10970** (\"IPv6 firewall: accept packets from related and established connections\") — landed in 4.20.2 and 4.22.0.0 — added the equivalent rule to the **FORWARD** chain only. This fixed the VM-return-traffic case (downloads, etc.) but did NOT add the rule to the **INPUT** chain, leaving the VR's own outbound BGP return traffic still dropped. The PR discussion mentions a second commit \"Remove rule from input chain\" — suggesting an earlier draft did add the INPUT rule but it was removed in review. The bug described here is the consequence of that removal: VR-originated v6 connections (BGP, but also NTP, DNS lookups, etc., that the systemvm itself initiates outbound) fail on the return.\n- `IsolatedV6RoutedFiltered` offering — affected\n- `IsolatedV6RoutedOffering` (no Firewall service) — not affected (no firewall service means no `ip6_firewall` table; v6 BGP works there because no nftables drop happens)\n- IPv4 ROUTED with same offering shape — works as expected (different code path: `fw_router_routing()` in `CsAddress.py` writes the INPUT `iifname \"eth2\" ct state related,established` rule for v4)\n","author":{"url":"https://github.com/jball-resetdata","@type":"Person","name":"jball-resetdata"},"datePublished":"2026-05-17T06:27:40.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":3},"url":"https://github.com/13171/cloudstack/issues/13171"}