load-balanced-route-servers

Example: Fully-meshed Route Servers

Description

• cRPD is used as route servers for EVPN connectivity to minimize peering requirements in a many-DC environment.
• Route servers, and Kubernetes nodes, are defined as "a" or "b" side. In this example, they exist within a single Kubernetes cluster using labels to differentiate between nodes. However, it is feasible to structure this with two entirely separate Kubernetes clusters.
• These route servers are deployed as StatefulSets; these could be used to provide for configuration persistence but do not in this example.
• Redundancy groups and anycast addressing:
  • Each node is assigned to redundancy group ("side") a or b.
  • cRPD pods are scheduled via StatefulSets, and will only be scheduled on a node of their appropriate side.
  • In this example, the same IP address is configured as a loopback on each device connecting to a k8s node. Serving as an anycast address, this enables to use of only one MetalLB BGPPeer manifest globally.
  • MetalLB is used to provide external addressing for BGP connectivity to the cRPD route servers, as well as load balancing if the StatefulSet's replicas setting is greater than 1 (or multiple deployments provide endpoints for the same service).
  • MetalLB is also used in this example to provide external addressing for traditional management connectivity (SSH) to the cRPD route servers.
  • LoadBalancer services use a Local externalTrafficPolicy. This is important to ensure that:
    • MetalLB only announces the external peering IP address of the route servers on nodes hosting route servers
    • cRPD's view of the peer shows the physical router's peering IP address (not critical for function but important for operational clarity)

Requirements

• Kubernetes environment with at least two nodes (one side=a and one side=b) and with coredns active. This example has been tested on k3s.
• MetalLB:

  helm repo add metallb https://metallb.github.io/metallb
  helm install -n metallb --create-namespace metallb metallb/metallb

Usage

1. Follow the instructions in Quickstart using the example YAML files in examples/load-balanced-route-servers.

2. Configure your routers servicing the nodes:

   1. With IP connectivity to the nodes.
   2. With the loopback IP address with which your MetalLB BGPPeer will peer. Do not advertise this IP address into a routing protocol.
   3. BGP peering to the Kubernetes nodes' physical interface IPs carrying unicast IPv4 routes for the spines' and leafs' loopback addresses.

   Configuration on the router may look like:

   Junos - If RS is in a DC fabric

   interfaces {
     lo0 {
       unit 0 {
         family inet {
           # Primary loopback address
           address 192.168.1.3/32 {
             primary;
             preferred;
           }
           # MetalLB Anycast Peer
           address 192.168.255.0/32;
         }
       }
     }
     ge-0/0/2 {
       description "meshrr-kube1 eth1";
       unit 0 {
         family inet {
           address 172.16.1.12/31
         }
       }
     }
   }
   policy-options {
     policy-statement ADVERTISE-LOOPBACKS {
       term DENY-MESHRR-LB {
         from {
           route-filter 192.168.255.0/32 exact;
         }
         then reject;
       }
       from {
         family inet;
         protocol direct;
         interface lo0.0;
       }
       then accept;
     }
     policy-statement ADVERTISE-MESHRR {
       from {
         family inet;
         protocol direct;
         interface ge-0/0/2.0;
       }
       then accept;
     }
   }
   protocols {
     bgp {
       group MESHRR-LB {
           type external;
           local-address 192.168.255.0;
           ttl 1;
           family inet {
               unicast;
           }
           peer-as 65000.3;
           local-as 65000.2 private;
           allow 172.16.0.0/16;        
       }
       export [ ADVERTISE-LOOPBACKS ADVERTISE-MESHRR ];
     }
   }
   

   Junos - If RS is on the MPLS network in a VRF

   interfaces lo0 {
     unit 100 {
       family inet {
         address 10.0.0.0/32
       }
     }
   }
   routing-instances {
     DCI {
         instance-type vrf;
         protocols {
             bgp {
                 group DCI-RS {
                     type external;
                     multihop {
                         ttl 2;
                     }
                     local-address 10.0.0.0;
                     peer-as 65000;
                     allow 10.1.0.0/24;
                 }
             }
         }
         interface lo0.100;
     }
   }
   

   IOS-XR - If RS is on the MPLS network in a VRF

   interface Loopback100
    vrf DCI
    ipv4 address 10.0.0.0 255.255.255.255
   !
   router bgp 100
    vrf DCI
      neighbor 10.1.0.0/24
      remote-as 65000
      ebgp-multihop 2
      update-source Loopback100
      address-family ipv4 unicast
        route-policy PASS in
        route-policy PASS out
      !
     !
    !
   !
   route-policy PASS
     pass
   end-policy
   !
   

3. (Optional) Modify configuration templates as necessary. Customizations can be applied using several approaches, including configmaps mapped onto volumes. Such is not included or necessary in this example; see 2regions-hrr if you require an example of this.

4. Modify the YAML files to your needs. At the least, <registryURL> will need to be replaced to reference your private registry. Load the YAML files for the Deployments, IPAddressPools, BGPAdvertisements, BGPPeers, and Services into Kubernetes as per Quickstart.

Required Kubernetes Manifests

• meshrr-core.service.yml
  • Service/meshrr-core - Provides a headless service coordinating the meshrr function of automatically forming full-mesh iBGP peerings between route servers. Also includes the IPAddressPool and corresponding L2Advertisement for the management network connectivity of the route servers. (In a lab environment, a single L2 domain for cRPD management was sufficient.)
• metallb-bgppeer-global.yml
  • bgppeers.metallb.io/asn100-global-lo100 - Peers MetalLB to the loopback deployed for on each router connecting to the Kubernetes cluster.
• routeserver-<side>.ss.yml
  • ipaddresspools.metallb.io/routeserver-<side> - Creates a pool containing the single address for the service per side. autoAssign: false ensures that the address is not allocated unless specifically requested by the service.
  • bgpadvertisements.metallb.io/routeserver-<side> - Advertises the address to all peers (by default) from all nodes that host and endpoint for the service.
  • Service/routeserver-<side> - Allocates the address based on the pool defined previously and uses it as an external address load balancing BGP to all healthy pods matching the criteria.
  • StatefulSet/routeserver-<side> - Creates a StatefulSet of the service for the side. In the routerserver-b example, replicas: 2, but for most production deployments, 1 should be sufficient and operationally simpler.