Skip to content

Networking

An OpenShift cluster is configured using an overlay software-defined network (SDN) for both the Pod and Service networks. By default, VMs are configured with connectivity to the SDN and have the same features/connectivity as Pod-based applications.

Host-level networking is configured for the primary interface during installation (br-0/SDN). All additional networks can be configured via the NMState operator.

network_bond

Bonded NICs for Management and SDN

The initial bond interface, consisting of two adapters bonded together with an IP address on the machine network specified and configured at install time, is used for the SDN, management traffic between the node and the control plane (and administrator access), and live migration traffic. During installation, use the host network interface configuration options to configure the bond and set the IP address needed.

Additional dedicated Network Interfaces for traffic types

The following is a sample NMstate configuration making use of two adapters on the host to create a bonded interface in the LACP (802.3ad) run mode. The bonds are intended to be used for isolating network traffic for different purposes. This provides the advantage of avoiding noisy neighbor scenarios for some interfaces that may have a large impact, for example a backup for a virtual machine consuming significant network throughput impacting ODF or etcd traffic on a shared interface.

Warning

Keep in mind, NodeNetworkConfigurationPolicy following not the typical Kubernetes declarative way. If you delete the following NodeNetworkConfigurationPolicy the bond1 for example will not be deleted.

To remove the bond1 from the nodes, you have to change the state from up to absent.

Node configuration, for bond1 (LACP)
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  annotations:
    description: a bond for VM traffic and VLANs
  name: bonding-policy
spec:
  desiredState:
    interfaces:
      - name: bond1
        state: up
        type: bond
        link-aggregation:
          mode: 802.3ad
          port:
            - enp6s0f0
            - enp6s0f1

OVN-Bridge vs Linux-Bridge

10.1.3.1. Comparing Linux bridge CNI and OVN-Kubernetes localnet topology

OVN-Bridge example

Component Version
OpenShift v4.22.1

An example configuration for VM network connectivity is below, note that the bond configuration should be a part of the same NodeNetworkConfigurationPolicy to ensure they are configured together.

Node configuration, for bond1, ovs-bridge (ovs-br1) and localnet (physnet-ovs-br1-bond1)
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  annotations:
    description: Localnet, OVS Bridge and bond1
  name: physnet-ovs-br1-bond1
spec:
  nodeSelector:
    node-role.kubernetes.io/worker: ''
  desiredState:
    interfaces:
    - name: bond1
      state: up
      type: bond
      link-aggregation:
        mode: active-backup
        port:
          - eth2
          - eth3
    - name: ovs-br1
      type: ovs-bridge
      state: up
      bridge:
        allow-extra-patch-ports: true
        options:
          stp: true
        port:
        - name: bond1
    ovn:
      bridge-mappings:
      - localnet: physnet-ovs-br1-bond1
        bridge: ovs-br1
        state: present

Project/namespace configuration for OVN-Bridge

To allow virtual machines to connect to a specific VLAN.

Info

Since OpenShift 4.19, ClusterUserDefinedNetwork — supports Localnet topology. It's not needed anymore to create NetworkAttachmentDefinitions by hand.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
apiVersion: k8s.ovn.org/v1
kind: ClusterUserDefinedNetwork
metadata:
  annotations:
    description: Localnet, OVS Bridge and bond1
  name: net-default-vlan-10
spec:
  namespaceSelector:
    matchLabels:
      openshift.pub/access-to-vlan: "11"
      # kubernetes.io/metadata.name: <namespace name>
  network:
    localnet:
      ipam:
        mode: Disabled
      mtu: 1500
      physicalNetworkName: physnet-ovs-br1-bond1
      role: Secondary
      vlan:
        access:
          id: 11
        mode: Access
    topology: Localnet

Linux-Bridge example

Component Version
OpenShift v4.22.1
Node configuration, for bond1 and linux-bridge (br1)
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  annotations:
    description: Linux-Bridge and bond1
  name: physnet-br1-bond1
spec:
  nodeSelector:
    node-role.kubernetes.io/worker: ''
  desiredState:
    interfaces:
      - name: bond1
        state: up
        type: bond
        link-aggregation:
          mode: active-backup
          port:
            - eth2
            - eth3
      - name: br1
        type: linux-bridge
        state: up
        ipv4:
          enabled: false
        ipv6:
          enabled: false
        bridge:
          options:
            stp:
              enabled: false
          port:
            - name: bond1

Project/namespace configuration for Linux-Bridge

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: bridge-network
  annotations:
    k8s.v1.cni.cncf.io/resourceName: bridge.network.kubevirt.io/br1
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "name": "bridge-network",
      "type": "bridge",
      "bridge": "br1",
      "macspoofchk": false,
      "vlan": 100,
      "disableContainerInterface": true,
      "preserveDefaultVlan": false
    }

Example: Bonding -> VLAN -> LocalNet & Bridge

  • Tested with OpenShift 4.17.11

NMState for initial setup / add-node 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
hosts:
- hostname: inf49
  rootDeviceHints:
    deviceName: /dev/sda
  interfaces:
    - macAddress: b4:99:ba:b4:49:d2
      name: enp3s0f0
    - macAddress: 00:1b:21:b5:6a:20
      name: ens2f0
    - macAddress: 00:1b:21:b5:6a:21
      name: ens2f1
  networkConfig:
    interfaces:
      - name: enp3s0f0
        type: ethernet
        ipv6:
          enabled: false
        ipv4:
          enabled: false
      - name: bond0.32
        type: vlan
        state: up
        ipv4:
          enabled: true
          dhcp: true
        ipv6:
          enabled: false
        vlan:
          base-iface: bond0
          id: 32
      - name: bond0
        type: bond
        state: up
        link-aggregation:
          mode: active-backup
          options:
            primary: ens2f0
            miimon: '140'
          port:
          - ens2f0
          - ens2f1
NodeNetworkConfigurationPolicy (NNCP), create linux bridge connected to bond0 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: coe-bridge
spec:
  desiredState:
    interfaces:
    - bridge:
        options:
          stp:
            enabled: false
        port:
        - name: bond0
      name: coe-bridge
      state: up
      type: linux-bridge
  nodeSelector:
    bond0-available: "true"
net-attach-def connect to bridge 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  annotations:
    k8s.v1.cni.cncf.io/resourceName: bridge.network.kubevirt.io/coe-bridge
  name: vlan1004
  namespace: coe-bridge-test
spec:
  config: |-
    {
        "cniVersion": "0.3.1",
        "name": "vlan1004",
        "type": "bridge",
        "bridge": "coe-bridge",
        "ipam": {},
        "macspoofchk": false,
        "preserveDefaultVlan": false,
        "vlan": 1004
    }

Example: OVN Bonding (balance-slb) (OVN)

  • Tested with OpenShift 4.18.13

Warning

Balance-slb is only supported with "OVN Bonding" and not with Linux Bonds!

NodeNetworkConfigurationPolicy, bond1 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: bond1
spec:
  desiredState:
    interfaces:
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      name: enp2s0
      state: up
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      name: enp3s0
      state: up
      type: ethernet
    - bridge:
        allow-extra-patch-ports: true
        port:
        - name: patch-phy-to-ex
        - link-aggregation:
            mode: balance-slb
            port:
            - name: enp2s0
            - name: enp3s0
          name: ovs-bond
      ipv4:
        dhcp: false
        enabled: false
      ipv6:
        dhcp: false
        enabled: false
      name: br-pub
      state: up
      type: ovs-bridge
    ovn:
      bridge-mappings:
      - bridge: br-pub
        localnet: localnet-pub
        state: present
  nodeSelector:
    kubernetes.io/hostname: ocp1-worker-0
NetworkAttachmentDefinition

Via YAML or WebUI

localnet-net-attach-def

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  annotations:
    k8s.ovn.org/network-id: '7'
    k8s.ovn.org/network-name: localnet-pub
  name: nad-localnet-pub
  namespace: bonding-test
spec:
  config: |-
    {
        "cniVersion": "0.4.0",
        "name": "localnet-pub",
        "type": "ovn-k8s-cni-overlay",
        "netAttachDefName": "bonding-test/nad-localnet-pub",
        "topology": "localnet"
    }

Example: Firewalling (Isolation)

Enable MultiNetworkPolicy

https://docs.redhat.com/en/documentation/openshift_container_platform/4.17/html/networking/multiple-networks#nw-multi-network-policy-enable_configuring-multi-network-policy

1
2
3
oc patch network.operator.openshift.io cluster \
  --type=merge \
  -p '{"spec":{"useMultiNetworkPolicy":true}}'

Wait for the rollout / configuration

1
2
3
$ oc get co/network
NAME      VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
network   4.18.17   True        True          False      3d17h   DaemonSet "/openshift-ovn-kubernetes/ovnkube-node" update is rolling out (3 out of 6 updated)

Create two VM's with coe network connect

1
oc apply -f https://examples.openshift.pub/kubevirt/networking/localnet-fedora-vm.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: fedora
spec:
  runStrategy: Always
  template:
    spec:
      domain:
        devices:
          disks:
            - disk:
                bus: virtio
              name: containerdisk
            - disk:
                bus: virtio
              name: cloudinit
          rng: {}
          interfaces:
            - bridge: {}
              model: virtio
              name: coe
        features:
          acpi: {}
          smm:
            enabled: true
        firmware:
          bootloader:
            efi:
              secureBoot: true
        resources:
          requests:
            memory: 1Gi
      terminationGracePeriodSeconds: 180
      networks:
        - multus:
            networkName: coe
          name: coe
      volumes:
        - name: containerdisk
          containerDisk:
            image: quay.io/containerdisks/fedora:41
        - name: cloudinit
          cloudInitNoCloud:
            networkData: |
              version: 2
              ethernets:
                eth0:
                  dhcp4: true
            userData: |-
              #cloud-config

              users:
                - name: coe
                  lock_passwd: false
                  # redhat // mkpasswd --method=SHA-512 --rounds=4096
                  hashed_passwd: "$6$rounds=4096$kmUERoUZHwzYfQMJ$G70T2Qg24d0XUhu.GTCH7Ia1F0B/B48JqIFdzVfigeMgfG5nsxp3dEWFKokfXGmhuetFXl4l41L8t1AZgEDW0."
                  sudo: ['ALL=(ALL) NOPASSWD:ALL']
                  chpasswd: { expire: False }
                  groups: wheel
                  shell: /bin/bash
                  ssh_authorized_keys:
                    - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEQM82o2imwpHyGVO7DxCNbdE0ZWnkp6oxdawb7/MOCT coe-muc

              packages:
                - httpd

              # install puppet (and dependencies); make sure apache and postgres
              # both start at boot-time
              runcmd:
                - [ systemctl, enable, httpd.service ]
                - [ systemctl, start, httpd.service ]

Let's apply some MultiNetworkPolicy

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
apiVersion: k8s.cni.cncf.io/v1beta1
kind: MultiNetworkPolicy
metadata:
  name: deny-by-default
  namespace: localnet-demo
  annotations:
    k8s.v1.cni.cncf.io/policy-for: coe
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
  ingress: []
  egress:  []

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
apiVersion: k8s.cni.cncf.io/v1beta1
kind: MultiNetworkPolicy
metadata:
  name: allow-dns-and-default-gateway
  namespace: localnet-demo
  annotations:
    k8s.v1.cni.cncf.io/policy-for: coe
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 10.32.96.1/32
    - ipBlock:
        cidr: 10.32.96.31/32
    - ipBlock:
        cidr: 10.32.111.254/32

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
apiVersion: k8s.cni.cncf.io/v1beta1
kind: MultiNetworkPolicy
metadata:
  name: allow-ingress
  namespace: localnet-demo
  annotations:
    k8s.v1.cni.cncf.io/policy-for: coe
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress:
    - from:
      - ipBlock:
          cidr: 0.0.0.0/0

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
apiVersion: k8s.cni.cncf.io/v1beta1
kind: MultiNetworkPolicy
metadata:
  name: allow-egress
  namespace: localnet-demo
  annotations:
    k8s.v1.cni.cncf.io/policy-for: coe
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
      - ipBlock:
          cidr: 0.0.0.0/0

Debugging purpose

Create br1 via nmcli

1
2
3
4
5
6
7
8
nmcli con show --active
nmcli con add type bridge ifname br1 con-name br1
nmcli con add type bridge-slave ifname ens3 master br1
nmcli con modify br1 bridge.stp no
nmcli con down 'Wired connection 1'
nmcli con up br1
nmcli con mod br1 connection.autoconnect yes
nmcli con mod 'Wired connection 1' connection.autoconnect no

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
[root@compute-0 ~]# nmcli con show
NAME                UUID                                  TYPE      DEVICE
br1                 2ae82518-2ff3-4d49-b95c-fc8fbf029d48  bridge    br1
bridge-slave-ens3   faac459f-ce51-4ce9-8616-ea9d23aff675  ethernet  ens3
Wired connection 1  e158d160-1743-3b00-9f67-258849993562  ethernet  --
[root@compute-0 ~]# nmcli -f bridge con show br1
bridge.mac-address:                     --
bridge.stp:                             no
bridge.priority:                        32768
bridge.forward-delay:                   15
bridge.hello-time:                      2
bridge.max-age:                         20
bridge.ageing-time:                     300
bridge.group-forward-mask:              0
bridge.multicast-snooping:              yes
bridge.vlan-filtering:                  no
bridge.vlan-default-pvid:               1
bridge.vlans:                           --
[root@compute-0 ~]# ip a show dev ens3
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br1 state UP group default qlen 1000
    link/ether 52:54:00:a8:34:0d brd ff:ff:ff:ff:ff:ff
[root@compute-0 ~]# ip a show dev br1
17: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:a8:34:0d brd ff:ff:ff:ff:ff:ff
    inet 192.168.52.13/24 brd 192.168.52.255 scope global dynamic noprefixroute br1
       valid_lft 3523sec preferred_lft 3523sec
    inet6 fe80::70f0:71c5:53ea:71ee/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

Connection problem with kubevirt.io/allow-pod-bridge-network-live-migration after live migration

HCP Cluster sendling

1
2
3
4
oc get nodes
NAME                      STATUS   ROLES    AGE   VERSION
sendling-d0c14274-6nbvl   Ready    worker   11d   v1.27.8+4fab27b
sendling-d0c14274-sz7rb   Ready    worker   11d   v1.27.8+4fab27b
Ping check details node/sendling-d0c14274-6nbvl 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
oc debug node/sendling-d0c14274-6nbvl
Starting pod/sendling-d0c14274-6nbvl-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.128.8.133
If you don't see a command prompt, try pressing enter.
sh-4.4# ping www.google.de
PING www.google.de (172.253.62.94) 56(84) bytes of data.
64 bytes from bc-in-f94.1e100.net (172.253.62.94): icmp_seq=1 ttl=99 time=112 ms
64 bytes from bc-in-f94.1e100.net (172.253.62.94): icmp_seq=2 ttl=99 time=98.3 ms
^C
--- www.google.de ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 98.310/105.047/111.785/6.745 ms
sh-4.4# exit
exit

Removing debug pod ...
Ping check details node/sendling-d0c14274-sz7rb 
1
2
3
4
5
6
7
$ oc debug node/sendling-d0c14274-sz7rb
Starting pod/sendling-d0c14274-sz7rb-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.131.9.28
If you don't see a command prompt, try pressing enter.
sh-4.4# ping www.google.de
PING www.google.de (172.253.62.94) 56(84) bytes of data.
  • Node sendling-d0c14274-6nbvl - Ping google ✅
  • Node sendling-d0c14274-sz7rb - Ping google ❌
1
2
3
4
5
6
7
$ oc get pods -l kubevirt.io=virt-launcher -o wide -n rbohne-hcp-sendling
NAME                                          READY   STATUS      RESTARTS   AGE     IP             NODE                 NOMINATED NODE   READINESS GATES
virt-launcher-sendling-d0c14274-6nbvl-pb6zd   1/1     Running     0          6d2h    10.128.8.133   inf8                 <none>           1/1
virt-launcher-sendling-d0c14274-sz7rb-cw5vj   1/1     Running     0          3d20h   10.131.9.28    ucs-blade-server-1   <none>           1/1
virt-launcher-sendling-d0c14274-sz7rb-mbmv8   0/1     Completed   0          3d20h   10.131.9.28    ucs-blade-server-3   <none>           1/1
virt-launcher-sendling-d0c14274-sz7rb-nb25r   0/1     Completed   0          6d2h    10.131.9.28    ucs-blade-server-1   <none>           1/1
$

Checkout node routing

Host subnets:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
$ oc get nodes -o custom-columns="NODE:.metadata.name,node-subnets:.metadata.annotations.k8s\.ovn\.org/node-subnets"
NODE                 node-subnets
...
inf8                 {"default":["10.131.8.0/21"]}
ucs-blade-server-1   {"default":["10.131.0.0/21"]}
ucs-blade-server-3   {"default":["10.130.8.0/21"]}
...

$ oc get pods -n openshift-ovn-kubernetes -o wide -l  app=ovnkube-node
NAME                 READY   STATUS    RESTARTS       AGE    IP             NODE                 NOMINATED NODE   READINESS GATES
...
ovnkube-node-9xt5n   8/8     Running   8              2d7h   10.32.96.101   ucs-blade-server-1   <none>           <none>
ovnkube-node-hhsx5   8/8     Running   8              2d7h   10.32.96.8     inf8                 <none>           <none>
ovnkube-node-qx9bh   8/8     Running   9 (2d6h ago)   2d7h   10.32.96.103   ucs-blade-server-3   <none>           <none>
...

$ oc exec -n openshift-ovn-kubernetes -c ovn-controller ovnkube-node-9xt5n -- ovn-nbctl lr-route-list ovn_cluster_router
IPv4 Routes
Route Table <main>:
             10.128.8.133                100.88.0.9 dst-ip
             10.129.8.107              10.129.8.107 dst-ip rtos-ucs-blade-server-1 ecmp
             10.129.8.107                100.88.0.8 dst-ip ecmp
             10.130.10.29              10.130.10.29 dst-ip rtos-ucs-blade-server-1
              10.131.8.41               10.131.8.41 dst-ip rtos-ucs-blade-server-1
              10.131.9.28               10.131.9.28 dst-ip rtos-ucs-blade-server-1 ecmp
              10.131.9.28                100.88.0.8 dst-ip ecmp
              10.131.9.44               10.131.9.44 dst-ip rtos-ucs-blade-server-1
               100.64.0.2                100.88.0.2 dst-ip
               100.64.0.3                100.88.0.3 dst-ip
               100.64.0.4                100.88.0.4 dst-ip
               100.64.0.5                100.64.0.5 dst-ip
               100.64.0.6                100.88.0.6 dst-ip
               100.64.0.8                100.88.0.8 dst-ip
               100.64.0.9                100.88.0.9 dst-ip
              100.64.0.10               100.88.0.10 dst-ip
            10.128.0.0/21                100.88.0.2 dst-ip
            10.128.8.0/21                100.88.0.6 dst-ip
           10.128.16.0/21               100.88.0.10 dst-ip
            10.129.0.0/21                100.88.0.3 dst-ip
            10.130.0.0/21                100.88.0.4 dst-ip
            10.130.8.0/21                100.88.0.8 dst-ip
            10.131.8.0/21                100.88.0.9 dst-ip
            10.128.0.0/14                100.64.0.5 src-ip

$ oc exec -n openshift-ovn-kubernetes -c ovn-controller ovnkube-node-hhsx5   -- ovn-nbctl lr-route-list ovn_cluster_router
IPv4 Routes
Route Table <main>:
             10.128.8.133              10.128.8.133 dst-ip rtos-inf8
             10.129.8.107                100.88.0.5 dst-ip ecmp
             10.129.8.107                100.88.0.8 dst-ip ecmp
             10.130.10.29                100.88.0.5 dst-ip
              10.131.8.41                100.88.0.5 dst-ip
              10.131.9.28                100.88.0.5 dst-ip ecmp
              10.131.9.28                100.88.0.8 dst-ip ecmp
              10.131.9.44                100.88.0.5 dst-ip
               100.64.0.2                100.88.0.2 dst-ip
               100.64.0.3                100.88.0.3 dst-ip
               100.64.0.4                100.88.0.4 dst-ip
               100.64.0.5                100.88.0.5 dst-ip
               100.64.0.6                100.88.0.6 dst-ip
               100.64.0.8                100.88.0.8 dst-ip
               100.64.0.9                100.64.0.9 dst-ip
              100.64.0.10               100.88.0.10 dst-ip
            10.128.0.0/21                100.88.0.2 dst-ip
            10.128.8.0/21                100.88.0.6 dst-ip
           10.128.16.0/21               100.88.0.10 dst-ip
            10.129.0.0/21                100.88.0.3 dst-ip
            10.130.0.0/21                100.88.0.4 dst-ip
            10.130.8.0/21                100.88.0.8 dst-ip
            10.131.0.0/21                100.88.0.5 dst-ip
            10.128.0.0/14                100.64.0.9 src-ip
$

$ oc exec -n openshift-ovn-kubernetes -c ovn-controller ovnkube-node-qx9bh -- ovn-nbctl lr-route-list ovn_cluster_router
IPv4 Routes
Route Table <main>:
             10.128.8.133                100.88.0.9 dst-ip
             10.129.8.107                100.88.0.5 dst-ip
             10.130.10.29                100.88.0.5 dst-ip
              10.131.8.41                100.88.0.5 dst-ip
              10.131.9.28                100.88.0.5 dst-ip
              10.131.9.44                100.88.0.5 dst-ip
               100.64.0.2                100.88.0.2 dst-ip
               100.64.0.3                100.88.0.3 dst-ip
               100.64.0.4                100.88.0.4 dst-ip
               100.64.0.5                100.88.0.5 dst-ip
               100.64.0.6                100.88.0.6 dst-ip
               100.64.0.8                100.64.0.8 dst-ip
               100.64.0.9                100.88.0.9 dst-ip
              100.64.0.10               100.88.0.10 dst-ip
            10.128.0.0/21                100.88.0.2 dst-ip
            10.128.8.0/21                100.88.0.6 dst-ip
           10.128.16.0/21               100.88.0.10 dst-ip
            10.129.0.0/21                100.88.0.3 dst-ip
            10.130.0.0/21                100.88.0.4 dst-ip
            10.131.0.0/21                100.88.0.5 dst-ip
            10.131.8.0/21                100.88.0.9 dst-ip
            10.128.0.0/14                100.64.0.8 src-ip
2026-06-30 2020-05-06 Contributors: Robert Bohne Robert Guske Shalom