Skip to content

Hosted Control Plane - KubeVirt Networking

Introduction

This guide demonstrates how to configure advanced networking for Hosted Control Plane (HCP) clusters running on KubeVirt infrastructure. It covers the setup of VLAN-isolated networks, bridge interfaces, and MetalLB load balancing to provide networking for guest OpenShift clusters.

Use Case

This configuration addresses scenarios where you need to:

  • Isolate hosted clusters using dedicated VLANs for security and network segmentation
  • Provide external connectivity for hosted cluster workloads through dedicated network interfaces
  • Enable load balancing with MetalLB for ingress traffic to hosted clusters
  • Support multiple hosted clusters on the same management cluster with proper network isolation
  • Integrate with existing enterprise networks that require VLAN tagging and specific IP ranges

This approach is particularly useful in enterprise environments where network isolation, compliance requirements, and integration with existing network infrastructure are critical for running multiple tenant clusters.

Tested with:

Component Version
OpenShift v4.18
OpenShift Virt v4.18

Overview

Test cluster information

Nodes

NAME STATUS ROLES Second interface
ucs55 Ready control-plane,master,virt-node,worker enp79s0f1
ucs56 Ready control-plane,master,virt-node,worker enp80s0f1
ucs57 Ready control-plane,master,virt-node,worker enp80s0f1

Cluster network configuration

1
oc edit network.operator

Change/Add settings:

1
2
3
4
5
6
spec:
  defaultNetwork:
    ovnKubernetesConfig:
      gatewayConfig:
        ipForwarding: Global
        routingViaHost: true

Prepare node interface

NodeNetworkConfigurationPolicy/coe-bridge-via-enp79s0f1 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: coe-bridge-via-enp79s0f1
spec:
  desiredState:
    interfaces:
      - name: enp79s0f1.2003
        type: vlan
        state: up
        vlan:
          base-iface: enp79s0f1
          id: 2003
        ipv4:
          enabled: false
        ipv6:
          enabled: false
      - bridge:
          options:
            stp:
              enabled: false
          port:
            - name: enp79s0f1.2003
        description: Linux Brige info COE Network via enp79s0f1.2003
        ipv4:
          enabled: true
          dhcp: true
        ipv6:
          enabled: false
        name: br-vlan-2003
        state: up
        type: linux-bridge
      - bridge:
          options:
            stp:
              enabled: false
          port:
            - name: enp79s0f1
        description: Linux Brige info COE Network via enp79s0f1
        ipv4:
          enabled: false
        name: coe-bridge
        state: up
        type: linux-bridge
  nodeSelector:
    coe.muc.redhat.com/second-nic: enp79s0f1
NodeNetworkConfigurationPolicy/coe-bridge-via-enp80s0f1 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: coe-bridge-via-enp80s0f1
spec:
  desiredState:
    interfaces:
      - name: enp80s0f1.2003
        type: vlan
        state: up
        vlan:
          base-iface: enp80s0f1
          id: 2003
        ipv4:
          enabled: false
        ipv6:
          enabled: false
      - bridge:
          options:
            stp:
              enabled: false
          port:
            - name: enp80s0f1.2003
        description: Linux Brige info COE Network via enp80s0f1.2003
        ipv4:
          enabled: true
          dhcp: true
        ipv6:
          enabled: false
        name: br-vlan-2003
        state: up
        type: linux-bridge
      - bridge:
          options:
            stp:
              enabled: false
          port:
            - name: enp80s0f1
        description: Linux Brige info COE Network via enp80s0f1
        ipv4:
          enabled: false
        name: coe-bridge
        state: up
        type: linux-bridge
  nodeSelector:
    coe.muc.redhat.com/second-nic: enp80s0f1
NetworkAttachmentDefinition/br-vlan-2003 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  annotations:
    k8s.v1.cni.cncf.io/resourceName: bridge.network.kubevirt.io/br-vlan-2003
  name: br-vlan-2003
  namespace: default
spec:
  config: |-
    {
        "cniVersion": "0.3.1",
        "name": "br-vlan-2003",
        "type": "bridge",
        "bridge": "br-vlan-2003",
        "ipam": {},
        "macspoofchk": false,
        "preserveDefaultVlan": false
    }

MetalLB

IPAddressPool 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: coe-2003
  namespace: metallb-system
spec:
  addresses:
    - 192.168.203.10-192.168.203.19
  autoAssign: true
  avoidBuggyIPs: false
  • Optional add label selector to use the pool only hypershift/hcp
L2Advertisement 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  annotations:
  name: coe-lab
  namespace: metallb-system
spec:
  ipAddressPools:
    - coe-2003
  interfaces:
    - enp79s0f1.2003
    - enp80s0f1.2003
  • Optional use a nodeSelector to run on specific nodes
  • Optional add specific interface

Start hosted cluster

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
export PULL_SECRET=${HOME}/redhat-pullsecret-rh-ee-rbohne.json
export KUBEVIRT_CLUSTER_NAME=hcp-2003-2
export TRUSTED_BUNDLE=${HOME}/Devel/gitlab.consulting.redhat.com/coe-lab/certificates/ca-bundle-v2.pem

hcp create cluster kubevirt \
  --name $KUBEVIRT_CLUSTER_NAME \
  --namespace clusters \
  --node-pool-replicas=2 \
  --memory '16Gi' \
  --cores '8' \
  --generate-ssh \
  --root-volume-size 120 \
  --root-volume-storage-class 'ocs-storagecluster-ceph-rbd-virtualization' \
  --pull-secret $PULL_SECRET \
  --etcd-storage-class ocs-storagecluster-ceph-rbd \
  --control-plane-availability-policy HighlyAvailable \
  --additional-trust-bundle $TRUSTED_BUNDLE \
  --release-image=quay.io/openshift-release-dev/ocp-release:4.18.13-x86_64 \
  --attach-default-network=false \
  --additional-network name:default/br-vlan-2003 \
  --external-dns-domain coe.muc.redhat.com
Render yaml 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: null
  name: clusters
spec: {}
status: {}
---
apiVersion: v1
data:
  ca-bundle.crt: |
    xxx
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: user-ca-bundle
  namespace: clusters
---
apiVersion: hypershift.openshift.io/v1beta1
kind: HostedCluster
metadata:
  creationTimestamp: null
  name: hcp-2003-2
  namespace: clusters
spec:
  additionalTrustBundle:
    name: user-ca-bundle
  autoscaling: {}
  configuration: {}
  controllerAvailabilityPolicy: HighlyAvailable
  dns:
    baseDomain: ""
  etcd:
    managed:
      storage:
        persistentVolume:
          size: 8Gi
          storageClassName: ocs-storagecluster-ceph-rbd
        type: PersistentVolume
    managementType: Managed
  fips: false
  infraID: hcp-2003-2-t4x5c
  networking:
    clusterNetwork:
      - cidr: 10.132.0.0/14
    networkType: OVNKubernetes
    serviceNetwork:
      - cidr: 172.31.0.0/16
  olmCatalogPlacement: management
  platform:
    kubevirt:
      baseDomainPassthrough: true
    type: KubeVirt
  pullSecret:
    name: hcp-2003-2-pull-secret
  release:
    image: quay.io/openshift-release-dev/ocp-release:4.18.13-x86_64
  secretEncryption:
    aescbc:
      activeKey:
        name: hcp-2003-2-etcd-encryption-key
    type: aescbc
  services:
    - service: APIServer
      servicePublishingStrategy:
        type: Route
    - service: Ignition
      servicePublishingStrategy:
        type: Route
    - service: Konnectivity
      servicePublishingStrategy:
        type: Route
    - service: OAuthServer
      servicePublishingStrategy:
        type: Route
  sshKey:
    name: hcp-2003-2-ssh-key
status:
  controlPlaneEndpoint:
    host: ""
    port: 0
---
apiVersion: hypershift.openshift.io/v1beta1
kind: NodePool
metadata:
  creationTimestamp: null
  name: hcp-2003-2
  namespace: clusters
spec:
  arch: amd64
  clusterName: hcp-2003-2
  management:
    autoRepair: false
    upgradeType: Replace
  nodeDrainTimeout: 0s
  nodeVolumeDetachTimeout: 0s
  platform:
    kubevirt:
      additionalNetworks:
        - name: default/br-vlan-2003
      attachDefaultNetwork: false
      compute:
        cores: 8
        memory: 16Gi
      networkInterfaceMultiqueue: Enable
      rootVolume:
        persistent:
          size: 120Gi
          storageClass: ocs-storagecluster-ceph-rbd-virtualization
        type: Persistent
    type: KubeVirt
  release:
    image: quay.io/openshift-release-dev/ocp-release:4.18.13-x86_64
  replicas: 2
status:
  replicas: 0
---

Configure Metallb for Ingress on hosted cluster

Follow: Optional MetalLB Configuration Steps

Adjusted IPAddressPool 
1
2
3
4
5
6
7
8
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: metallb
  namespace: metallb-system
spec:
  addresses:
  - 192.168.203.31-192.168.203.35

Ingress Service

with RFE Enable preallocation of a predictable NodePort for Ingress it would be much easier.

Service/router-loadbalancer-default 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
apiVersion: v1
kind: Service
metadata:
  labels:
    app: router
    ingresscontroller.operator.openshift.io/owning-ingresscontroller: default
    router: router-loadbalancer-default
  name: router-loadbalancer-default
  namespace: openshift-ingress
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    ingresscontroller.operator.openshift.io/deployment-ingresscontroller: default
  sessionAffinity: None
  type: LoadBalancer

DNS

Fetch information from hosted cluster:

1
2
3
4
5
6
7
8
[cloud-user@router ~]$ oc get svc -n openshift-ingress router-loadbalancer-default
NAME                          TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)                      AGE
router-loadbalancer-default   LoadBalancer   172.31.241.155   192.168.203.31   80:31572/TCP,443:30706/TCP   2m22s
[cloud-user@router ~]$

[cloud-user@router ~]$ oc get ingresscontroller -n openshift-ingress-operator default -o jsonpath="{.spec.domain}";echo
apps.hcp1.apps.rhine.coe.muc.redhat.com
[cloud-user@router ~]$

Create an A record based on the information:

1
*.apps.hcp1.apps.rhine.coe.muc.redhat.com. IN A 192.168.203.31
2025-08-08 2025-08-07 Contributors: Robert Bohne