Skip to content

IBM Fusion Access SAN

Official documentation: Deploying IBM Fusion Access for SAN

Tested with:

Component Version
OpenShift v4.20.4
OpenShift Virt v4.20.x
IBM Fusion Access for SAN 2.12.0

Prerequisites

  • IBM account (IBMid). Create an account via https://www.ibm.com/account/reg/us-en/signup?formid=urx-19776
  • Access to IBM Storage Fusion, for example via the 60-day trial
  • IBM Entitlement key to allow your OpenShift cluster to pull images from IBM Registry: https://myibm.ibm.com/products-services/containerlibrary
  • OpenShift cluster with at least three worker nodes, each with ~32 GB memory
    • All nodes (at least 3) need a shared disk (via iSCSI, FC, or shared disks in a KVM lab).
  • Access to a container registry for the GPFS kernel modules
    • Internal registry (requires registry storage)
    • External registry, for example quay.io with a private repository
  • If Secure Boot is enabled: Create and roll out your signing key. (IBM Fusion Access for SAN builds and loads its own kernel module via KMM)

Let's start the installation

If Secure Boot is enabled: Create and roll out your signing key

Documentation:

Create an key pair

Here are the commands executed on RHEL 10.

  • Create public and private key

    1
2
3
4
5
    efikeygen --dbdir /etc/pki/pesign \
  --self-sign \
  --module \
  --common-name 'CN=Organization signing key' \
  --nickname 'Custom Secure Boot key'

  • Export public key to sb_cert.cer

    1
2
3
4
    certutil -d /etc/pki/pesign \
  -n 'Custom Secure Boot key' \
  -Lr \
  > sb_cert.cer

  • Export private key to sb_cert.p12

    1
2
3
    pk12util -o sb_cert.p12 \
  -n 'Custom Secure Boot key' \
  -d /etc/pki/pesign

  • Export the unencrypted private key:

    1
2
3
4
5
    openssl pkcs12 \
  -in sb_cert.p12 \
  -out sb_cert.priv \
  -nocerts \
  -noenc

Roll out the public key

This may differ in your environment.

In my virtual lab environment running on KVM/libvirt with hetzner-ocp4:

  • Copy the public key to the VM via SSH

    1
    scp sb_cert.cer core@compute-X:~/

  • Check & import the key via mokutil on the node:

    1
2
3
4
5
6
7
8
    % sudo mokutil --list-enrolled | grep 'Subject:'
        Subject: O=Red Hat, Inc., CN=Red Hat Secure Boot CA 5/emailAddress=secalert@redhat.com
        Subject: CN=Red Hat Secure Boot CA 8/emailAddress=secalert@redhat.com

# Set a simple password; it will be needed later in the UEFI shell.
% sudo mokutil --import sb_cert.cer
input password:
input password again:

  • Reboot the node, enter the MOK manager, and enroll the key

  • Check the key via mokutil at the Node:

    1
2
3
4
    % mokutil --list-enrolled | grep 'Subject:'
        Subject: O=Red Hat, Inc., CN=Red Hat Secure Boot CA 5/emailAddress=secalert@redhat.com
        Subject: CN=Red Hat Secure Boot CA 8/emailAddress=secalert@redhat.com
        Subject: CN=Organization signing key

Install IBM Fusion Access for SAN operator

https://www.ibm.com/docs/en/fusion-software/2.12.0?topic=san-installing-fusion-access-operator

Do NOT create the FusionAccess custom resource yet!

At this point there is a YouTube video available that does not cover Secure Boot or an external registry.

Create a pull secret with IBM Entitlement Key

You can get/review the entitlement keys at the IBM Container library

1
2
oc create secret -n ibm-fusion-access generic fusion-pullsecret \
--from-literal=ibm-entitlement-key=<ibm-entitlement-key>

If Secure Boot is enabled: provide signing key

  • Private key

    1
2
3
    oc create secret generic secureboot-signing-key \
  -n ibm-fusion-access \
  --from-file=key=sb_cert.priv

  • Public key

    1
2
3
    oc create secret generic secureboot-signing-key-pub \
  -n ibm-fusion-access \
  --from-file=cert=sb_cert.cer

Optional: Configure external registry for kernel module container image

  • Create ConfigMap with external registry information:

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
    apiVersion: v1
kind: ConfigMap
metadata:
  name: kmm-image-config
  namespace: ibm-fusion-access
data:
  kmm_image_registry_url: quay.io
  kmm_image_repo: rbohne/kernel-ibm-fusion-access
  # kmm_tls_insecure: "false"
  # kmm_tls_skip_verify: "false"

  # Contains the Red Hat pull secret and the secret used to push
  kmm_image_registry_secret_name: rbohne-robot-kernel-ibm-fusion-access-pull-secret

  • Provide push secret to quay.io

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
    apiVersion: v1
kind: Secret
metadata:
name: rbohne-robot-kernel-ibm-fusion-access-pull-secret
namespace: ibm-fusion-access
stringData:
.dockerconfigjson: |
    {
    "auths": {
        "quay.io/rbohne/kernel-ibm-fusion-access": {
        "auth": "...",
        "email": "..."
        }
    }
    }
type: kubernetes.io/dockerconfigjson

    Note

    It's important to be as specific as possible here. If I only specify quay.io, the pull secret will also be used for pulling OpenShift images, which will then cause the build to fail.

Creating the FusionAccess custom resource

https://www.ibm.com/docs/en/fusion-software/2.12.0?topic=san-creating-fusionaccess-cr

Wait until the Web Console plugin is available to create the storage cluster.

Creating a storage cluster

https://www.ibm.com/docs/en/fusion-software/2.12.0?topic=san-creating-storage-cluster

After creating the storage cluster, it's building the kernel module container image. Check builds in ibm-fusion-access project.

Check the pods in the following projects:

  • ibm-fusion-access
  • ibm-spectrum-scale

To check which devices are found, you can run:

1
oc describe LocalVolumeDiscoveryResult -n ibm-fusion-access

Or check the pod logs of the following pods:

1
oc get pods -l app=devicefinder-discovery -o wide -n ibm-fusion-access

To check the kernel module build settings, look at:

1
oc get module -n ibm-fusion-access gpfs-module -o yaml

Here's an example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
apiVersion: kmm.sigs.x-k8s.io/v1beta1
kind: Module
metadata:
  finalizers:
    - kmm.node.kubernetes.io/module-finalizer
  name: gpfs-module
  namespace: ibm-fusion-access
spec:
  imageRepoSecret:
    name: kmm-registry-push-pull-secret
  moduleLoader:
    container:
      imagePullPolicy: Always
      inTreeModuleToRemove: ''
      kernelMappings:
        - build:
            baseImageRegistryTLS: {}
            buildArgs:
              - name: IBM_SCALE
                value: 'cp.icr.io/cp/gpfs/ibm-spectrum-scale-core-init@sha256:51dc287dd9ae2f8dcb60c1678fe8b535bb72e29faad24108d55b7cfe62362777'
            dockerfileConfigMap:
              name: kmm-dockerfile
          containerImage: 'quay.io/rbohne/kernel-ibm-fusion-access:${KERNEL_FULL_VERSION}-51dc287dd9ae2f8dcb60c1678fe8b535'
          inTreeModuleToRemove: ''
          literal: ''
          regexp: ^.*\.x86_64$
          sign:
            certSecret:
              name: secureboot-signing-key-pub
            filesToSign:
              - '/opt/lib/modules/${KERNEL_FULL_VERSION}/mmfslinux.ko'
              - '/opt/lib/modules/${KERNEL_FULL_VERSION}/mmfs26.ko'
              - '/opt/lib/modules/${KERNEL_FULL_VERSION}/tracedev.ko'
            keySecret:
              name: secureboot-signing-key
            unsignedImageRegistryTLS: {}
      modprobe:
        dirName: /opt
        firmwarePath: /opt/lxtrace/
        moduleName: mmfs26
        modulesLoadingOrder:
          - mmfs26
          - mmfslinux
          - tracedev
      registryTLS: {}
    serviceAccountName: fusion-access-operator-controller-manager
  selector:
    kubernetes.io/arch: amd64
    scale.spectrum.ibm.com/role: storage
status:
  devicePlugin: {}
  moduleLoader:
    nodesMatchingSelectorNumber: 3

To watch the build logs:

1
oc get builds -n ibm-fusion-access -l node.kubernetes.io/module.name=gpfs-module -l app.kubernetes.io/name=kmm

After a successful build, the build objects disappear.

Creating a filesystem

https://www.ibm.com/docs/en/fusion-software/2.12.0?topic=san-creating-filesystem

Check the details of the FileSystem if it takes too long.

In case you recognize the following message:

Disk has Spectrum Scale filesystem data structures on it. Set the 'existingDataSkipVerify' spec-parameter of the LocalDisk 0x5000c500155a3456 to true if the disk should be formatted and re-used nevertheless.

Check the LocalDisk:

1
2
3
% oc get LocalDisk -n ibm-spectrum-scale
NAME                 TYPE   READY   USED    AVAILABLE   FILESYSTEM   SIZE   AGE
0x5000c500155a3456          False   False   Unknown                         26m

If you are sure the shared disk can be wiped, set existingDataSkipVerify to true:

1
oc patch LocalDisk -n ibm-spectrum-scale 0x5000c500155a3456 --type merge -p '{"spec":{"existingDataSkipVerify":true}}'

Don't forget to review the IBM Fusion dashboard

Notes for various lab environments

Add a shared disk to all worker nodes.

Plain KVM environment

I deployed via hetzner-ocp4. Now let's add a shared LVM disk because everything is running on one node.

Info

The following issue is related to Secure Boot:

1
2
3
4
5
6
7
I0105 18:25:24.083670 1 funcs_kmod.go:12] "Starting worker" logger="kmm-worker" version="" git commit=""
I0105 18:25:24.083696 1 funcs_kmod.go:24] "Reading config" logger="kmm-worker" path="/etc/kmm-worker/config.yaml"
I0105 18:25:24.083968 1 worker.go:77] "preparing firmware for loading" logger="kmm-worker" image directory="/tmp/opt/lxtrace" host mount directory="/var/lib/firmware"
I0105 18:25:24.084219 1 modprobe.go:33] "Running modprobe" logger="kmm-worker" command="/usr/sbin/modprobe -vd /tmp/opt mmfs26"
I0105 18:25:24.086346 1 cmdlog.go:70] "modprobe: ERROR: could not insert 'mmfs26': Key was rejected by service" logger="kmm-worker.modprobe.stderr"
I0105 18:25:24.086394 1 cmdlog.go:70] "insmod /tmp/opt/lib/modules/5.14.0-570.72.1.el9_6.x86_64/tracedev.ko " logger="kmm-worker.modprobe.stdout"
E0105 18:25:24.086679 1 cmdutils.go:11] "Fatal error" err="error while waiting on the command: exit status 1" logger="kmm-worker"

Two options to solve:

  • Disable Secure Boot
  • Work with a KMM signing key / Machine Owner Key (MOK), documented above.
1
lvcreate -L1T -n fusion vg0

1
2
3
4
export CLUSTER_NAME=pluto
for node in ${CLUSTER_NAME}-compute-0  ${CLUSTER_NAME}-compute-1  ${CLUSTER_NAME}-compute-2 ; do
    virsh attach-disk $node /dev/mapper/vg0-fusion sdb --targetbus scsi --cache none --persistent --live --wwn 5000c500155a3456
done

iSCSI & RHCOS

This is ugly as hell, but works for quick testing.

1
oc apply -f https://examples.openshift.pub/storage/ibm-fusion-access-san/iscsi-helper.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
---
apiVersion: v1
kind: Namespace
metadata:
  name: iscsi-helper
spec: {}
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: iscsi-helper
  namespace: iscsi-helper
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: system:openshift:scc:privileged
  namespace: iscsi-helper
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:openshift:scc:privileged
subjects:
  - kind: ServiceAccount
    name: iscsi-helper
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: iscsi-helper
  namespace: iscsi-helper
spec:
  selector:
    matchLabels:
      name: iscsi-helper
  template:
    metadata:
      annotations:
        openshift.io/required-scc: "privileged"
      labels:
        name: iscsi-helper
    spec:
      nodeSelector:
        node-role.kubernetes.io/worker: ""
      serviceAccountName: iscsi-helper
      tolerations:
        - key: "node-role.kubernetes.io/master"
          operator: "Exists"
          effect: "NoSchedule"
      containers:
        - name: iscsi-helper
          image: registry.access.redhat.com/ubi10-micro:10.1
          securityContext:
            privileged: true
            capabilities:
              add:
                - SYS_CHROOT
            runAsGroup: 0
            runAsUser: 0
          env:
            - name: ISCSI_HELPER_TARGET
              value: "iqn.1992-08.com.netapp:sn.4f0586e4962411efb20d00a0987cd31a:vs.19"
          command:
            - /usr/sbin/chroot
            - /host
            - sh
            - -c
            - |

              iscsiadm --mode discovery --op update --type sendtargets --portal   10.32.97.20
              for i in $(seq 20 23); do  iscsiadm -m node --targetname ${ISCSI_HELPER_TARGET} -p 10.32.97.${i} --login ; done

              multipath -ll
              sleep infinity
          volumeMounts:
            - name: host-root
              mountPath: /host
              readOnly: false
      hostNetwork: true
      hostPID: true
      hostIPC: true
      volumes:
        - name: host-root
          hostPath:
            path: /
            type: Directory
2025-12-30 2025-12-30 Contributors: Robert Bohne