Skip to content

Windows Container auf VMware IPI

Doc Bugs: * https://bugzilla.redhat.com/show_bug.cgi?id=1947052 * https://bugzilla.redhat.com/show_bug.cgi?id=1943587

High level steps:

  1. Install OpenShift 4.7+ with OVNKubernetes SDN and hybridOverlayConfig

  2. Prepare a Windows golden image
    Perfect time is during cluster installation ;-)

  3. Expose DNS Record api-int... for Windows Machines

  4. Install Windows Machine Config Operator (WMCO)

  1. Configure private key (public installed in golden image)
  1. Create MachineSet

Cluster installation

Create install-config.yaml

1
openshift-install create install-config --dir=cluster

Adjust install-config.yaml

1
2
cp -v cluster/install-config.yaml cluster/install-config-plain.yaml
sed -i 's/OpenShiftSDN/OVNKubernetes/' cluster/install-config.yaml

Create manifests

1
openshift-install create manifests --dir=cluster/

Configure hypride overlay

Note

Import on vSphere is the hybridOverlayVXLANPort because of Pod-to-pod connectivity between hosts is broken on my Kubernetes cluster running on vSphere

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
cat >  cluster/manifests/cluster-network-03-config.yml << EOF
apiVersion: operator.openshift.io/v1
kind: Network
metadata:
  creationTimestamp: null
  name: cluster
spec:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  externalIP:
    policy: {}
  networkType: OVNKubernetes
  serviceNetwork:
  - 172.30.0.0/16
  defaultNetwork:
    type: OVNKubernetes
    ovnKubernetesConfig:
      hybridOverlayConfig:
        hybridClusterNetwork:
        - cidr: 10.132.0.0/14
          hostPrefix: 23
        # Not supported with Windows 2019 LTSC
        hybridOverlayVXLANPort: 9898
status: {}
EOF

Install cluster

1
openshift-install create cluster --dir=cluster/

Prepare Windows golden image

Perfect time during OpenShift 4 installation :-)

Get a Windows 1909 ISO & Install a VM

Maybe 2019 works too

Note

  • Windows Server 2019 => LTSC
  • Windows Server 1909 => SAC

Update Windows

Remote Desktop

PS C:\Users\Administrator> Set-ItemProperty -Path 'HKLM:\System\CurrentControlSet\Control\Terminal Server' -name "fDenyTSConnections" -value 0 PS C:\Users\Administrator> Enable-NetFirewallRule -DisplayGroup "Remote Desktop"

GUI or CLI 

1
2
3
4
5
Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force
Set-PSRepository PSGallery -InstallationPolicy Trusted
Install-Module PSWindowsUpdate
Get-WindowsUpdate
Install-WindowsUpdate -AcceptAll -Install -IgnoreReboot
Source: win-updates.ps1

Disable IPv6

1
2
3
> Get-NetAdapterBinding

> Disable-NetAdapterBinding -Name <Name> -ComponentID ms_tcpip6

Install VMware Tools

Business as usual :-)

cmd: d:\setup64 /s /v "/qb REBOOT=R"

Configure VMware Tools

1
"exclude-nics=" | Set-Content -Path 'C:\ProgramData\VMware\VMware Tools\tools.conf'

Install all Windows Updates

Install OpenSSH

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0

Set-Service -Name ssh-agent -StartupType 'Automatic'
Set-Service -Name sshd -StartupType 'Automatic'
Start-Service ssh-agent
Start-Service sshd

$pubKeyConf = (Get-Content -path C:\ProgramData\ssh\sshd_config) -replace '#PubkeyAuthentication yes','PubkeyAuthentication yes'
$pubKeyConf | Set-Content -Path C:\ProgramData\ssh\sshd_config
$passwordConf = (Get-Content -path C:\ProgramData\ssh\sshd_config) -replace '#PasswordAuthentication yes','PasswordAuthentication yes'
$passwordConf | Set-Content -Path C:\ProgramData\ssh\sshd_config

Restart-Service sshd

Setup Public Key

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHHYQEdg8wxKijObWr4fM69/zaZv/ll7mR2ua0o2qxJMauiFMWpcD24Liihy20mOYCJEfGU3F+sAUeXL5vdSnxf21jt1bk04KAAKhLF0KlNN7eGVcG9cLR2+inugNIUIArVNNPKC4+bJkRmKS9XMByHDC20X82dTEETTtwS57au0GeuzeKK9tNOrZRABiBX3bTplyESx3KzSNoY9IhKYL58Z6RF7bus3dVtYWHpFw1FYl9E1eSnwKjN4fmAqILQe6CaJESoDRgMnOmMFFrDfgGHtw8tkhEera7iDI2ZBShccxaFLauCj5r8RtBS8NlvVJumVMsVooCHf6CVxCK0xFv17wbpW5+E8MNU9qkPVBznEed4jPyYbzEhmEHtXsgh75Y6Xp2Ycegxzl6Y1rxmSillki472kHZftihEWIGVyG3QW7vYdjemkJKDiYnqMYur6HgXEJc7L5h/rg1N7IDAOzRHGPmDbgj7QYkSPHL9CkpA/Y2CX48JtLVaLGywZO9X8= rbohne@stormshiftdeploy.coe.muc.redhat.com" | Set-Content -Path 'C:\ProgramData\ssh\administrators_authorized_keys'

# Fix permission
$acl = Get-Acl C:\ProgramData\ssh\administrators_authorized_keys
$acl.SetAccessRuleProtection($true, $false)
$administratorsRule = New-Object system.security.accesscontrol.filesystemaccessrule("Administrators","FullControl","Allow")
$systemRule = New-Object system.security.accesscontrol.filesystemaccessrule("SYSTEM","FullControl","Allow")
$acl.SetAccessRule($administratorsRule)
$acl.SetAccessRule($systemRule)
$acl | Set-Acl

Allow incoming connection for container logs:

1
2
3
$firewallRuleName = "ContainerLogsPort"
$containerLogsPort = "10250"
New-NetFirewallRule -DisplayName $firewallRuleName -Direction Inbound -Action Allow -Protocol TCP -LocalPort $containerLogsPort -EdgeTraversalPolicy Allow

Install container runtime

1
2
3
4
5
Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force
Set-PSRepository PSGallery -InstallationPolicy Trusted
Install-Module -Name DockerMsftProvider -Repository PSGallery -Force
Install-Package -Name docker -ProviderName DockerMsftProvider -Force
Restart-Computer -Force

Pre pull images

TBD

Optional: Clone VM

After sysprep you can not modifi the golden image anymore. I recommend to clone the VM and run the sysprep in the clone. If you want to change the golden image, you follow the process: 1) Made changes you want 2) Clone the VM 3) Run sysprep in the clone

Sysprep to have a propper Template

Prepare unattend.xml

Run Sysprep tool

1
2
cd 'C:\Windows\System32\Sysprep\'
.\sysprep.exe /generalize /oobe /shutdown /unattend:C:\Users\Administrator\unattend-1909.xml

Windows SDN Debugging

Windows 2019 - Failed

WMCO Pod: 

1
2021-04-01T11:27:59.081Z    ERROR   controller-runtime.controller   Reconciler error    {"controller": "windowsmachine-controller", "request": "openshift-machine-api/win-hx4pn", "error": "failed to configure Windows VM 42037020-b6d0-1820-df2e-bb90efdaa952: configuring node network failed: error waiting for k8s.ovn.org/hybrid-overlay-distributed-router-gateway-mac node annotation for win-hx4pn: timeout waiting for k8s.ovn.org/hybrid-overlay-distributed-router-gateway-mac node annotation: timed out waiting for the condition", "errorVerbose": "timed out waiting for the condition\ntimeout waiting for k8s.ovn.org/hybrid-overlay-distributed-router-gateway-mac node annotation\ngithub.com/openshift/windows-machine-config-operator/pkg/controller/windowsmachine/nodeconfig.(*nodeConfig).waitForNodeAnnotation\n\t/remote-source/build/windows-machine-config-operator/pkg/controller/windowsmachine/nodeconfig/nodeconfig.go:264\ngithub.com/openshift/w

Missing VXLAN Port in networkdetailed created via .\collectlogs.ps1

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
PS C:\Users\Administrator> Get-ComputerInfo | select WindowsProductName, WindowsVersion, OsHardwareAbstractionLayer

WindowsProductName                      WindowsVersion OsHardwareAbstractionLayer
------------------                      -------------- --------------------------
Windows Server 2019 Standard Evaluation 1809           10.0.17763.737


PS C:\Users\Administrator> (Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion").BuildLabEx
17763.1.amd64fre.rs5_release.180914-1434

PS C:\Users\Administrator> $PSVersionTable.PSVersion

Major  Minor  Build  Revision
-----  -----  -----  --------
5      1      17763  592

Nach Updates:

PS C:\Users\Administrator> Get-ComputerInfo | select WindowsProductName, WindowsVersion, OsHardwareAbstractionLayer

WindowsProductName                      WindowsVersion OsHardwareAbstractionLayer
------------------                      -------------- --------------------------
Windows Server 2019 Standard Evaluation 1809           10.0.17763.1790


PS C:\Users\Administrator> Get-ComputerInfo | select WindowsProductName, WindowsVersion, OsHardwareAbstractionLayer

WindowsProductName                      WindowsVersion OsHardwareAbstractionLayer
------------------                      -------------- --------------------------
Windows Server 2019 Standard Evaluation 1809           10.0.17763.1790


PS C:\Users\Administrator> $PSVersionTable.PSVersion

Major  Minor  Build  Revision
-----  -----  -----  --------
5      1      17763  1490

--------



PS C:\Users\Administrator> Get-ComputerInfo | select WindowsProductName, WindowsVersion, OsHardwareAbstractionLayer

WindowsProductName                      WindowsVersion OsHardwareAbstractionLayer
------------------                      -------------- --------------------------
Windows Server 2019 Standard Evaluation 1809           10.0.17763.1852



PS C:\Users\Administrator> (Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion").BuildLabEx
17763.1.amd64fre.rs5_release.180914-1434
PS C:\Users\Administrator> $PSVersionTable.PSVersion

Major  Minor  Build  Revision
-----  -----  -----  --------
5      1      17763  1852
Last update: April 7, 2021