Egress IP testing¶
Namespace egress IP is a good way to fine tune access to services external to OpenShift, like databases. By default containers running on OpenShift will get IP from underlying node when they connect external services. This means that you without feature like egress IP you have to open firewall to external services from all OpenShift cluster nodes....this is not a good and security solution for production use.
With namespace egress IP you define IP address that all workload in namespace will have when they connect outside OpenShift. Official documentation can be found in here
Following tests have been executed in OCP 4.3 cluster running on Hetzner bare metal host, more info
What you need to provided examples
- some exernal service that has access log that shows request IP
- range or set of IP addresses that you can use in your environments
- container for testing, curl is enough
Test environment¶
Tests use Hetzner environment (CentOS 8) so if you rung tests in other env, you might need to change some commands.
External HTTP server¶
1 |
|
You myst change binding port to 8080 since 80 is in use. Modify httpd.conf and change Listen 80 to Listen 8080
1 |
|
1 |
|
Create test HTML page
1 2 3 4 5 6 |
|
Test that you page works
1 |
|
Container for testing¶
Create project and build and deploy container for testing
1 2 |
|
Once container is running (oc get po -w
), you can can testing connecting to external http server. In my env I can use gateway as address since Hetzner host acts as environment gateway. I could also use external hostname registered to my host.
1 |
|
Output should be something like this in response
1 2 3 4 5 6 |
|
And httpd server should have something like this in access log (tail -10 /var/log/httpd/access_log
)
1 |
|
IP adress 192.168.50.14 belongs to node where container is running
1 2 3 |
|
IP addresses to use¶
In hetzner environment there is 192.168.50.0/24 network reserved for OpenShift. OpenShift only use addresses between 192.168.50.2 - 192.168.50.15 so we take range from 192.168.50.128/25 for testing egress IP (192.168.50.128-255)
Basic egress IP test¶
Setting egress IP contains two steps. Adding egress IP range and egress IPs to hostsubnet and then adding namespace egress IP to netnamespace object.
First I'll set above mentioned range to node compute-0
1 |
|
Set egress IP for namespace
1 |
|
To egress IP really work, you need to have IP address linked to NIC un underlying info. OpenShift SDN is managed by several operators. These operatos apply changes to hostsubnet to underlying nodes, in this compute-0
1 2 3 4 5 6 7 8 9 |
|
From the output you can see that egress IP range from hostsubnet definition
Now if is test again, access log should have request from IP that we just defined.
1 |
|
1 2 |
|
Failover with single node¶
In previous example we assigned egress IP range to only single node. What happens if that node goes doen?
1 2 |
|
Now node that had our egress IP address in NIC is down.
1 2 3 4 5 6 7 8 |
|
Now lets run test again.
1 |
|
Test fails since there is not network that can carry request out.
Failover with 2+ nodes¶
How to fix this, attach that egress range to 2+ nodes.
1 2 |
|
Check that all nods are running and Ready and check that all computes nodes have egress CIDR defined
1 2 3 4 5 6 7 8 |
|
Openshift second terminal and run our test in a loop
1 |
|
Next shutdown node where egress IP is bound, in my case it is compute-1.
Once OpenShift notices that node with egress IP is down, IP will be moved to new node that has egress range defined.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
|
Automated Management of Egress IPs¶
Egress IP management can be automated with Operator based approach. Here is a blog post about implementing that.
Fully Automated Management of Egress IPs with the egressip-ipam-operator