Skip to content

Log Forwarding demo

Start fluentd to file logging

Create new project

1
oc new-project fluentd

Deploy fluentd

1
oc apply -f https://examples.openshift.pub/cluster-configuration/logging/forwarding-demo/deploy-fluentd.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: fluentd-log
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Gi
  storageClassName: managed-nfs-storage
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fluentd
spec:
  selector:
    matchLabels:
      app: fluentd
  replicas: 1
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      containers:
        - name: fluentd
          image: docker.io/fluent/fluentd:v1.3-debian-1
          ports:
            - containerPort: 24224
          volumeMounts:
          - mountPath: /fluentd/log
            name: fluentd-log
      volumes:
      - name: fluentd-log
        persistentVolumeClaim:
          claimName: fluentd-log
---
apiVersion: v1
kind: Service
metadata:
  name: fluentd
spec:
  selector:
    app: fluentd
  ports:
    - protocol: TCP
      port: 24224
      targetPort: 24224

Test fluentd

Login into the pod:

1
$ oc rsh deployment/fluentd bash

Run command inside the pod: 

1
2
3
$ echo '{"message":"fooobar"}' | fluent-cat debug.log --host fluentd.fluentd.svc.cluster.local  --port 24224
$ grep fooobar /fluentd/log/data.log
2020-10-28T10:42:24+00:00   debug.log   {"message":"fooobar"}

Deploy Cluster Logging

  • Deploy OpenShift Logging Operator
  • Deploy Elastic Search Operator from Red Hat.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
oc create -f - <<EOF
apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  name: instance
  namespace: openshift-logging
spec:
  collection:
    logs:
      fluentd: {}
      type: fluentd
EOF

Deploy Log forwarding api

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
oc create -f - <<EOF
apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputs:
   - name: fluentd-server-insecure
     type: fluentdForward
     url: 'tcp://fluentd.fluentd.svc.cluster.local:24224'
  pipelines:
   - name: fluentd-server-insecure-name
     inputRefs:
     - audit
     outputRefs:
     - fluentd-server-insecure
     parse: json
     labels:
       clusterId: "C1234"
EOF

Viewing audit logs

Usefull tool: cluster-debug-tools

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
# Extract auditlog from PV
$ cat data.log | grep k8s-audit.log | cut -f3- > audit.log

$ kubectl dev_tool audit -f audit.log  | head
14:31:55 [ WATCH][1h29m7.534747s] [200] /api/v1/namespaces/openshift-console-user-settings/configmaps?watch=true&fieldSelector=metadata.name%3Duser-settings-kubeadmin                                                                                                                              [kube:admin]
14:52:32 [ WATCH][1h8m30.575234s] [200] /apis/console.openshift.io/v1/consolenotifications?watch=true&resourceVersion=79435                                                                                                                                                                         [kube:admin]
15:02:13 [ WATCH][58m50.081621s] [200]  /apis/console.openshift.io/v1/consolelinks?watch=true&resourceVersion=83292                                                                                                                                                                                 [kube:admin]
15:08:26 [ WATCH][52m36.517786s] [200]  /apis/apiregistration.k8s.io/v1/apiservices?watch=true&resourceVersion=85827                                                                                                                                                                                [kube:admin]
15:14:16 [ WATCH][46m47.349073s] [200]  /apis/console.openshift.io/v1/consolequickstarts?watch=true&resourceVersion=88167                                                                                                                                                                           [kube:admin]
15:24:42 [ WATCH][36m20.76974s] [200]   /apis/config.openshift.io/v1/clusterversions?watch=true&fieldSelector=metadata.name%3Dversion                                                                                                                                                               [kube:admin]
15:49:29 [ WATCH][9m51.001757s] [200]   /api/v1/namespaces/openshift-kube-scheduler-operator/endpoints?allowWatchBookmarks=true&resourceVersion=102447&timeout=9m51s&timeoutSeconds=591&watch=true                                                                                                  [system:serviceaccount:openshift-monitoring:prometheus-k8s]
15:49:30 [ WATCH][9m55.001174s] [200]   /apis/config.openshift.io/v1/ingresses?allowWatchBookmarks=true&resourceVersion=102455&timeout=9m55s&timeoutSeconds=595&watch=true                                                                                                                          [system:serviceaccount:openshift-apiserver-operator:openshift-apiserver-operator]
15:49:37 [ WATCH][9m57.000757s] [200]   /api/v1/namespaces/openshift-marketplace/pods?allowWatchBookmarks=true&resourceVersion=102513&timeout=9m57s&timeoutSeconds=597&watch=true                                                                                                                   [system:serviceaccount:openshift-monitoring:prometheus-k8s]
15:49:40 [ WATCH][9m50.001387s] [200]   /api/v1/namespaces?allowWatchBookmarks=true&resourceVersion=102501&timeoutSeconds=590&watch=true                                                                                                                                                            [system:serviceaccount:openshift-monitoring:prometheus-operator]
2021-10-25 2021-10-25 Contributors: Robert Bohne