Manual Kubernetes Deployment

Instruction set for deploying Kubernetes via Rancher on OpenStack using the out-of tree cloud control manager.

Preface

Since a few years, Kubernetes is trying to move away from in-tree support for cloud provider to out-of-tree or external support. See https://kubernetes.io/blog/2019/04/17/the-future-of-cloud-providers-in-kubernetes/ This essentially means that instead of containing the code for interacting with a given cloud provider in the main Kubernetes repository, the cloud controller manager takes over this task instead. This controller can then be expanded with a plugin to handle a respective cloud provider. OpenStack in-tree support has recently been deprecated such that future deployments of Kubernetes have to use the out-of-tree approach.

The code for the official OpenStack plugin for the cloud control manager can be found here: https://github.com/kubernetes/cloud-provider-openstack Furthermore, the configurations shown later in this guide are either based on the official OpenStack plugin or on this repo https://github.com/rootsami/terraform-rancher2

These instructions were tested on both Rancher2.5 and Rancher2.6 for Kubernetes 1.20 and 1.23.

Preparation

This must be done once and can be used to deploy any number of Kubernetes cluster.

Kubectl must be installed for later configuration of the cluster.

(Optional) Setup OpenStack CLI

The OpenStack CLI can be used instead of the Horizon web interface for configuring OpenStack. https://github.com/openstack/python-openstackclient

Install via pip

pip install python-openstackclient

Set the required variables

export OS_AUTH_URL="https://api.prod.cloud.gwdg.de:5000/v3"
export OS_IDENTITY_API_VERSION= "3"
export OS_PROJECT_NAME=
export OS_PROJECT_DOMAIN_NAME= "GWDG"
export OS_USERNAME=
export OS_USER_DOMAIN_NAME= "GWDG"
export OS_PASSWORD=

Project name is the name of the OpenStack project that is shown at the top when logged in to OpenStack horizon. Username and password are your username and password used for logging in to OpenStack. If password is not provided, the CLI will query it for every command. When using export to set the password make sure that the machine is secure or that the entry is removed from the shell history.

Enable OpenStack Node Driver

The OpenStack Node Driver must be enabled in Rancher for the node deployment to work.

Rancher2.5

In the web UI -> Tools -> Drivers -> Node Drivers -> Select OpenStack and activate

Rancher2.6

In the web UI -> Open Left sidebar -> Cluster Management -> Drivers -> Node Drivers -> Select OpenStack and activate it

Security group

OpenStack web UI -> Network -> Security Groups -> Create a new group called “k8s-node” -> Manage its rules and add the following rules. CIDR should always be set as 10.254.1.0/24 as this is the internal IP range used by OpenStack for its nodes and it prevents external access. This security group should also be added to the Rancher server hosts. If Rancher is not hosted via OpenStack on this IP range, add all rules again with the CIDR of the Rancher server. Rules:

Protocol	Port(s)
TCP	22 (SSH)
TCP	80 (HTTP)
TCP	443 (HTTPS)
TCP	2376
TCP	2379
TCP	2380
TCP	6443
TCP	6783
TCP	6784
TCP	8443
TCP	8472
TCP	9099
TCP	9100
TCP	9433
TCP	9913
TCP	10248
TCP	10250
TCP	10254
TCP	30000-32767
UDP	6783
UDP	6784
UDP	8443
UDP	8472
UDP	30000-32767

This prevents future headaches by opening all ports that Rancher might need according to its documentation: https://rancher.com/docs/rancher/v2.6/en/installation/requirements/ports/

via CLI

openstack security group create k8s-node
openstack security group rule create --remote-ip 10.254.1.0/24 --dst-port PORT --protocol PROTOCOL k8s-node

Insert for PORT the number specified above. For port ranges use start:end. For PROTOCOL insert either TCP or UDP as specified in the table.

Node Template

Find the following values in the OpenStack web UI or via the CLI and note them down.

flavorName:

OpenStack web UI -> Compute -> Instances -> Launch Instance -> Flavor -> choose one and note down the name, for example: ‘m1.medium’

Or via CLI -> openstack flavor list

imageName:

OpenStack web UI -> Compute -> Images -> choose one, for example: ‘Ubuntu 20.04.3 Server x86_64 (ssd)’

Or via CLI -> openstack image list

netId:

OpenStack web UI -> Network -> select Networks -> pick a private network -> select overview -> ID

Or via CLI -> openstack network list -> private network ID

password:

The password for your OpenStack account you use to login.

tenantId:

OpenStack web UI -> Identity -> Projects -> Project ID

Or via CLI -> openstack project list -> ID

username:

The username for your OpenStack account you sue to login.

Node template

Rancher2.5

Rancher web UI -> click profile picture in top right -> Node Templates -> Add Template -> OpenStack -> fill the template according to the following template while filling in the values gathered above

Rancher2.6

In the web UI -> Open Left sidebar -> Cluster Management -> REK1 Configuration -> Node Templates -> Add Template -> OpenStack -> fill the template according to the following template while filling in the values gathered above

Node template template

authURL: https://api.prod.cloud.gwdg.de:5000/v3
domainName: GWDG
endpointType: publicURL
flavorName:
imageName:
netId:
password:
region: RegionOne
secGroups: k8s-node,default
sshUser: cloud
tenantDomainName: GWDG
tenantId:
username:
volumeName: rancher-volume
volumeSize: 0
volumeType: ssd

Kubernetes Cluster Deployment

Kubernetes Cluster deployment on OpenStack using Rancher.

Cluster Creation

Rancher2.5

Rancher web UI -> Global context -> Add Cluster -> OpenStack

Rancher2.6

Rancher web UI -> Home -> Create -> OpenStack

Set a name for the cluster. Add at least one node pool using the node template created above. Set for at least one node pool control plane and etcd to active. Give the node a fitting name. This will be used as a prefix for nodes of this pool, for example CLUSTERNAME-master and CLUSTERNAME-worker, where CLUSTERNAME is the name of the Kubernetes cluster you are about to create.

Under Kubernetes Options select the desired Kubernetes Version and under Cloud Provider select External.

Press create. This will create node instance on OpenStack using the Node template and spin up a Kubernetes cluster. This will take some time, usually 10 to 15 min.

When viewed in Rancher, all nodes with be marked with a taint “uninitialized=NoSchedule”.

To finish initialization, the external cloud control manager for OpenStack must be configured and deployed.

OpenStack cloud control manager

The next steps require kubectl to be installed and working.

Rancher2.5

In the Rancher Web UI -> Select the new cluster -> Top right Kubeconfig File

Rancher2.6

In the Rancher Web UI -> Select the new cluster -> Top right Copy or Download Kubeconfig File

Copy these settings to a file called config and place it under ~/.kube/config on your host. See [[#Managing multiple kubeconfigs]] for how to handle multiple kubeconfigs.

Next gather the details required to fill out the cloud conf file.

username:

Your OpenStack username.

password:

Your OpenStack password.

tenant-id:

Your OpenStack project id, see tenantId above.

subnet-id:

OpenStack Web UI -> Network -> Networks -> private network -> Subnets -> ID

Or via CLI -> openstack subnet list -> ID

floating-network-id:

OpenStack Web UI -> Network -> Networks -> public -> Overview -> ID

Or via CLI -> openstack network list -> public network ID

router-id:

OpenStack Web UI -> Network -> Routers -> select router -> Overview -> ID

Or via CLI -> openstack router list -> ID

Cloud conf

Create a file called cloud.conf and fill it with the template below. Then fill in the open fields as described above.

[Global]
auth-url = https://api.prod.cloud.gwdg.de:5000/v3
username =
password =
region = RegionOne
tenant-id =
domain-name = GWDG

[BlockStorage]
ignore-volume-az = true
trust-device-path = false

[Networking]
public-network-name = public

[LoadBalancer]
use-octavia = false
subnet-id =
floating-network-id =
create-monitor = false
manage-security-groups = true
monitor-max-retries = 0
enabled = true
lb-version = v2
lb-provider = haproxy

[Route]
router-id =

[Metadata]
request-timeout = 0

Afterwards use kubectl to create a secret from this config:

kubectl create secret -n kube-system generic cloud-config --from-file=cloud.conf

Deploying the cloud controller manager

Create yaml files with the following contents:

cloud-controller-manager-role-bindings.yaml

apiVersion: v1
items:
- apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRoleBinding
  metadata:
    name: system:cloud-node-controller
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: system:cloud-node-controller
  subjects:
  - kind: ServiceAccount
    name: cloud-node-controller
    namespace: kube-system
- apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRoleBinding
  metadata:
    name: system:pvl-controller
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: system:pvl-controller
  subjects:
  - kind: ServiceAccount
    name: pvl-controller
    namespace: kube-system
- apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRoleBinding
  metadata:
    name: system:cloud-controller-manager
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: system:cloud-controller-manager
  subjects:
  - kind: ServiceAccount
    name: cloud-controller-manager
    namespace: kube-system
kind: List
metadata: {}

cloud-controller-manager-roles.yaml

apiVersion: v1
items:
- apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    name: system:cloud-controller-manager
  rules:
  - apiGroups:
    - coordination.k8s.io
    resources:
    - leases
    verbs:
    - get
    - create
    - update
  - apiGroups:
    - ""
    resources:
    - events
    verbs:
    - create
    - patch
    - update
  - apiGroups:
    - ""
    resources:
    - nodes
    verbs:
    - '*'
  - apiGroups:
    - ""
    resources:
    - nodes/status
    verbs:
    - patch
  - apiGroups:
    - ""
    resources:
    - services
    verbs:
    - list
    - patch
    - update
    - watch
  - apiGroups:
    - ""
    resources:
    - serviceaccounts
    verbs:
    - create
    - get
  - apiGroups:
    - ""
    resources:
    - serviceaccounts/token
    verbs:
    - create
  - apiGroups:
    - ""
    resources:
    - persistentvolumes
    verbs:
    - '*'
  - apiGroups:
    - ""
    resources:
    - endpoints
    verbs:
    - create
    - get
    - list
    - watch
    - update
  - apiGroups:
    - ""
    resources:
    - configmaps
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - ""
    resources:
    - secrets
    verbs:
    - list
    - get
    - watch
- apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    name: system:cloud-node-controller
  rules:
  - apiGroups:
    - ""
    resources:
    - nodes
    verbs:
    - '*'
  - apiGroups:
    - ""
    resources:
    - nodes/status
    verbs:
    - patch
  - apiGroups:
    - ""
    resources:
    - events
    verbs:
    - create
    - patch
    - update
- apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    name: system:pvl-controller
  rules:
  - apiGroups:
    - ""
    resources:
    - persistentvolumes
    verbs:
    - '*'
  - apiGroups:
    - ""
    resources:
    - events
    verbs:
    - create
    - patch
    - update
kind: List
metadata: {}

openstack-cloud-controller-manager-ds.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: cloud-controller-manager
  namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: openstack-cloud-controller-manager
  namespace: kube-system
  labels:
    k8s-app: openstack-cloud-controller-manager
spec:
  selector:
    matchLabels:
      k8s-app: openstack-cloud-controller-manager
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        k8s-app: openstack-cloud-controller-manager
    spec:
      nodeSelector:
        node-role.kubernetes.io/controlplane: "true"
      securityContext:
        runAsUser: 1001
      tolerations:
      - key: node.cloudprovider.kubernetes.io/uninitialized
        value: "true"
        effect: NoSchedule
      - key: node-role.kubernetes.io/controlplane
        effect: NoSchedule
        value: "true"
      - key: node-role.kubernetes.io/etcd
        effect: NoExecute
        value: "true"
      serviceAccountName: cloud-controller-manager
      containers:
        - name: openstack-cloud-controller-manager
          image: docker.io/k8scloudprovider/openstack-cloud-controller-manager:latest
          args:
            - /bin/openstack-cloud-controller-manager
            - --v=1
            - --cloud-config=$(CLOUD_CONFIG)
            - --cloud-provider=openstack
            - --use-service-account-credentials=true
            - --bind-address=127.0.0.1
            - --cluster-cidr=10.254.1.0/24
          volumeMounts:
            - mountPath: /etc/kubernetes/pki
              name: k8s-certs
              readOnly: true
            - mountPath: /etc/ssl/certs
              name: ca-certs
              readOnly: true
            - mountPath: /etc/config
              name: cloud-config-volume
              readOnly: true
          resources:
            requests:
              cpu: 200m
          env:
            - name: CLOUD_CONFIG
              value: /etc/config/cloud.conf
      hostNetwork: true
      volumes:
      - hostPath:
          path: /etc/kubernetes/pki
          type: DirectoryOrCreate
        name: k8s-certs
      - hostPath:
          path: /etc/ssl/certs
          type: DirectoryOrCreate
        name: ca-certs
      - name: cloud-config-volume
        secret:
          secretName: cloud-config

Notice that the third file sets the flag –cluster-cidr=10.254.1.0/24. If used in a different ip range, this must be updated.

Apply the three files using kubectl

kubectl apply -f cloud-controller-manager-roles.yaml
kubectl apply -f cloud-controller-manager-role-bindings.yaml
kubectl apply -f openstack-cloud-controller-manager-ds.yaml

This should after a short time update the nodes as shown in Rancher to become initialized and be ready for deployment. This further configures how to deploy load balancer using the GWDG OpenStack deployment and its rather outdated load balancer system.

Cinder CSI Driver

To enable Kubernetes to satisfy PVC using Cinder volumes, the cinder csi plugin must be installed.

Create another yaml file for this.

cinder-csi-plugin.yaml

# This YAML file contains RBAC API objects,
# which are necessary to run csi controller plugin

apiVersion: v1
kind: ServiceAccount
metadata:
  name: csi-cinder-controller-sa
  namespace: kube-system

---
# external attacher
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-attacher-role
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "patch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["csinodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments"]
    verbs: ["get", "list", "watch", "patch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments/status"]
    verbs: ["patch"]


---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-attacher-binding
subjects:
  - kind: ServiceAccount
    name: csi-cinder-controller-sa
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: csi-attacher-role
  apiGroup: rbac.authorization.k8s.io

---
# external Provisioner
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-provisioner-role
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["csinodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshots"]
    verbs: ["get", "list"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotcontents"]
    verbs: ["get", "list"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-provisioner-binding
subjects:
  - kind: ServiceAccount
    name: csi-cinder-controller-sa
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: csi-provisioner-role
  apiGroup: rbac.authorization.k8s.io

---
# external snapshotter
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-snapshotter-role
rules:
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
  # Secret permission is optional.
  # Enable it if your driver needs secret.
  # For example, `csi.storage.k8s.io/snapshotter-secret-name` is set in VolumeSnapshotClass.
  # See https://kubernetes-csi.github.io/docs/secrets-and-credentials.html for more details.
  #  - apiGroups: [""]
  #    resources: ["secrets"]
  #    verbs: ["get", "list"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotcontents"]
    verbs: ["create", "get", "list", "watch", "update", "delete"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotcontents/status"]
    verbs: ["update"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-snapshotter-binding
subjects:
  - kind: ServiceAccount
    name: csi-cinder-controller-sa
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: csi-snapshotter-role
  apiGroup: rbac.authorization.k8s.io
---

# External Resizer
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-resizer-role
rules:
  # The following rule should be uncommented for plugins that require secrets
  # for provisioning.
  # - apiGroups: [""]
  #   resources: ["secrets"]
  #   verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "patch"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims/status"]
    verbs: ["update", "patch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-resizer-binding
subjects:
  - kind: ServiceAccount
    name: csi-cinder-controller-sa
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: csi-resizer-role
  apiGroup: rbac.authorization.k8s.io

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: kube-system
  name: external-resizer-cfg
rules:
- apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["get", "watch", "list", "delete", "update", "create"]

---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-resizer-role-cfg
  namespace: kube-system
subjects:
  - kind: ServiceAccount
    name: csi-cinder-controller-sa
    namespace: kube-system
roleRef:
  kind: Role
  name: external-resizer-cfg
  apiGroup: rbac.authorization.k8s.io

---
# This YAML file contains CSI Controller Plugin Sidecars
# external-attacher, external-provisioner, external-snapshotter
# external-resize, liveness-probe

kind: Service
apiVersion: v1
metadata:
  name: csi-cinder-controller-service
  namespace: kube-system
  labels:
    app: csi-cinder-controllerplugin
spec:
  selector:
    app: csi-cinder-controllerplugin
  ports:
    - name: dummy
      port: 12345

---
kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: csi-cinder-controllerplugin
  namespace: kube-system
spec:
  serviceName: "csi-cinder-controller-service"
  replicas: 1
  selector:
    matchLabels:
      app: csi-cinder-controllerplugin
  template:
    metadata:
      labels:
        app: csi-cinder-controllerplugin
    spec:
      serviceAccount: csi-cinder-controller-sa
      containers:
        - name: csi-attacher
          image: k8s.gcr.io/sig-storage/csi-attacher:v3.1.0
          args:
            - "--csi-address=$(ADDRESS)"
            - "--timeout=3m"
          env:
            - name: ADDRESS
              value: /var/lib/csi/sockets/pluginproxy/csi.sock
          imagePullPolicy: "IfNotPresent"
          volumeMounts:
            - name: socket-dir
              mountPath: /var/lib/csi/sockets/pluginproxy/
        - name: csi-provisioner
          image: k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1
          args:
            - "--csi-address=$(ADDRESS)"
            - "--timeout=3m"
            - "--default-fstype=ext4"
            - "--feature-gates=Topology=true"
            - "--extra-create-metadata"
          env:
            - name: ADDRESS
              value: /var/lib/csi/sockets/pluginproxy/csi.sock
          imagePullPolicy: "IfNotPresent"
          volumeMounts:
            - name: socket-dir
              mountPath: /var/lib/csi/sockets/pluginproxy/
        - name: csi-snapshotter
          image: k8s.gcr.io/sig-storage/csi-snapshotter:v2.1.3
          args:
            - "--csi-address=$(ADDRESS)"
            - "--timeout=3m"
          env:
            - name: ADDRESS
              value: /var/lib/csi/sockets/pluginproxy/csi.sock
          imagePullPolicy: Always
          volumeMounts:
            - mountPath: /var/lib/csi/sockets/pluginproxy/
              name: socket-dir
        - name: csi-resizer
          image: k8s.gcr.io/sig-storage/csi-resizer:v1.1.0
          args:
            - "--csi-address=$(ADDRESS)"
            - "--timeout=3m"
            - "--handle-volume-inuse-error=false"
          env:
            - name: ADDRESS
              value: /var/lib/csi/sockets/pluginproxy/csi.sock
          imagePullPolicy: "IfNotPresent"
          volumeMounts:
            - name: socket-dir
              mountPath: /var/lib/csi/sockets/pluginproxy/
        - name: liveness-probe
          image: k8s.gcr.io/sig-storage/livenessprobe:v2.1.0
          args:
            - "--csi-address=$(ADDRESS)"
          env:
            - name: ADDRESS
              value: /var/lib/csi/sockets/pluginproxy/csi.sock
          volumeMounts:
            - mountPath: /var/lib/csi/sockets/pluginproxy/
              name: socket-dir
        - name: cinder-csi-plugin
          image: docker.io/k8scloudprovider/cinder-csi-plugin:latest
          args:
            - /bin/cinder-csi-plugin
            - "--nodeid=$(NODE_ID)"
            - "--endpoint=$(CSI_ENDPOINT)"
            - "--cloud-config=$(CLOUD_CONFIG)"
            - "--cluster=$(CLUSTER_NAME)"
          env:
            - name: NODE_ID
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: CSI_ENDPOINT
              value: unix://csi/csi.sock
            - name: CLOUD_CONFIG
              value: /etc/config/cloud.conf
            - name: CLUSTER_NAME
              value: kubernetes
          imagePullPolicy: "IfNotPresent"
          ports:
            - containerPort: 9808
              name: healthz
              protocol: TCP
          # The probe
          livenessProbe:
            failureThreshold: 5
            httpGet:
              path: /healthz
              port: healthz
            initialDelaySeconds: 10
            timeoutSeconds: 10
            periodSeconds: 60
          volumeMounts:
            - name: socket-dir
              mountPath: /csi
            - name: secret-cinderplugin
              mountPath: /etc/config
              readOnly: true
      volumes:
        - name: socket-dir
          emptyDir:
        - name: secret-cinderplugin
          secret:
            secretName: cloud-config
---
# This YAML defines all API objects to create RBAC roles for csi node plugin.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: csi-cinder-node-sa
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-nodeplugin-role
rules:
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-nodeplugin-binding
subjects:
  - kind: ServiceAccount
    name: csi-cinder-node-sa
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: csi-nodeplugin-role
  apiGroup: rbac.authorization.k8s.io

---
# This YAML file contains driver-registrar & csi driver nodeplugin API objects,
# which are necessary to run csi nodeplugin for cinder.

kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: csi-cinder-nodeplugin
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: csi-cinder-nodeplugin
  template:
    metadata:
      labels:
        app: csi-cinder-nodeplugin
    spec:
      tolerations:
        - operator: Exists
      serviceAccount: csi-cinder-node-sa
      hostNetwork: true
      containers:
        - name: node-driver-registrar
          image: k8s.gcr.io/sig-storage/csi-node-driver-registrar:v1.3.0
          args:
            - "--csi-address=$(ADDRESS)"
            - "--kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)"
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "rm -rf /registration/cinder.csi.openstack.org /registration/cinder.csi.openstack.org-reg.sock"]
          env:
            - name: ADDRESS
              value: /csi/csi.sock
            - name: DRIVER_REG_SOCK_PATH
              value: /var/lib/kubelet/plugins/cinder.csi.openstack.org/csi.sock
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          imagePullPolicy: "IfNotPresent"
          volumeMounts:
            - name: socket-dir
              mountPath: /csi
            - name: registration-dir
              mountPath: /registration
        - name: liveness-probe
          image: k8s.gcr.io/sig-storage/livenessprobe:v2.1.0
          args:
            - --csi-address=/csi/csi.sock
          volumeMounts:
            - name: socket-dir
              mountPath: /csi
        - name: cinder-csi-plugin
          securityContext:
            privileged: true
            capabilities:
              add: ["SYS_ADMIN"]
            allowPrivilegeEscalation: true
          image: docker.io/k8scloudprovider/cinder-csi-plugin:latest
          args:
            - /bin/cinder-csi-plugin
            - "--nodeid=$(NODE_ID)"
            - "--endpoint=$(CSI_ENDPOINT)"
            - "--cloud-config=$(CLOUD_CONFIG)"
          env:
            - name: NODE_ID
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: CSI_ENDPOINT
              value: unix://csi/csi.sock
            - name: CLOUD_CONFIG
              value: /etc/config/cloud.conf
          imagePullPolicy: "IfNotPresent"
          ports:
            - containerPort: 9808
              name: healthz
              protocol: TCP
          # The probe
          livenessProbe:
            failureThreshold: 5
            httpGet:
              path: /healthz
              port: healthz
            initialDelaySeconds: 10
            timeoutSeconds: 3
            periodSeconds: 10
          volumeMounts:
            - name: socket-dir
              mountPath: /csi
            - name: kubelet-dir
              mountPath: /var/lib/kubelet
              mountPropagation: "Bidirectional"
            - name: pods-probe-dir
              mountPath: /dev
              mountPropagation: "HostToContainer"
            - name: secret-cinderplugin
              mountPath: /etc/config
              readOnly: true
      volumes:
        - name: socket-dir
          hostPath:
            path: /var/lib/kubelet/plugins/cinder.csi.openstack.org
            type: DirectoryOrCreate
        - name: registration-dir
          hostPath:
            path: /var/lib/kubelet/plugins_registry/
            type: Directory
        - name: kubelet-dir
          hostPath:
            path: /var/lib/kubelet
            type: Directory
        - name: pods-probe-dir
          hostPath:
            path: /dev
            type: Directory
        - name: secret-cinderplugin
          secret:
            secretName: cloud-config

---
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  name: cinder.csi.openstack.org
spec:
  attachRequired: true
  podInfoOnMount: true
  volumeLifecycleModes:
  - Persistent
  - Ephemeral

---
# This YAML file contains StorageClass definition
# and makes it default storageclass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-sc-cinderplugin
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: cinder.csi.openstack.org

And apply it using

kubectl apply -f cinder-csi-plugin.yaml

This creates a storage class called csi-sc-cinderplugin that should be used for creating volumes.

Updating cloud.conf

Should it be necessary to update cloud.conf it can be done like this.

Create or update a file called cloud.conf as described above. Replace the existing secret using this:

kubectl create secret generic cloud-config --from-file=cloud.conf --dry-run -n kube-system -o yaml | kubectl apply -n kube-system -f -

Next find the pod running the openstack control manager using

kubectl get pod -n kube-system

and delete it

kubectl delete pod openstack-cloud-controller-manager-xxxx

xxxx will be some random code that needs to be identified via the first command. This will cause the pod to be recreated and to load the new cloud.conf file.

Should any problem occur, the logs of the openstack-cloud-controller-manager pod can also help troubleshoot the issue.

kubectl logs openstack-cloud-controller-manager-xxxx

Validate that it works

The next steps are optional and are just to confirm that the cluster can: - Deploy workloads - Deploy Load balancer that connect to OpenStack - Expose a service - Claim a volume

Deploy a workload

Rancher2.5

Rancher UI -> Global context -> Select your new cluster -> Projects/Namespaces -> Under Project: Default press Add Namespace -> name it ‘test’ -> Create -> Click Project: Default -> Workloads overview should be open -> Deploy -> name it “test-hello” -> as docker image set “rancher/hello-world” -> Launch

Rancher2.6

Rancher UI -> Home -> Select your new cluster -> Projects/Namespaces -> Under Project: Default press Create Namespace -> Name it “test” -> Create -> Workload -> Create -> Deployment -> Namespace to “test” -> Name the workload “test-hello” -> as docker image set “rancher/hello-world” -> Create

Observe that it should be scheduled after a few seconds.

Deploy a load balancer

Rancher2.5

Select Apps -> Launch -> search for Nginx and select NGINX Ingress Controller -> and click it -> namespace to use existing -> test -> Launch

Click on the nginx app and observe that it is deployed and a load balancer is created in the cluster.

Rancher2.6

Select Apps & Marketplace -> search for Nginx and select NGINX Ingress Controller -> click it -> Install -> Set namespace to test -> Next -> change ingress class to “nginx-test” -> Install

After about a minute the load balancer should also appear in OpenStack. Find the load balancer under OpenStack web UI -> Network -> Load Balancers

Via the CLI -> The OpenStack CLI only supports loadbalancers via octavia, the API can still be accessed via manual requests. See [[#Access loadbalancers outside of the web UI]]

If you currently have no free floating IPs, the load balancer setup will not complete. If the load balancer has not claimed a floating IP, release one: OpenStack web UI -> Network -> Floating IPs -> actions -> release floating IP Be careful not to release the floating IP already claimed by the load balancer.

Via the CLI -> Use openstack floating ip list and openstack floating ip delete ID to release one floating IP

Once the load balancer in OpenStack is ready and has claimed an IP, the nginx app in Rancher should also be ready.

Make a note of the floating ip that was claimed.

Expose a service

Rancher2.5

From the nginx App overview -> Resources -> Workloads -> Load Balancing -> Observe that a L4 Balancer already exists -> Add Ingress -> name it “test-ingress” -> Set namespace to “test” -> Set path to “/” -> choose “test-hello” as target -> Set port as 80 -> Open Labels & Annotations drop down -> Add annotation with key “kubernetes.io/ingress.class” and value “nginx” -> Save A hostname ending in sslip.io will be generated.

Rancher2.6

First create a service Service Discovery -> Services -> Create -> ClusterIP -> name it “test-service” -> listening port and target port to 80 -> Selectors -> Set key as “workload.user.cattle.io/workloadselector” and value to “apps.deployment-test-test-hello” -> Create -> Observe that “test-hello” appears as a pod for that service Then create an ingress Service Discovery -> Ingresses -> Create -> name it “test-ingress” -> set request host to “http://test-ingress.test.FLOATING-IP.sslip.io”, where FLOATING-IP is the floating IP noted earlier -> set Path prefix to / -> Select “test-service” as target service and set port to 80 -> Labels&Annotations -> Add Annotation -> Set key to “kubernetes.io/ingress.class” and value “nginx-test” -> Create

The hostname should point to a web page with the Rancher logo writing “Hello World!”. It might take a moment for the link to work. Click the link and confirm that the workload is exposed.

Nginx webhook validation workaround

When trying to create the above described ingress, Rancher might refuse with the error:

Internal error occurred: failed calling webhook “validate.nginx.ingress.kubernetes.io”: an error on the server (“”) has prevented the request from succeeding

A simple workaround for this is to run

kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission

This removes the offending validation hook.

Claiming a volume

Rancher2.5

Select Apps -> Launch -> search for mysql and select it -> namespace to use an existing namespace and pick “test” -> activate PVC and ensure it uses the default storage class that is “csi-sc-cinderplugin” -> Launch

Click on the mysql app and observe that it is deployed and a volume claim is created in the cluster.

Rancher2.6

Select Apps&Marketplace -> search for sql and select cockroachdb -> Install -> namespace to “test” -> Next -> Storage per Node to 10Gi -> Install

After a moment the app should deploy and open volume claim(s) that becomes satisfied by volume(s) from OpenStack after a few moments.

In the OpenStack web UI -> Volumes -> Volumes And one volume should have the description “Created by OpenStack Cinder CSI driver”

Or via the CLI -> openstack volume list

Cleanup

Delete the namespace test to cleanup all created resources.

Rancher2.5

Rancher UI -> Global context -> Select cluster -> Projects/Namespaces -> check the box next to the test namepsace -> Delete -> Confirm

Rancher2.6

Rancher UI -> Cluster -> Projects/Namespaces -> check the box next to the test namepsace -> Hold CTRL and press Delete

This also removes the load balancer and volumes in OpenStack. CockroachDB might not remove all its volumes, check manually and remove the remaining volumes.

Appendix

Managing multiple kubeconfigs

Use a tool such as kubectx to manage what context to load. Option 1: One large config Append any new config to the config file. Option 2: Multiple configs Add all files to $KUBECONFIG. You can do so in your shell config by adding

export KUBECONFIG=$KUBECONFIG:$HOME/.kube/config2

Access loadbalancers outside of the web UI

This uses the keystoneauth1 package for authentication and making API requests.

pip install keystoneauth1

import keystoneauth1
# Fill in your credentials
my_username = ""
my_password = ""
my_project_id = ""
# Setting up a session
password_method = keystoneauth1.identity.v3.PasswordMethod(username=my_username,
password=my_password,
user_domain_name="GWDG")
auth = keystoneauth1.identity.v3.Auth(auth_url="https://api.prod.cloud.gwdg.de:5000/v3", auth_methods=[password_method], project_id=my_project_id, project_domain_name="GWDG")
sess = keystoneauth1.session.Session(auth=auth)
# API docs: https://wiki.openstack.org/wiki/Neutron/LBaaS/API_2.0
lbaasv2_api = "https://api.prod.cloud.gwdg.de:9696/v2.0"
# List all Load Balancers
r = sess.request(f"{lbaasv2_api}/lbaas/loadbalancers", "GET",
 headers={"Accept": "application/json"})
print(r.text)