***************************** Manual Kubernetes Deployment ***************************** Instruction set for deploying Kubernetes via Rancher on OpenStack using the out-of tree cloud control manager. Preface ######## Since a few years, Kubernetes is trying to move away from in-tree support for cloud provider to out-of-tree or external support. See https://kubernetes.io/blog/2019/04/17/the-future-of-cloud-providers-in-kubernetes/ This essentially means that instead of containing the code for interacting with a given cloud provider in the main Kubernetes repository, the cloud controller manager takes over this task instead. This controller can then be expanded with a plugin to handle a respective cloud provider. OpenStack in-tree support has recently been deprecated such that future deployments of Kubernetes have to use the out-of-tree approach. The code for the official OpenStack plugin for the cloud control manager can be found here: https://github.com/kubernetes/cloud-provider-openstack Furthermore, the configurations shown later in this guide are either based on the official OpenStack plugin or on this repo https://github.com/rootsami/terraform-rancher2 These instructions were tested on both Rancher2.5 and Rancher2.6 for Kubernetes 1.20 and 1.23. Preparation ############# This must be done once and can be used to deploy any number of Kubernetes cluster. Kubectl must be installed for later configuration of the cluster. (Optional) Setup OpenStack CLI ****************************** The OpenStack CLI can be used instead of the Horizon web interface for configuring OpenStack. https://github.com/openstack/python-openstackclient Install via pip .. code:: console pip install python-openstackclient Set the required variables .. code:: console export OS_AUTH_URL="https://api.prod.cloud.gwdg.de:5000/v3" export OS_IDENTITY_API_VERSION= "3" export OS_PROJECT_NAME= export OS_PROJECT_DOMAIN_NAME= "GWDG" export OS_USERNAME= export OS_USER_DOMAIN_NAME= "GWDG" export OS_PASSWORD= Project name is the name of the OpenStack project that is shown at the top when logged in to OpenStack horizon. Username and password are your username and password used for logging in to OpenStack. If password is not provided, the CLI will query it for every command. When using export to set the password make sure that the machine is secure or that the entry is removed from the shell history. Enable OpenStack Node Driver ***************************** The OpenStack Node Driver must be enabled in Rancher for the node deployment to work. Rancher2.5 ---------- In the web UI -> Tools -> Drivers -> Node Drivers -> Select OpenStack and activate Rancher2.6 ---------- In the web UI -> Open Left sidebar -> Cluster Management -> Drivers -> Node Drivers -> Select OpenStack and activate it Security group *************** OpenStack web UI -> Network -> Security Groups -> Create a new group called "k8s-node" -> Manage its rules and add the following rules. CIDR should always be set as 10.254.1.0/24 as this is the internal IP range used by OpenStack for its nodes and it prevents external access. This security group should also be added to the Rancher server hosts. If Rancher is not hosted via OpenStack on this IP range, add all rules again with the CIDR of the Rancher server. Rules: +----------+-------------+ | Protocol | Port(s) | +==========+=============+ | TCP | 22 (SSH) | +----------+-------------+ | TCP | 80 (HTTP) | +----------+-------------+ | TCP | 443 (HTTPS) | +----------+-------------+ | TCP | 2376 | +----------+-------------+ | TCP | 2379 | +----------+-------------+ | TCP | 2380 | +----------+-------------+ | TCP | 6443 | +----------+-------------+ | TCP | 6783 | +----------+-------------+ | TCP | 6784 | +----------+-------------+ | TCP | 8443 | +----------+-------------+ | TCP | 8472 | +----------+-------------+ | TCP | 9099 | +----------+-------------+ | TCP | 9100 | +----------+-------------+ | TCP | 9433 | +----------+-------------+ | TCP | 9913 | +----------+-------------+ | TCP | 10248 | +----------+-------------+ | TCP | 10250 | +----------+-------------+ | TCP | 10254 | +----------+-------------+ | TCP | 30000-32767 | +----------+-------------+ | UDP | 6783 | +----------+-------------+ | UDP | 6784 | +----------+-------------+ | UDP | 8443 | +----------+-------------+ | UDP | 8472 | +----------+-------------+ | UDP | 30000-32767 | +----------+-------------+ This prevents future headaches by opening all ports that Rancher might need according to its documentation: https://rancher.com/docs/rancher/v2.6/en/installation/requirements/ports/ via CLI .. code:: console openstack security group create k8s-node openstack security group rule create --remote-ip 10.254.1.0/24 --dst-port PORT --protocol PROTOCOL k8s-node Insert for PORT the number specified above. For port ranges use start:end. For PROTOCOL insert either TCP or UDP as specified in the table. Node Template ************** Find the following values in the OpenStack web UI or via the CLI and note them down. flavorName: ----------- OpenStack web UI -> Compute -> Instances -> Launch Instance -> Flavor -> choose one and note down the name, for example: 'm1.medium' Or via CLI -> `openstack flavor list` imageName: ----------- OpenStack web UI -> Compute -> Images -> choose one, for example: 'Ubuntu 20.04.3 Server x86_64 (ssd)' Or via CLI -> `openstack image list` netId: ----------- OpenStack web UI -> Network -> select Networks -> pick a private network -> select overview -> ID Or via CLI -> `openstack network list` -> private network ID password: ----------- The password for your OpenStack account you use to login. tenantId: ----------- OpenStack web UI -> Identity -> Projects -> Project ID Or via CLI -> `openstack project list` -> ID username: ----------- The username for your OpenStack account you sue to login. Node template ============= Rancher2.5 ----------- Rancher web UI -> click profile picture in top right -> Node Templates -> Add Template -> OpenStack -> fill the template according to the following template while filling in the values gathered above Rancher2.6 ----------- In the web UI -> Open Left sidebar -> Cluster Management -> REK1 Configuration -> Node Templates -> Add Template -> OpenStack -> fill the template according to the following template while filling in the values gathered above Node template template ---------------------- .. code:: console authURL: https://api.prod.cloud.gwdg.de:5000/v3 domainName: GWDG endpointType: publicURL flavorName: imageName: netId: password: region: RegionOne secGroups: k8s-node,default sshUser: cloud tenantDomainName: GWDG tenantId: username: volumeName: rancher-volume volumeSize: 0 volumeType: ssd Kubernetes Cluster Deployment ############################# Kubernetes Cluster deployment on OpenStack using Rancher. Cluster Creation **************** Rancher2.5 ---------- Rancher web UI -> Global context -> Add Cluster -> OpenStack Rancher2.6 ---------- Rancher web UI -> Home -> Create -> OpenStack Set a name for the cluster. Add at least one node pool using the node template created above. Set for at least one node pool control plane and etcd to active. Give the node a fitting name. This will be used as a prefix for nodes of this pool, for example `CLUSTERNAME-master` and `CLUSTERNAME-worker`, where CLUSTERNAME is the name of the Kubernetes cluster you are about to create. Under Kubernetes Options select the desired Kubernetes Version and under Cloud Provider select External. Press create. This will create node instance on OpenStack using the Node template and spin up a Kubernetes cluster. This will take some time, usually 10 to 15 min. When viewed in Rancher, all nodes with be marked with a taint "uninitialized=NoSchedule". To finish initialization, the external cloud control manager for OpenStack must be configured and deployed. OpenStack cloud control manager ******************************* The next steps require kubectl to be installed and working. Rancher2.5 ---------- In the Rancher Web UI -> Select the new cluster -> Top right Kubeconfig File Rancher2.6 ---------- In the Rancher Web UI -> Select the new cluster -> Top right Copy or Download Kubeconfig File Copy these settings to a file called config and place it under `~/.kube/config` on your host. See [[#Managing multiple kubeconfigs]] for how to handle multiple kubeconfigs. Next gather the details required to fill out the cloud conf file. username: ========= Your OpenStack username. password: ========= Your OpenStack password. tenant-id: ========== Your OpenStack project id, see tenantId above. subnet-id: ========== OpenStack Web UI -> Network -> Networks -> private network -> Subnets -> ID Or via CLI -> `openstack subnet list` -> ID floating-network-id: ==================== OpenStack Web UI -> Network -> Networks -> public -> Overview -> ID Or via CLI -> `openstack network list` -> public network ID router-id: ========== OpenStack Web UI -> Network -> Routers -> select router -> Overview -> ID Or via CLI -> `openstack router list` -> ID Cloud conf ========== Create a file called cloud.conf and fill it with the template below. Then fill in the open fields as described above. .. code:: ini [Global] auth-url = https://api.prod.cloud.gwdg.de:5000/v3 username = password = region = RegionOne tenant-id = domain-name = GWDG [BlockStorage] ignore-volume-az = true trust-device-path = false [Networking] public-network-name = public [LoadBalancer] use-octavia = false subnet-id = floating-network-id = create-monitor = false manage-security-groups = true monitor-max-retries = 0 enabled = true lb-version = v2 lb-provider = haproxy [Route] router-id = [Metadata] request-timeout = 0 Afterwards use kubectl to create a secret from this config: .. code:: console kubectl create secret -n kube-system generic cloud-config --from-file=cloud.conf Deploying the cloud controller manager ====================================== Create yaml files with the following contents: cloud-controller-manager-role-bindings.yaml .. code:: yaml apiVersion: v1 items: - apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: system:cloud-node-controller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:cloud-node-controller subjects: - kind: ServiceAccount name: cloud-node-controller namespace: kube-system - apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: system:pvl-controller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:pvl-controller subjects: - kind: ServiceAccount name: pvl-controller namespace: kube-system - apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: system:cloud-controller-manager roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:cloud-controller-manager subjects: - kind: ServiceAccount name: cloud-controller-manager namespace: kube-system kind: List metadata: {} cloud-controller-manager-roles.yaml .. code:: yaml apiVersion: v1 items: - apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: system:cloud-controller-manager rules: - apiGroups: - coordination.k8s.io resources: - leases verbs: - get - create - update - apiGroups: - "" resources: - events verbs: - create - patch - update - apiGroups: - "" resources: - nodes verbs: - '*' - apiGroups: - "" resources: - nodes/status verbs: - patch - apiGroups: - "" resources: - services verbs: - list - patch - update - watch - apiGroups: - "" resources: - serviceaccounts verbs: - create - get - apiGroups: - "" resources: - serviceaccounts/token verbs: - create - apiGroups: - "" resources: - persistentvolumes verbs: - '*' - apiGroups: - "" resources: - endpoints verbs: - create - get - list - watch - update - apiGroups: - "" resources: - configmaps verbs: - get - list - watch - apiGroups: - "" resources: - secrets verbs: - list - get - watch - apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: system:cloud-node-controller rules: - apiGroups: - "" resources: - nodes verbs: - '*' - apiGroups: - "" resources: - nodes/status verbs: - patch - apiGroups: - "" resources: - events verbs: - create - patch - update - apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: system:pvl-controller rules: - apiGroups: - "" resources: - persistentvolumes verbs: - '*' - apiGroups: - "" resources: - events verbs: - create - patch - update kind: List metadata: {} openstack-cloud-controller-manager-ds.yaml .. code:: yaml apiVersion: v1 kind: ServiceAccount metadata: name: cloud-controller-manager namespace: kube-system --- apiVersion: apps/v1 kind: DaemonSet metadata: name: openstack-cloud-controller-manager namespace: kube-system labels: k8s-app: openstack-cloud-controller-manager spec: selector: matchLabels: k8s-app: openstack-cloud-controller-manager updateStrategy: type: RollingUpdate template: metadata: labels: k8s-app: openstack-cloud-controller-manager spec: nodeSelector: node-role.kubernetes.io/controlplane: "true" securityContext: runAsUser: 1001 tolerations: - key: node.cloudprovider.kubernetes.io/uninitialized value: "true" effect: NoSchedule - key: node-role.kubernetes.io/controlplane effect: NoSchedule value: "true" - key: node-role.kubernetes.io/etcd effect: NoExecute value: "true" serviceAccountName: cloud-controller-manager containers: - name: openstack-cloud-controller-manager image: docker.io/k8scloudprovider/openstack-cloud-controller-manager:latest args: - /bin/openstack-cloud-controller-manager - --v=1 - --cloud-config=$(CLOUD_CONFIG) - --cloud-provider=openstack - --use-service-account-credentials=true - --bind-address=127.0.0.1 - --cluster-cidr=10.254.1.0/24 volumeMounts: - mountPath: /etc/kubernetes/pki name: k8s-certs readOnly: true - mountPath: /etc/ssl/certs name: ca-certs readOnly: true - mountPath: /etc/config name: cloud-config-volume readOnly: true resources: requests: cpu: 200m env: - name: CLOUD_CONFIG value: /etc/config/cloud.conf hostNetwork: true volumes: - hostPath: path: /etc/kubernetes/pki type: DirectoryOrCreate name: k8s-certs - hostPath: path: /etc/ssl/certs type: DirectoryOrCreate name: ca-certs - name: cloud-config-volume secret: secretName: cloud-config Notice that the third file sets the flag `--cluster-cidr=10.254.1.0/24`. If used in a different ip range, this must be updated. Apply the three files using kubectl .. code:: console kubectl apply -f cloud-controller-manager-roles.yaml kubectl apply -f cloud-controller-manager-role-bindings.yaml kubectl apply -f openstack-cloud-controller-manager-ds.yaml This should after a short time update the nodes as shown in Rancher to become initialized and be ready for deployment. This further configures how to deploy load balancer using the GWDG OpenStack deployment and its rather outdated load balancer system. Cinder CSI Driver ================= To enable Kubernetes to satisfy PVC using Cinder volumes, the cinder csi plugin must be installed. Create another yaml file for this. cinder-csi-plugin.yaml .. code:: yaml # This YAML file contains RBAC API objects, # which are necessary to run csi controller plugin apiVersion: v1 kind: ServiceAccount metadata: name: csi-cinder-controller-sa namespace: kube-system --- # external attacher kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: csi-attacher-role rules: - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "patch"] - apiGroups: ["storage.k8s.io"] resources: ["csinodes"] verbs: ["get", "list", "watch"] - apiGroups: ["storage.k8s.io"] resources: ["volumeattachments"] verbs: ["get", "list", "watch", "patch"] - apiGroups: ["storage.k8s.io"] resources: ["volumeattachments/status"] verbs: ["patch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: csi-attacher-binding subjects: - kind: ServiceAccount name: csi-cinder-controller-sa namespace: kube-system roleRef: kind: ClusterRole name: csi-attacher-role apiGroup: rbac.authorization.k8s.io --- # external Provisioner kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: csi-provisioner-role rules: - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["nodes"] verbs: ["get", "list", "watch"] - apiGroups: ["storage.k8s.io"] resources: ["csinodes"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["list", "watch", "create", "update", "patch"] - apiGroups: ["snapshot.storage.k8s.io"] resources: ["volumesnapshots"] verbs: ["get", "list"] - apiGroups: ["snapshot.storage.k8s.io"] resources: ["volumesnapshotcontents"] verbs: ["get", "list"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: csi-provisioner-binding subjects: - kind: ServiceAccount name: csi-cinder-controller-sa namespace: kube-system roleRef: kind: ClusterRole name: csi-provisioner-role apiGroup: rbac.authorization.k8s.io --- # external snapshotter kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: csi-snapshotter-role rules: - apiGroups: [""] resources: ["events"] verbs: ["list", "watch", "create", "update", "patch"] # Secret permission is optional. # Enable it if your driver needs secret. # For example, `csi.storage.k8s.io/snapshotter-secret-name` is set in VolumeSnapshotClass. # See https://kubernetes-csi.github.io/docs/secrets-and-credentials.html for more details. # - apiGroups: [""] # resources: ["secrets"] # verbs: ["get", "list"] - apiGroups: ["snapshot.storage.k8s.io"] resources: ["volumesnapshotclasses"] verbs: ["get", "list", "watch"] - apiGroups: ["snapshot.storage.k8s.io"] resources: ["volumesnapshotcontents"] verbs: ["create", "get", "list", "watch", "update", "delete"] - apiGroups: ["snapshot.storage.k8s.io"] resources: ["volumesnapshotcontents/status"] verbs: ["update"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: csi-snapshotter-binding subjects: - kind: ServiceAccount name: csi-cinder-controller-sa namespace: kube-system roleRef: kind: ClusterRole name: csi-snapshotter-role apiGroup: rbac.authorization.k8s.io --- # External Resizer kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: csi-resizer-role rules: # The following rule should be uncommented for plugins that require secrets # for provisioning. # - apiGroups: [""] # resources: ["secrets"] # verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "patch"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["persistentvolumeclaims/status"] verbs: ["update", "patch"] - apiGroups: [""] resources: ["events"] verbs: ["list", "watch", "create", "update", "patch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: csi-resizer-binding subjects: - kind: ServiceAccount name: csi-cinder-controller-sa namespace: kube-system roleRef: kind: ClusterRole name: csi-resizer-role apiGroup: rbac.authorization.k8s.io --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: namespace: kube-system name: external-resizer-cfg rules: - apiGroups: ["coordination.k8s.io"] resources: ["leases"] verbs: ["get", "watch", "list", "delete", "update", "create"] --- kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: csi-resizer-role-cfg namespace: kube-system subjects: - kind: ServiceAccount name: csi-cinder-controller-sa namespace: kube-system roleRef: kind: Role name: external-resizer-cfg apiGroup: rbac.authorization.k8s.io --- # This YAML file contains CSI Controller Plugin Sidecars # external-attacher, external-provisioner, external-snapshotter # external-resize, liveness-probe kind: Service apiVersion: v1 metadata: name: csi-cinder-controller-service namespace: kube-system labels: app: csi-cinder-controllerplugin spec: selector: app: csi-cinder-controllerplugin ports: - name: dummy port: 12345 --- kind: StatefulSet apiVersion: apps/v1 metadata: name: csi-cinder-controllerplugin namespace: kube-system spec: serviceName: "csi-cinder-controller-service" replicas: 1 selector: matchLabels: app: csi-cinder-controllerplugin template: metadata: labels: app: csi-cinder-controllerplugin spec: serviceAccount: csi-cinder-controller-sa containers: - name: csi-attacher image: k8s.gcr.io/sig-storage/csi-attacher:v3.1.0 args: - "--csi-address=$(ADDRESS)" - "--timeout=3m" env: - name: ADDRESS value: /var/lib/csi/sockets/pluginproxy/csi.sock imagePullPolicy: "IfNotPresent" volumeMounts: - name: socket-dir mountPath: /var/lib/csi/sockets/pluginproxy/ - name: csi-provisioner image: k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1 args: - "--csi-address=$(ADDRESS)" - "--timeout=3m" - "--default-fstype=ext4" - "--feature-gates=Topology=true" - "--extra-create-metadata" env: - name: ADDRESS value: /var/lib/csi/sockets/pluginproxy/csi.sock imagePullPolicy: "IfNotPresent" volumeMounts: - name: socket-dir mountPath: /var/lib/csi/sockets/pluginproxy/ - name: csi-snapshotter image: k8s.gcr.io/sig-storage/csi-snapshotter:v2.1.3 args: - "--csi-address=$(ADDRESS)" - "--timeout=3m" env: - name: ADDRESS value: /var/lib/csi/sockets/pluginproxy/csi.sock imagePullPolicy: Always volumeMounts: - mountPath: /var/lib/csi/sockets/pluginproxy/ name: socket-dir - name: csi-resizer image: k8s.gcr.io/sig-storage/csi-resizer:v1.1.0 args: - "--csi-address=$(ADDRESS)" - "--timeout=3m" - "--handle-volume-inuse-error=false" env: - name: ADDRESS value: /var/lib/csi/sockets/pluginproxy/csi.sock imagePullPolicy: "IfNotPresent" volumeMounts: - name: socket-dir mountPath: /var/lib/csi/sockets/pluginproxy/ - name: liveness-probe image: k8s.gcr.io/sig-storage/livenessprobe:v2.1.0 args: - "--csi-address=$(ADDRESS)" env: - name: ADDRESS value: /var/lib/csi/sockets/pluginproxy/csi.sock volumeMounts: - mountPath: /var/lib/csi/sockets/pluginproxy/ name: socket-dir - name: cinder-csi-plugin image: docker.io/k8scloudprovider/cinder-csi-plugin:latest args: - /bin/cinder-csi-plugin - "--nodeid=$(NODE_ID)" - "--endpoint=$(CSI_ENDPOINT)" - "--cloud-config=$(CLOUD_CONFIG)" - "--cluster=$(CLUSTER_NAME)" env: - name: NODE_ID valueFrom: fieldRef: fieldPath: spec.nodeName - name: CSI_ENDPOINT value: unix://csi/csi.sock - name: CLOUD_CONFIG value: /etc/config/cloud.conf - name: CLUSTER_NAME value: kubernetes imagePullPolicy: "IfNotPresent" ports: - containerPort: 9808 name: healthz protocol: TCP # The probe livenessProbe: failureThreshold: 5 httpGet: path: /healthz port: healthz initialDelaySeconds: 10 timeoutSeconds: 10 periodSeconds: 60 volumeMounts: - name: socket-dir mountPath: /csi - name: secret-cinderplugin mountPath: /etc/config readOnly: true volumes: - name: socket-dir emptyDir: - name: secret-cinderplugin secret: secretName: cloud-config --- # This YAML defines all API objects to create RBAC roles for csi node plugin. apiVersion: v1 kind: ServiceAccount metadata: name: csi-cinder-node-sa namespace: kube-system --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: csi-nodeplugin-role rules: - apiGroups: [""] resources: ["events"] verbs: ["get", "list", "watch", "create", "update", "patch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: csi-nodeplugin-binding subjects: - kind: ServiceAccount name: csi-cinder-node-sa namespace: kube-system roleRef: kind: ClusterRole name: csi-nodeplugin-role apiGroup: rbac.authorization.k8s.io --- # This YAML file contains driver-registrar & csi driver nodeplugin API objects, # which are necessary to run csi nodeplugin for cinder. kind: DaemonSet apiVersion: apps/v1 metadata: name: csi-cinder-nodeplugin namespace: kube-system spec: selector: matchLabels: app: csi-cinder-nodeplugin template: metadata: labels: app: csi-cinder-nodeplugin spec: tolerations: - operator: Exists serviceAccount: csi-cinder-node-sa hostNetwork: true containers: - name: node-driver-registrar image: k8s.gcr.io/sig-storage/csi-node-driver-registrar:v1.3.0 args: - "--csi-address=$(ADDRESS)" - "--kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)" lifecycle: preStop: exec: command: ["/bin/sh", "-c", "rm -rf /registration/cinder.csi.openstack.org /registration/cinder.csi.openstack.org-reg.sock"] env: - name: ADDRESS value: /csi/csi.sock - name: DRIVER_REG_SOCK_PATH value: /var/lib/kubelet/plugins/cinder.csi.openstack.org/csi.sock - name: KUBE_NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName imagePullPolicy: "IfNotPresent" volumeMounts: - name: socket-dir mountPath: /csi - name: registration-dir mountPath: /registration - name: liveness-probe image: k8s.gcr.io/sig-storage/livenessprobe:v2.1.0 args: - --csi-address=/csi/csi.sock volumeMounts: - name: socket-dir mountPath: /csi - name: cinder-csi-plugin securityContext: privileged: true capabilities: add: ["SYS_ADMIN"] allowPrivilegeEscalation: true image: docker.io/k8scloudprovider/cinder-csi-plugin:latest args: - /bin/cinder-csi-plugin - "--nodeid=$(NODE_ID)" - "--endpoint=$(CSI_ENDPOINT)" - "--cloud-config=$(CLOUD_CONFIG)" env: - name: NODE_ID valueFrom: fieldRef: fieldPath: spec.nodeName - name: CSI_ENDPOINT value: unix://csi/csi.sock - name: CLOUD_CONFIG value: /etc/config/cloud.conf imagePullPolicy: "IfNotPresent" ports: - containerPort: 9808 name: healthz protocol: TCP # The probe livenessProbe: failureThreshold: 5 httpGet: path: /healthz port: healthz initialDelaySeconds: 10 timeoutSeconds: 3 periodSeconds: 10 volumeMounts: - name: socket-dir mountPath: /csi - name: kubelet-dir mountPath: /var/lib/kubelet mountPropagation: "Bidirectional" - name: pods-probe-dir mountPath: /dev mountPropagation: "HostToContainer" - name: secret-cinderplugin mountPath: /etc/config readOnly: true volumes: - name: socket-dir hostPath: path: /var/lib/kubelet/plugins/cinder.csi.openstack.org type: DirectoryOrCreate - name: registration-dir hostPath: path: /var/lib/kubelet/plugins_registry/ type: Directory - name: kubelet-dir hostPath: path: /var/lib/kubelet type: Directory - name: pods-probe-dir hostPath: path: /dev type: Directory - name: secret-cinderplugin secret: secretName: cloud-config --- apiVersion: storage.k8s.io/v1 kind: CSIDriver metadata: name: cinder.csi.openstack.org spec: attachRequired: true podInfoOnMount: true volumeLifecycleModes: - Persistent - Ephemeral --- # This YAML file contains StorageClass definition # and makes it default storageclass apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: csi-sc-cinderplugin annotations: storageclass.kubernetes.io/is-default-class: "true" provisioner: cinder.csi.openstack.org And apply it using .. code:: console kubectl apply -f cinder-csi-plugin.yaml This creates a storage class called `csi-sc-cinderplugin` that should be used for creating volumes. Updating cloud.conf =================== Should it be necessary to update cloud.conf it can be done like this. Create or update a file called cloud.conf as described above. Replace the existing secret using this: .. code:: console kubectl create secret generic cloud-config --from-file=cloud.conf --dry-run -n kube-system -o yaml | kubectl apply -n kube-system -f - Next find the pod running the openstack control manager using .. code:: console kubectl get pod -n kube-system and delete it .. code:: console kubectl delete pod openstack-cloud-controller-manager-xxxx xxxx will be some random code that needs to be identified via the first command. This will cause the pod to be recreated and to load the new cloud.conf file. Should any problem occur, the logs of the openstack-cloud-controller-manager pod can also help troubleshoot the issue. .. code:: console kubectl logs openstack-cloud-controller-manager-xxxx Validate that it works ###################### The next steps are optional and are just to confirm that the cluster can: - Deploy workloads - Deploy Load balancer that connect to OpenStack - Expose a service - Claim a volume Deploy a workload ***************** Rancher2.5 ---------- Rancher UI -> Global context -> Select your new cluster -> Projects/Namespaces -> Under Project: Default press Add Namespace -> name it 'test' -> Create -> Click Project: Default -> Workloads overview should be open -> Deploy -> name it "test-hello" -> as docker image set "rancher/hello-world" -> Launch Rancher2.6 ---------- Rancher UI -> Home -> Select your new cluster -> Projects/Namespaces -> Under Project: Default press Create Namespace -> Name it "test" -> Create -> Workload -> Create -> Deployment -> Namespace to "test" -> Name the workload "test-hello" -> as docker image set "rancher/hello-world" -> Create Observe that it should be scheduled after a few seconds. Deploy a load balancer ********************** Rancher2.5 ---------- Select Apps -> Launch -> search for Nginx and select NGINX Ingress Controller -> and click it -> namespace to use existing -> test -> Launch Click on the nginx app and observe that it is deployed and a load balancer is created in the cluster. Rancher2.6 ---------- Select Apps & Marketplace -> search for Nginx and select NGINX Ingress Controller -> click it -> Install -> Set namespace to test -> Next -> change ingress class to "nginx-test" -> Install After about a minute the load balancer should also appear in OpenStack. Find the load balancer under OpenStack web UI -> Network -> Load Balancers Via the CLI -> The OpenStack CLI only supports loadbalancers via octavia, the API can still be accessed via manual requests. See [[#Access loadbalancers outside of the web UI]] If you currently have no free floating IPs, the load balancer setup will not complete. If the load balancer has not claimed a floating IP, release one: OpenStack web UI -> Network -> Floating IPs -> actions -> release floating IP Be careful not to release the floating IP already claimed by the load balancer. Via the CLI -> Use `openstack floating ip list` and `openstack floating ip delete ID` to release one floating IP Once the load balancer in OpenStack is ready and has claimed an IP, the nginx app in Rancher should also be ready. Make a note of the floating ip that was claimed. Expose a service **************** Rancher2.5 ---------- From the nginx App overview -> Resources -> Workloads -> Load Balancing -> Observe that a L4 Balancer already exists -> Add Ingress -> name it "test-ingress" -> Set namespace to "test" -> Set path to "/" -> choose "test-hello" as target -> Set port as 80 -> Open Labels & Annotations drop down -> Add annotation with key "kubernetes.io/ingress.class" and value "nginx" -> Save A hostname ending in sslip.io will be generated. Rancher2.6 ---------- First create a service Service Discovery -> Services -> Create -> ClusterIP -> name it "test-service" -> listening port and target port to 80 -> Selectors -> Set key as "workload.user.cattle.io/workloadselector" and value to "apps.deployment-test-test-hello" -> Create -> Observe that "test-hello" appears as a pod for that service Then create an ingress Service Discovery -> Ingresses -> Create -> name it "test-ingress" -> set request host to "http://test-ingress.test.FLOATING-IP.sslip.io", where FLOATING-IP is the floating IP noted earlier -> set Path prefix to / -> Select "test-service" as target service and set port to 80 -> Labels&Annotations -> Add Annotation -> Set key to "kubernetes.io/ingress.class" and value "nginx-test" -> Create The hostname should point to a web page with the Rancher logo writing "Hello World!". It might take a moment for the link to work. Click the link and confirm that the workload is exposed. Nginx webhook validation workaround =================================== When trying to create the above described ingress, Rancher might refuse with the error: `Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": an error on the server ("") has prevented the request from succeeding` A simple workaround for this is to run .. code:: console kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission This removes the offending validation hook. Claiming a volume ***************** Rancher2.5 ---------- Select Apps -> Launch -> search for mysql and select it -> namespace to use an existing namespace and pick "test" -> activate PVC and ensure it uses the default storage class that is "csi-sc-cinderplugin" -> Launch Click on the mysql app and observe that it is deployed and a volume claim is created in the cluster. Rancher2.6 ---------- Select Apps&Marketplace -> search for sql and select cockroachdb -> Install -> namespace to "test" -> Next -> Storage per Node to 10Gi -> Install After a moment the app should deploy and open volume claim(s) that becomes satisfied by volume(s) from OpenStack after a few moments. In the OpenStack web UI -> Volumes -> Volumes And one volume should have the description "Created by OpenStack Cinder CSI driver" Or via the CLI -> `openstack volume list` Cleanup ******** Delete the namespace test to cleanup all created resources. Rancher2.5 ---------- Rancher UI -> Global context -> Select cluster -> Projects/Namespaces -> check the box next to the test namepsace -> Delete -> Confirm Rancher2.6 ---------- Rancher UI -> Cluster -> Projects/Namespaces -> check the box next to the test namepsace -> Hold CTRL and press Delete This also removes the load balancer and volumes in OpenStack. CockroachDB might not remove all its volumes, check manually and remove the remaining volumes. Appendix ######## Managing multiple kubeconfigs ***************************** Use a tool such as kubectx to manage what context to load. Option 1: One large config Append any new config to the config file. Option 2: Multiple configs Add all files to $KUBECONFIG. You can do so in your shell config by adding .. code:: console export KUBECONFIG=$KUBECONFIG:$HOME/.kube/config2 Access loadbalancers outside of the web UI ****************************************** This uses the keystoneauth1 package for authentication and making API requests. .. code:: console pip install keystoneauth1 .. code:: python3 import keystoneauth1 # Fill in your credentials my_username = "" my_password = "" my_project_id = "" # Setting up a session password_method = keystoneauth1.identity.v3.PasswordMethod(username=my_username, password=my_password, user_domain_name="GWDG") auth = keystoneauth1.identity.v3.Auth(auth_url="https://api.prod.cloud.gwdg.de:5000/v3", auth_methods=[password_method], project_id=my_project_id, project_domain_name="GWDG") sess = keystoneauth1.session.Session(auth=auth) # API docs: https://wiki.openstack.org/wiki/Neutron/LBaaS/API_2.0 lbaasv2_api = "https://api.prod.cloud.gwdg.de:9696/v2.0" # List all Load Balancers r = sess.request(f"{lbaasv2_api}/lbaas/loadbalancers", "GET", headers={"Accept": "application/json"}) print(r.text)