Kubernetes Cluster Setup

Basic Requirement

You have to configuration your OpenStack CLI has described on dedicated section. The example are based on this configuration, if you have some specific configuration you have to adapt.

You will also have to install magnum client. Magnum is the OpenStack compoment that manage kubernetes provisioning.

$ source ~/.venv/openstack-cli/bin/activate
(openstack-cli) $ pip install python-magnumclient

Danger

Provisioning k8s cluster on cloud@vd require some other privileges not available by normal users. You have to contact us for more details.

Danger

magnum use a specific API to store trust, unfortunately, trust cannot be managed by application credential by default. To deploy a k8s cluster through magnum, you have to add --unrestricted option while creating application credential.

Cluster Configuration

Templates

Magnum use templates to define the kind of k8s infrastructure you want to start. By default, VirtualData provide a set of public template. You can, of course, tweak those templates, but it will not be documented on page. Our template follow our own naming convention k8s-<version>-<node-os-version>[-lb] where :

  • version is the k8s version
  • node-os-version is the base os of k8s infrastructure
  • -lb is present if a load-balancer is configured in front of k8s

To list availables template, you have to launch the following command

$ openstack coe cluster template list
+--------------------------------------+----------------------+------+
| uuid                                 | name                 | tags |
+--------------------------------------+----------------------+------+
[...]
| 4cad1f92-dc10-434a-966a-7166fb6db3e4 | k8s-1.30-coreos40-lb | None |
+--------------------------------------+----------------------+------+

Auto-scaling

Magnum support k8s auto-scaling. This mean that magnum will monitor k8s usage and start (or stop) a node if more resources are required. When started a cluster, you have to specify the minimum and the maximum number of node of this cluster.

Note

Each node take resource from your project quota when a node is started. You need to take care of having enough resources available or the scaling will fail.

Multi-master

Kubernetes support multiple master configuration. This is mandatory to have 2 masters to allow rolling upgrade of kubernetes infrastructure.

Our first kubernetes infrastructure

Starting our cluster

openstack coe cluster create my-k8s-${USERNAME} \
  --cluster-template k8s-1.30-coreos40-lb       \
  --key              vd-key                     \
  --master-count     2                          \
  --node-count       2                          \
  --time             30                         \
  --labels           min_node_count=2           \
  --labels           max_node_count=5           \
  --merge-labels

Get your k8s status

OpenStack will provide you two monitor value:

  • status which is the status of kubernetes infrastructure in OpenStack point-of-view. If it's CREATE_COMPLETE this mean that all virtual machine, all cinder volumes and load balancer are up and running.
  • health_status which is the status in k8s point-of-view. If it's HEALTHY then all pods are running and a user can use the cluster.
$ openstack coe cluster show my-k8s-${USERNAME} -c status
+---------------+-----------------+
| Field         | Value           |
+---------------+-----------------+
| status        | CREATE_COMPLETE |
| health_status | HEALTHY         |
+---------------+-----------------+

Retrieve k8s credentials

The credential to access k8s are generated by OpenStack during cluster creation and are independent to your cloud@vd credential. After your cluster is running, you have to get those credentials to access to your cluster.

$ mkdir -p ~/.k8s/my-k8s-${USERNAME}
$ openstack coe cluster config my-k8s-${USERNAME} --dir ~/.k8s/my-k8s/
$ export KUBECONFIG=~/.k8s/my-k8s-${USERNAME}/config
$

Get k8s cluster informations

To check your k8s cluster status, you just have to launch

$ kubectl cluster-info
Kubernetes control plane is running at https://<ip-add>:6443
CoreDNS is running at https://<ip-add>:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

OpenStack ressources for k8s

Of course, all k8s ressources are runs on OpenStack. You can list all OpenStack ressources like the loadbalancer and servers.

Note

To use loadbalancer openstack command you have to install octavia client with the command pip install python-octaviaclient

$ openstack loadbalancer list
+---------+-------------+------------+----------+---------+---------+---------+
| id      | name        | project_id | vip[...] | pr[...] | op[...] | pr[...] |
+---------+-------------+------------+----------+---------+---------+---------+
| 68[...] | my-k8s[...] | 5e426[...] | 10.[...] | ACTIVE  | ONLINE | amphora  |
| 80[...] | my-k8s[...] | 5e426[...] | 10.[...] | ACTIVE  | ONLINE | amphora  |
+---------+-------------+------------+----------+---------+---------+---------+
$  openstack server list
+-------+-----------------------+-+-----------------------------+---------+-+
| ID    | Name                  | | Networks                    | Image   | |
+-------+-----------------------+-+-----------------------------+---------+-+
| [...] | my-k8s[...]-node-0    | | my-k8s=10.[...], 157.[...]  | fe[...] | |
| [...] | my-k8s-[...]-node-1   | | my-k8s=10.[...], 157.[...]  | fe[...] | |
| [...] | my-k8s-[...]-master-1 | | my-k8s=10.[...], 157.[...]  | fe[...] | |
| [...] | my-k8s-[...]-master-0 | | my-k8s=10.[...], 157.[...]  | fe[...] | |
+-------+-----------------------+-+-----------------------------+---------+-+

Deploy application

For the documentation, we will consider to start a basic web service with a nginx frontend and a mysql database. A set of templates are available in gitlab repository you can clone it and take it as example.

The templates are been created to work out-of-the-box with cloud@vd but can be used on every OpenStack infrastructure. Some part of .yml are OpenStack specific as k8s need to provision volume and ingress control through OpenStack services.

Nginx

nginx installation

In this section, we will install Nginx in your Kubernetes cluster using a Deployment. The Deployment configuration defines the number of replicas, container image, ports, and resource limits for the nginx containers.

nginx deployment

Below is the content of the nginx/deployment.yml file which we will use to create the nginx deployment:

$ kubectl apply -f nginx/deployment.yml
$ kubectl get deployment
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   2/2     2            2           5m17s
$ kubectl get pods
NAME                    READY   STATUS    RESTARTS   AGE
nginx-785d6689b-2twqw   1/1     Running   0          15m
nginx-785d6689b-j4j88   1/1     Running   0          15m

Deployment will start two nginx pods based on nginx:latest docker image. But the service will not be available outside k8s cluster. The next step of our tutorial will be to expose the http port (80).

Expose nginx service

After creating the deployment, apply the service configuration file to expose nginx to the network.

$ kubectl apply -f nginx/service.yml
$ kubernetes % kubectl get service
NAME         TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
[...]
nginx        LoadBalancer   10.254.108.111   <pending>     80:30966/TCP   5s

If you get service status just after the apply, you will have a <pending> external IP. You have to wait some minutes to have the external IP available

$ kubectl get service  
NAME         TYPE           CLUSTER-IP       EXTERNAL-IP       PORT(S)        AGE
[...]
nginx        LoadBalancer   10.254.108.111   157.136.xxx.yyy   80:30966/TCP   2m33s

By following these steps, you will have a fully functional nginx deployment in your Kubernetes cluster. The deployment ensures that two replicas of the nginx container are running, and the service exposes nginx to handle incoming traffic.

You can test it with curl command

$ curl http://157.136.xxx.yyy > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   615  100   615    0     0  31185      0 --:--:-- --:--:-- --:--:-- 32368

Custom html pages

We will create a ConfigMap to hold the HTML content and mount it to the nginx pod. This will overwrite the default index.html file provide by nginx:latest docker image.

$ kubectl apply -f nginx/configmap.yml
$ kubectl get ConfigMap  
NAME               DATA   AGE
kube-root-ca.crt   1      11d
nginx-index        1      50s

To link the ConfigMap containing the HTML content to the nginx pods, you need to modify the deployment. To do that, you can use nginx/deployment-with-config-map.yml which is based on nginx/deployment.yml.

$ diff -u nginx/deployment.yml nginx/deployment-with-config-map.yml
--- nginx/deployment.yml        2024-09-28 19:21:27
+++ nginx/deployment-with-config-map.yml        2024-09-28 19:24:54
@@ -22,3 +22,10 @@
           requests:
             memory: "64Mi"
             cpu: "250m"
+        volumeMounts:
+        - name: nginx-index
+          mountPath: /usr/share/nginx/html
+      volumes:
+      - name: nginx-index
+        configMap:
+          name: nginx-index
$ kubectl apply -f nginx/deployment-with-config-map.yml
deployment.apps/nginx configured

Now, nginx service will provide your custom index.html file.

$ curl http://157.136.xxx.yyy
Hi VirtualData !
$

OpenStack ressources for a deployment

Of course, compute ressource are provide by my-k8s-[...]-node-x virtual machine, but the loadbalancer service for nginx has been started by k8s has a OpenStack octavia service

$ openstack loadbalancer list
+---------+-------------------------------------+-+-------------+-+-+----------+
| id      | name                                | | vip_address | | | provider |
+---------+-------------------------------------+-+-------------+-+-+----------+
| 79[...] | kube_service_[...]_default_nginx    | | 10.[...]    | | | amphora  |
+---------+-------------------------------------+-+-------------+-+-+----------+

Mysql

Danger

This part and beyond are not tested

Installation

In this section, we will install MySQL in your Kubernetes cluster using a Deployment. The Deployment configuration defines the container image, ports, and resource limits for the MySQL containers. Additionally, it will include persistent storage to ensure data durability.

To create the MySQL Deployment, apply the following YAML configuration file:

$ kubectl apply -f mysql/deployment.yml
$

Finally, apply the Service configuration to expose MySQL:

kubectl apply -f mysql/service.yml

MYSQL Configuration

After applying the deployment, you can verify that the MySQL pods are running by executing:

kubectl get deployments
kubectl get pods

To check if the MySQL service is running correctly and exposed, you can use the following command:

kubectl get services

Ingress Controller Installation

Problematic

Currently, in our example, our NGINX server uses a single IP. If we multiply the number of web sites, we will end up using multiple IPs, resulting in fewer available IPs. To address this, we will set up a reverse proxy using the Octavia Ingress Controller to direct users to the correct application based on the URL used.

Installation Octavia Ingress Controller

exposur with reverse-proxy

We will install the Octavia Ingress Controller in the cluster and configure it to handle HTTP(S) traffic, routing requests to the appropriate services based on the URL.

Compatibility matrix

For this Kubernetes cluster, we need specific requirements to use the Octavia Ingress Controller:

  • Communication between Octavia Ingress Controller and Octavia is needed.
  • Octavia stable/queens or higher version is required because of features such as bulk pool members operation.
  • OpenStack Key Manager (Barbican) service is required for TLS Ingress, otherwise Ingress creation will fail.
Install Octavia Ingress Controller

Follow the instructions provided in the Octavia Ingress Controller documentation to install the controller.

Apply the service account configuration defined in serviceaccount.yaml for the Octavia Ingress Controller.

kubectl apply -f octavia-ingress-controller/serviceaccount.yaml

Note

Apply the configuration settings specified in config.yaml for the Octavia Ingress Controller.

$ kubectl apply -f octavia-ingress-controller/config.yaml
$

Deploy the Octavia Ingress Controller using the settings defined in deployment.yaml.

$ kubectl apply -f octavia-ingress-controller/deployment.yaml
$

Apply the Ingress resource configuration defined in ingress.yaml to configure traffic routing for the Octavia Ingress Controller. In this file you have to define the url for the service access

$ kubectl apply -f octavia-ingress-controller/ingress.yaml
$
loadbalancer

Load Balancer Configuration

When deploying the reverse proxy, it will dynamically fetch the load balancer's IP address from OpenStack. During the initial deployment of the reverse proxy pods, it is recommended to avoid specifying a fixed IP address for the load balancer in the configuration. Ensure that the IP address assigned by OpenStack is accessible and meets your requirements.

To find the IP address, use the following command:

kubectl get ingress

Add this IP address to the "octavia-ingress-controller/ingress.yaml" file.

Alternatively, you can directly access your nginx web server without using the octavia-ingress-controller. When launching the nginx service, a load balancer is automatically configured for nginx.

To retrieve the IP address of the nginx service, use:

$ kubectl get service
$

You can then make an HTTP request using the IP address of the nginx service.

In our tutorial we will use the ingress controller to limit the use of IPs We are going to delete the LB on the nginx-service

$ kubectl patch service nginx-service -p '{"spec": {"type": "NodePort"}}'
$

DNS

Now an IP is allocated to the pods of ingress-controller, to have a link beetween the IP and the ingress routes created we will have to associate CNAMEs with the IP that Openstack created for the ingress-controller

$ kubectl get ingress
$

TLS

Github documentation

kubectl create secret tls tls-secret \
  --cert nginx-yh-server.ijclab.in2p3.fr.crt \
  --key nginx-yh-server.ijclab.in2p3.fr.key

kubectl apply -f octavia-ingress-controller/default-backend.yaml
kubectl apply -f octavia-ingress-controller/ingress-update.yaml

Now we can access to http://nginx-yh-server.ijclab.in2p3.fr and http://nginx-yh-server.ijclab.in2p3.fr

Install nginx-ingress & Let's Encrypt

This tutorial on the Cert-Manager website guides you through the process of installing and configuring Cert-Manager with NGINX Ingress to automate the acquisition and renewal of SSL/TLS certificates using the ACME protocol. Ideal for securing communications in Kubernetes applications, this guide is a valuable resource for developers and systems administrators looking to implement HTTPS certificates efficiently and automatically.

Installation

Volume persistent

Introduction

This section introduces the necessary configuration for automatic storage management in Kubernetes through a specific StorageClass This creates the storage class needed to automatically provision volumes.

kubectl apply -f volume/storageclass-cinder.yaml

Persistant Volume

A Persistent Volume (PV) in Kubernetes is a storage unit in the cluster that has been provisioned manually by an administrator or dynamically by Kubernetes via the StorageClass. The PV is crucial for applications that require durable data retention, independent of the lifespan of individual pods.

Persistent Volume Claim

The Persistent Volume Claim (PVC) is a storage request issued by a user, specifying the required size and access mode. The PVC binds to the most suitable available PV to ensure a secure and isolated environment for the data.

Cas de test

Cas 1 Removal of the Deployment

Objective: To confirm that the deletion of the Deployment does not lead to loss of data stored in the database.

Creation of namespace

kubectl create namespace test-pv-pvc

Creation of PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pvc
  namespace: test-pv-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
    storage: 5Gi

Creation of deployement

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
  namespace: test-pv-pvc
spec:
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
  spec:
      containers:
      - name: mysql
      image: mysql
        resources:
          requests:
            memory: "1Gi"
            cpu: 2
          limits:
            memory: "2Gi"
            cpu: 3
        ports:
        - containerPort: 3306
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: "password" # Remplacez par votre mot de passe sécurisé
        volumeMounts:
        - name: mysql-storage
          mountPath: /var/lib/mysql
        volumes:
        - name: mysql-storage
          persistentVolumeClaim:
          claimName: mysql-pvc

Connection to the database

kubectl exec -it [mysql-pod-name] --namespace=test-pv-pvc -- mysql -u root -p

Addition of elements

CREATE DATABASE testdb;
USE testdb;
CREATE TABLE example (
    id INT AUTO_INCREMENT PRIMARY KEY,
    data VARCHAR(100)
);
INSERT INTO example (data) VALUES ('Test data');

We then remove the pods

kubectl delete pod [mysql-pod-name] --namespace=test-pv-pvc

With the deployment set up, the pod is automatically restarted We will check that the created elements are always present in the mysql pods Deleting the deployment and then recreating it does not impact the database.

Reconnection to the database

kubectl exec -it [mysql-pod-name] --namespace=test-pv-pvc -- mysql -u root -p

Check that items are present

USE testdb;
SELECT * FROM example;

mysql> USE testdb;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> SELECT * FROM example;
+----+-----------+
| id | data      |
+----+-----------+
|  1 | Test data |
+----+-----------+
1 row in set (0.00 sec)

mysql>
Cas 2 Removal of the Namespace & Cluster

Objectif: Test the effect of deleting the namespace

kubectl delete namespace test-pv-pvc

All the elements present in the namespace are removed Data from the database can no longer be retrieved. To retrieve them after deleting the namespace we will add the persistent volume (PV)

To create the PV we must first create the volume on openstack

openstack volume create --size <size> --description "<Description>" <Name>

Volume observation created with

openstack volume list

Creation of namespace

kubectl create namespace test-pv-pvc

Creation of Persistant Volume

apiVersion: v1
kind: PersistentVolume
metadata:
  name: cinder-pv
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  storageClassName: cinder
  cinder:
    fsType: ext4
    volumeID: "ID" #METTRE ID DU VOLUME
  persistentVolumeReclaimPolicy: Retain

Creation of Persistant Volume Claim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cinder-pvc
  namespace: test-pv-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: cinder
  resources:
    requests:
      storage: 5Gi
  volumeName: cinder-pv

Creation of deployement

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
  namespace: test-pv-pvc
spec:
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql
        resources:
          requests:
            memory: "1Gi"
            cpu: 2
          limits:
            memory: "2Gi"
            cpu: 3
        ports:
        - containerPort: 3306
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: "password"  # Remplacez par votre mot de passe sécurisé
        volumeMounts:
        - name: mysql-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-storage
        persistentVolumeClaim:
          claimName: cinder-pvc

I add the data in the database as in the first example Now I destroy the namespace (or is present the pvc and the deployment) They are removed if I restart it (namespace->PVC->Deployment) The pvc tells me that the "volume "cinder-pv" already bound to a different claim." I therefore delete all resources, namely the namespace (pvc ,deployment ) and PV To restart them in the order: NS and PV -> PVC -> deploy

I connect to the databases and observe the data created previously are in. NOTE: I am getting the same error when removing the PVC and deployment I have to repeat the same procedure

Removal of the cluster

If the cluster and delete data can still be retrieved with the same procedure.

Authentication with Webhook in Kubernetes using Keystone

This documentation outlines the steps to set up and configure a Webhook for authentication in a Kubernetes cluster using Keystone from OpenStack. The test environment assumes a cluster without floating IP or master load balancers (LBAs), with the Kubernetes API server located on the master node.

Prerequisites

  • My tests were done on a Kubernetes cluster without a floating IP or master LB.
  • OpenStack environment with Keystone configured.
  • The project used for the Kubernetes cluster in OpenStack:

bash & openstack project list NUMBER | ijclab-ing-inf-exploit-container-test

  • Roles used in Keystone: admin, users.

Removing Existing Webhook Configurations

Before starting, remove any unused or non-functional Webhook configurations (Daemonstat, Pods, ConfigMaps, Clusterrole {systeme:k8s-keystone-auth}, Clusterrolebindings {systeme:k8s-keystone-auth} , ServiceAccount, in the cluster to avoid potential issues. A webhook is pre-installed in the cluster because in the cluster-template we have labels: 'k8s_keystone_auth_tag': 'v1.XX.0'

Creating the Authorization Policy ConfigMap

This ConfigMap gives the permissions to allow users with the "user" role in keystone on any resource in any namespace. But the authorization part is not functional during my tests Create a ConfigMap for user authorization policies:

kubectl apply -f webhook/1config-map.yaml

more details

Apply this ConfigMap using kubectl.

Generating and Configuring Certificates

Generate a self-signed certificate (not recommended for production):

cd PATH/cert
# Generate a self-signed certificate
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem \
-days 365 -nodes -subj /CN=k8s-keystone-auth.kube-system/
# Create a secret kubernetes using the certificate
kubectl --namespace kube-system create secret tls \
keystone-auth-certs --cert=cert.pem --key=key.pem

Setting Up ServiceAccount and RBAC

Create a ServiceAccount and ClusterRole for the Webhook:

kubectl apply -f webhook/3Service-Account-RBAC.yaml

Deploying the Webhook

Create the Deployment for the Webhook:

kubectl apply -f webhook/4Deployment.yaml

Configuring the Service

Initially, configure the service as ClusterIP:

kubectl apply -f webhook/5.1Service-ClusterIP.yaml

Now you can check if the pods is accessible with the service

Testing the Webhook

Authentification

Retrieve the OpenStack token:

token=$(openstack token issue -f value -c id)

Test the Webhook with curl:

kubectl run curl --rm -it --restart=Never --image curlimages/curl -- \
  -k -XPOST https://k8s-keystone-auth-service.kube-system:8443/webhook -d '
{
  "apiVersion": "authentication.k8s.io/v1beta1",
  "kind": "TokenReview",
  "metadata": {
    "creationTimestamp": null
  },
  "spec": {
    "token": "'$token'"
  }
}'

You should see the response from k8s-keystone-auth service.

For debugging, change the service type to NodePort to access it via the node's IP and port:

kubectl apply -f webhook/5Service-NodePort.yaml

Retrieve the OpenStack token:

token=$(openstack token issue -f value -c id)

Test the Webhook with curl:

  #Get IP of node
kubectl get node -o wide

curl -k -XPOST https://{ip-node-master}:30000/webhook -d '{
  "apiVersion": "authentication.k8s.io/v1beta1",
  "kind": "TokenReview",
  "metadata": {
    "creationTimestamp": null
  },
  "spec": {
    "token": "'$token'"
  }
}'
Authorization

For my tests the permission with the webhook is not working Pay attention to the current permissions set up gives total access if for you the permission works

Exemple :

kubectl run curl --rm -it --restart=Never --image curlimages/curl -- \
  -k -XPOST https://k8s-keystone-auth-service.kube-system:8443/webhook -d '
{
  "apiVersion": "authorization.k8s.io/v1",
  "kind": "SubjectAccessReview",
  "spec": {
    "resourceAttributes": {
      "namespace": "default",
      "verb": "get",
      "groups": "",
      "resource": "pods",
      "name": "pod1"
    },
    "user": "demo",
    "groups": ["ID-OF-THE-PROJECT"],
    "extra": {
        "alpha.kubernetes.io/identity/project/id": ["ID-OF-THE-PROJECT"],
        "alpha.kubernetes.io/identity/project/name": ["demo"],
        "alpha.kubernetes.io/identity/roles": ["load-balancer_member","users"]
    }
  }
}'

To try with the master node IP:

curl -k -XPOST https://{ip-node-master}:30000/webhook -d '
  -k -XPOST https://k8s-keystone-auth-service.kube-system:8443/webhook -d '
{
  "apiVersion": "authorization.k8s.io/v1",
  "kind": "SubjectAccessReview",
  "spec": {
    "resourceAttributes": {
      "namespace": "default",
      "verb": "get",
      "groups": "",
      "resource": "pods",
      "name": "pod1"
    },
    "user": "demo",
    "groups": ["ID-OF-THE-PROJECT"],
    "extra": {
        "alpha.kubernetes.io/identity/project/id": ["ID-OF-THE-PROJECT"],
        "alpha.kubernetes.io/identity/project/name": ["demo"],
        "alpha.kubernetes.io/identity/roles": ["load-balancer_member","users"]
    }
  }
}'

Configuration on K8S master for authentication and/or authorization

We will connect to the master in order to make changes to remove the management of permissions from the webhook.

# ssh on the master (you need to have the ssh-key)
ssh core@ip-node-master
# Root privileges
sudo su
# List files
ls /etc/kubernetes
# Modification du fichier de configuration de l'api server
nano /etc/kubernetes/apiserver

# check the "--authorization-mode=Node,Webhook,RBAC" is in the file
# remove :
# --authorization-webhook-config-file=/etc/kubernetes/keystone_webhook_config.yaml
# and remove the Webhook posibility on the --authorization-mode
# --authorization-mode=Node,RBAC

We will now change the use of the configuration file in the master to define the ip of the node or the ip of the ClusterIP if it works and the port to be used nano /etc/kubernetes/keystone_webhook_config.yaml

kubectl apply -f webhook/6Config.yaml

Danger

Security Warning : The port being open it will just have to allow in local and block attempts external to the network (via the security groups of openstack)

Add Security Groups

In Openstack security groups you find the one that matches the machine Then Manage the rules Then add a rule Adds a Description (optional) Direction: Entrance Open Port Port: Webhook port Remote CIDR CIDR Networks

Client Configuration

Download the client-keystone-auth binary

This link automatically downloads the client

Download Manually client-keystone-auth Find a release : client-keystone-auth At the bottom of the web page Click on assets And now click on the client-keystone-auth

Retrieve the cluster certificate from the admin's config file.

Run kubectl config set-credentials openstackuser, this command creates the following entry in the ~/.kube/config file.

- name: openstackuser
  user: {}

Run kubectl config set-credentials openstackuser, this command creates the following entry in the ~/.kube/config file. Config kubectl to use client-keystone-auth binary for the user openstackuser. Replace my-k8s-${USERNAME} with your own cluster name. Replace [DATA] with the certificate of the cluster. Replace [PATH] with the path to acces to the client-keystone-auth. Replace [IP] with the IP address of the master ip.

Here’s what your ./kube/config should look like

$ cat .kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: [DATA]
    server: https://[IP]:6443
  name: my-k8s-${USERNAME}
contexts:
- context:
    cluster: my-k8s-${USERNAME}
    user: openstack
  name: openstack@my-k8s-${USERNAME}
current-context: default
kind: Config
preferences: {}
users:
- name: openstack
  user:
    exec:
      command: "/[PATH]/client-keystone-auth"
      apiVersion: "client.authentication.k8s.io/v1beta1"

Connection

If the authorization webhook work test that everything works as expected try:

Put the variable in the RC file before trying a connection with the kubectl add this line export OS_DOMAIN_NAME="ijclab"in the file Source your environment file (RC file)

kubectl --context openstackuser@my-k8s-${USERNAME} get pods

In case you are using this Webhook just for the authentication, you should get an authorization error

  ## Output
Error from server (Forbidden): pods is forbidden:
User "username" cannot list pods in the namespace "default"
Configuring RBAC Authorization

If the Webhook fails, fallback to using Kubernetes RBAC for authorization. Create a RoleBinding for user access: Change 'username' to the real user name

kubectl create rolebinding username-view --clusterrole view\
--user username --namespace default

Retry :

kubectl --context openstackuser@my-k8s-${USERNAME} get pods

Notes and Recommendations

  • Use a trusted CA-signed certificate for production environments.
  • Secure the node port with OpenStack security groups to prevent unauthorized access.
  • Simplify Kubernetes user management by syncing Keystone and Kubernetes roles.

Additional Resources

Monitoring the cluster

Introduction Prometheus

In this section, I will install two prometheus, one outside the cluster to have a centralized prometheus that retrieves all metrics of the different services we want to monitor with prometheus

Prometheus

Installation Prometheus centralized

Download link Configuration link

Also, make sure that the port used by prometheus is open. Port 9090, you can test its connectivity from another machine of the prometheus server with

curl http://IP-Monitor:9090

If the port is not open, add a rule in the Security Groups that is used by the VM hosting the Central Prometheus

Installation Prometheus on kubernetes with helm

Installation

helm install my-prometheus -f values.yaml prometheus-community/prometheus

If we modify after the installation the values.yaml file, we apply these modifications to the cluster with:

helm upgrade my-prometheus -f values.yaml prometheus-community/prometheus

Le fichier values.yaml est récupérable directement depuis le site

La partie modifier du fichier values.yaml est la séction :

remoteWrite:
## https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_read
  - url: http://IP-Monitor:9090/api/v1/write
  # The URL of the external Prometheus server where the metrics will be sent.
    remote_timeout: 5s
    # Sets the timeout duration for the remote write requests.
    # If a request takes longer than 300 seconds, it will time out.
    write_relabel_configs:
    # This section is used to apply transformations to the metrics
    # before sending them to the external Prometheus.
      - source_labels: [__name__]
      # Specifies the metric name as the source label.
        regex: ".*"
        # Matches all metric names.
        action: keep
        # Retains all metrics. No filtering is done at this stage.
      # keep only specific labels for the kept metrics
      - regex: '.*'
        action: 'labelkeep'
      # This rule adds or replaces a label called 'source'
      # with the value 'kubernetes-prometheus'.
      # It helps identify that the metrics are coming from the Kubernetes Prometheus.
      - action: 'replace'
        target_label: 'source'
        replacement: 'kubernetes-prometheus-1.30'
Configuration in the centralized prometheus

To make the central prometheus capable of receiving metrics from the Kubernetes cluster, we need to adapt its configuration.

In the configuration file: /etc/systemd/system/prometheus.service

ExecStart=/usr/share/prometheus/prometheus \
--config.file=/usr/share/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus \
--web.enable-remote-write-receiver \
--web.listen-address="0.0.0.0:9090"

The option --web.enable-remote-write-receiver must be set.

The prometheus service is relaunched to take changes into account

sudo systemctl daemon-reload
sudo systemctl restart prometheus.service
sudo systemctl status prometheus.service
Test
  • Go to the url : IP-Monitor:9090
  • Type in "Expression": "up" then "Execute"
  • Note: you should have multiple instances with the label: source="kubernetes-prometheus-1.30"
Additional Resources Prometheus

Ressource 1 Ressource 2

Grafana

Introduction Grafana

We will install Grafana to have a graphical interface to read prometheus metrics.

Installation Grafana

Installation After installation we access the interface of grafana with port 3000 in http so with this URL: IP:3000/ Ensure that the server is only accessible inside the lab with Security Groups in Openstack

Configuration

All of the above we will add the local prometheus Datasource (because metrics are stored on the centralize prometheus) IP-Monitor:3000/connections/datasources/new

Home > Connections > Data sources > Add data source > Prometheus

  • Add a name in the name section:
  • Add the url to the connection: localhost:9090,
  • Validate the information with a "Save&test"
Dashboard

In order to display these metrics we will retrieve dashboards pre-created by the community

  • For example this dashboard
  • Copy dashboard ID: 10000
  • Go to the url to add dashboard: IP-Monitor:3000/dashboard/import
  • Choose your {name, UID(if desired), Prometheus(datasource to use)}
  • Valid with import
  • It is made editable in settings

For some analysis one has "No data" it means that the promQL query is not good is that you must modify it in order to have the value of the search desired.

Let us analyze this request:

sum (container_memory_working_set_bytes{image!="",
name=~"^k8s*",kubernetes_io_hostname=~"^$Node$"}) by (pod_name)
  • It contains a metrics: ‘container_memory_working_set_bytes’
  • In Prometheus I enter the metrics we can see an example of the mysql pod:
container_memory_working_set_bytes{beta_kubernetes_io_arch="amd64",
beta_kubernetes_io_instance_type="vd.4", beta_kubernetes_io_os="linux",
draino_enabled="true", failure_domain_beta_kubernetes_io_region="lal",
failure_domain_beta_kubernetes_io_zone="nova",
id="/kubepods/burstable/pod3f392a81-6e42-436e-aebc-62b95a665190",
instance="k8s-1-30-coreos40-lb-yh-public-2i3uldrmsozm-node-0",
job="kubernetes-nodes-cadvisor", kubernetes_io_arch="amd64",
kubernetes_io_hostname="k8s-1-30-coreos40-lb-yh-public-2i3uldrmsozm-node-0",
kubernetes_io_os="linux", magnum_openstack_org_nodegroup="default-worker",
magnum_openstack_org_role="worker", namespace="default",
node_kubernetes_io_instance_type="vd.4", pod="mysql-68d8d6696d-xnp4p",
source="kubernetes-prometheus-1.30",
topology_cinder_csi_openstack_org_zone="nova", topology_kubernetes_io_region="lal",
topology_kubernetes_io_zone="nova"}
= 463773696
  • No key named "name"
  • no key named "pod_name" it must match "pod"
  • Do not forget to add a value key source="kubernetes-prometheus-1.30" if one day there are several metrics it may be difficult to find

I change the query to:

sum (container_memory_working_set_bytes{image!="",
source="kubernetes-prometheus-1.30",kubernetes_io_hostname=~"^$Node$"}) by (pod)

The value $Node$ is linked to a variable that can be modified and observed in the section "Variables" in the Settings, in our case it is well configured

AlertManager

Introduction AlertManager

We will install AlertManager to receive alerts.

Installation Alertmanager

Creation of a user

useradd --no-create-home --shell /bin/false alertmanager

Creation of a configuration directory and data

mkdir /etc/alertmanager
mkdir -p /var/lib//alertmanager/data
chown alertmanager:alertmanager /var/lib/alertmanager/data

Download and installation: have the latest version

wget https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz

Place the binaries in the path

cd alertmanager-0.27.0.linux-amd64
cp alertmanager /usr/local/bin/
cp amtool /usr/local/bin/

Change of rights

chown alertmanager:alertmanager /usr/local/bin/alertmanager
chown alertmanager:alertmanager /usr/local/bin/amtool
Configuration alertmanager

In the file that will be created: ‘/etc/systemd/system/alertmanager.service’

[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
WorkingDirectory=/etc/alertmanager/
ExecStart=/usr/local/bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml\
--web.external-url http://IP-Monitor:9093 --cluster.advertise-address="IP-Monitor:9093"

[Install]
WantedBy=multi-user.target

The file /etc/alertmanager/alertmanager.yml I created a Telegram bot to receive notifications creates it in parallel

For general configuration

There are different ways to send notifications you can find them here

global:
  resolve_timeout: 2m
  smtp_require_tls: false

route:
  group_by: ['kubernetes_io_hostname', 'severity']
  # Send all notifications to me.
  group_wait: 5s  # Temps d'atente avant notification
  group_interval: 1m  # Delai par rapport aux alertes du meme groupe
  repeat_interval: 5s  # Attente avant repetition
  receiver: 'telegram'

receivers:
- name: 'telegram'
  telegram_configs:
  - bot_token: ##############################################
    api_url: https://api.telegram.org
    chat_id: ###########
Configuration prometheus

In the prometheus file /usr/share/prometheus/prometheus.yml we add or remove comments, the alerting part with alertmanagers

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets: ["localhost:9093"]

systemctl daemon-reload
systemctl restart prometheus
systemctl restart alertmanager
First Alert

Configuration Alert the file/etc/prometheus/alert_rules.yml

## The backslash (\) at the end of the line indicates
## that the expression should continue on the same line in this example.
groups:
- name: cpu_alerts
  rules:
  - alert: HighCpuLoad
    expr: sum by (kubernetes_io_hostname)\
    (rate(container_cpu_usage_seconds_total{id="/", kubernetes_io_hostname=~".+"}[5m]))\
    / sum by (kubernetes_io_hostname) (machine_cpu_cores{kubernetes_io_hostname=~".+"})\
    * 100 > 85
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High CPU Load on {{ $labels.kubernetes_io_hostname }}"
      description: "Node {{ $labels.kubernetes_io_hostname }}\
      has a CPU load higher than 85% for more than 5 minutes\
      (current value: {{ $value }}%)."

This alert monitors the CPU usage of Kubernetes nodes and triggers if the load exceeds 85% on a node. It checks this condition over a period of 5 minutes to avoid temporary peaks. If the alert is triggered, a "warning" severity notification is sent with details about the affected node and the CPU load value.

Changes must be applied

systemctl restart alertmanager
Other
Create a bot telegram

Bot telegram

Use a webhook to integrate rocketchats on alertmanager

Medium link Github project 1 Github project 2

Additional Resources understanding

Ressource 1 Ressource 2

Save Database on kubernetes

Introduction database save

First one see if the storageclass is created.

kubectl get storageclasses.storage.k8s.io
NAME               PROVISIONER
cinder (default)   cinder.csi.openstack.org

if is not created apply this yaml configuration

kubectl apply -f volume/storageclass-cinder.yaml

Now created Persistant Volume Claim.

kubectl apply -f backup/pvc-mysql.yaml

Installation Database

Creation of database

kubectl apply -f backup/deployment-mysql.yaml

Creation of Service

kubectl apply -f backup/service-mysql.yaml
Input data in Database

Now we have to fill this database with information. First you will have to launch a terminal of the pods with the correct suffix auto-generate by kubernetes

kubectl get pods
kubectl exec -it mysql-SUFFIXE -- /bin/bash

Inside the pod we will launch this command

mysql -u testuser -ptestpassword
SHOW DATABASES;
USE testdb;
CREATE TABLE personnes (
     id INT AUTO_INCREMENT PRIMARY KEY,
     nom VARCHAR(100),
     prenom VARCHAR(100),
     annee_naissance INT
);

INSERT INTO personnes (nom, prenom, annee_naissance)
VALUES ('Doe', 'John', 1990),
        ('Smith', 'Jane', 1985);

#/
SELECT * FROM personnes;

Image Dockerfile for the CronJob

For extract the database from the job i need a ubuntu with ssh and mysql packages installed. You can see my image here backup/dockerfile

Export Database to server

First creates the secret in kubernetes with the ssh priver key. For more security generate a ssh key exclusively for this use (The public key (.pub) must be in the file authorized_keys (/root/. ssh/authorized_keys) of the server that will receive the dump))

kubectl create secret generic ssh-key-secret --from-file=ssh-privatekey=/home/USER/.ssh/KEY.pem

The CronJob

kubectl apply -f backup/cronjob-mysql.yaml

Display of the database dump on the server

After the job is executed in kubernetes we can observe the dump on our server. Docker must be installed on the server to import it in order to validate that the database is well presented I will use a docker image with the same version as used in our test.

sudo docker run --name mysql-container -e MYSQL_ROOT_PASSWORD=testpassword -d mysql:9.0.1
sudo docker cp DATABASE/dump-2024-09-19.sql mysql-container:/dump.sql
sudo docker exec -it mysql-container bash
````

Into the container

```sh
mysql -u root -ptestpassword
SOURCE /dump.sql;
SHOW DATABASES;
USE testdb;
SHOW TABLES;
SELECT * FROM personnes;

Dynamically manage our cluster with k9s

GitHub Project k9s

Installation 1

Installation with snap
snap install k9s --devmode

Installation 2

Download the binary

For example, if you are using a 64-bit Linux system

wget https://github.com/derailed/k9s/releases/download/v0.28.2/k9s_Linux_amd64.tar.gz
echo 'Extract the binary'
tar -xzf k9s_Linux_amd64.tar.gz
echo 'Make it executable'
chmod +x k9s
echo 'Move it to a bin directory'
sudo mv k9s /usr/local/bin/