Kubernetes - Container orchestration

Introduction

Kubernetes (or K8s) is an open-source system used for container orchestration. Indeed, it complements container applications such as Docker. It permits the consistent deployment and management of containers. Kubernetes was developed by Google and later donated to the Cloud Native Computing Foundation. The first version was released in July 2015.

Compatibility

At the beginning, Kubernetes was only compatible with Docker. Over time, Kubernetes became able to work with other container applications such as rkt. To accomplish this, they introduced an interface called the Container Runtime Interface (CRI). It allows any application to work with any container application as long as they adhere to the Open Container Initiative (OCI) standards.

The Open Container Initiative (OCI) consists of an imagespec and a runtimespec :

• imagespec : It defines the specifications on how an image should be built.
• runtimespec : It defines the standards for how any container runtime should be developed.

However, Docker is not compatible with Container Runetime Interface because he has been developed before the Open Container Initative standars. So, Kubernetes had to adapted their application for the CRI and Docker as well. To be able to support, they have set up dockershim.

A few years later, after an update, Kubernetes stopped support dockershim, and Docker was not supported anymore by Kubernetes for the runtime, but only for the imagespec because it followed the standard of OCI. Nowadays, a group of developers has created cri-dockerd, a runtime that is compatible with Kubernetes.

Architecture

The first thing we find in the architecture of container orchestration is the Node (prevously named Minions). A Node can be a physical or virtual machine where containers are run by Kubernetes.

You should have multiple Nodes to use Kubernetes properly. Indeed, if one Node fails, the others are still available to take over and keep the application running. This setup is called a cluster.

The Master Node should be set up to move the workload. The Master Node watches over the Nodes in the Cluster and is responsible for container orchestration.

Components

Kubernetes is composed of multiple components that are automatically installed when you set up it on your system :

• API Server : This acts as the front end for Kubernetes, managing and interacting with Nodes in the cluster ;
• Scheduler : It is responsible for distributing the workload and containers across the multiple Nodes in the cluster. When a new container is created, it automatically assigns it to a Node.
• Controller : It acts as the brain of this orchestration. Indeed, it notices and responds when Nodes or containers go down. It makes decisions about when new containers need to be brought up.
• etcd : It is a key-value store that holds all the information about the Nodes and Masters in the cluster ;
• Container Runtime : This is the software used to run containers (Docker in our case).
• Kubelet : It is the agent running on each node in the cluster. It ensures that the containers are running as expected on the Node.

There are also additional tools available for configuring and managing a Kubernetes cluster :

• Kubectl : Kubectl is the command-line interface used to interact with a Kubernetes cluster. It allows users to create, modify, retrieve information about, or delete resources.
• Kubeadm : Kubeadm is a tool designed to simplify the installation and configuration of a Kubernetes cluster. It allows you to designate which machines will function as the Master and which will serve as Nodes.
• Minikube : similar to Kubeadm but is intended for small-scale configurations. For example, if you want to run both the master and the node on the same machine, Minikube is the appropriate tool to use.

In this setup, the Node having the kube-apiserver is designated as the cluster Master. Other Nodes, with the Kubelet agent, communicate with the master to share data and execute tasks upon request. Furthermore, components like etcd and the Scheduler are also present in the Master setup.

Debugging Containers Tool (circtl)

Kubernetes has its own command line tool named crictl, which is used to interact with container runtimes.

It is not used for deploying or managing but more for debugging container runtimes. Indeed, it works across different runtimes that are CRI compatible. The commands are quite similar to Docker.

• To pull an image, run the following command :

marijan$ crictl pull [IMAGE]

• To list existing images, execute :

marijan$ crictl images

• To get existing containers, run :

marijan$ crictl ps -a

• To execute a command inside a container, run :

marijan$ crictl exec -i -t [CONTAINER ID] [CMD]

• To view the logs :

marijan$ crictl logs [CONTAINER ID]

• To list pods :

marijan$ crictl pods

• To get low-level information on a container, image, or task, you can run the following command:

marijan$ crictl inspect [CONTAINER ID]

• To display a live stream of containers resource usage statistics, run:

marijan$ crictl stats [CONTAINER ID]

• To show the runtime version information :

marijan$ crictl version

Cluster Deployment (Linux)

In this section, I'll detail the deployment of my Kubernetes Cluster on a system Linux, including all the information regarding the tools and resources employed.

Prerequisites

Before setting up a Kubernetes cluster, several prerequisites must be done :

• Machines : If want to use Kubeadm, a minimum of two machines is necessary. Alternatively, Minikube can be used for single-machine setups.
• Operating system : The machines must run on an up-to-date operating system to ensure compatibility and security.
• Resources : Each machine should have at least 2 GB of RAM and 2 CPUs allocated to handle Kubernetes workloads effectively.
• Network connectivity : The machines must be able to communicate easily with each other to enable cluster communication and coordination.

In my setup, I've established two virtual machines running on Debian 11. To manage networking, I've implemented a firewall with configured rules to facilitate communication and provide access to these machines.

Installation

The installation process is the same for the Master and the Node machines.

Container Runtime (cri-dockerd)

Firstly, to run containers within Pods, you need a container runtime compatible with CRI. In our case, we'll use Docker Engine as the container runtime.

1. To install Docker Engine via the apt repository, you need to configure Docker's apt repository. First, Add Docker's official GPG key :

marijan$ apt-get install ca-certificates curl
marijan$ install -m 0755 -d /etc/apt/keyrings
marijan$ curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
marijan$ chmod a+r /etc/apt/keyrings/docker.asc

2. Then, add the repository to Apt sources and update the system :

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

marijan$ apt-get update

2. Next, install Docker packages by executing the following command :

marijan$ apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

3. Lastly, to verify if everything works well, run an image :

marijan$ docker run hello-world

As explained in the compatibility section, Docker Engine is no longer supported by Kubernetes. To make it compatible, you must install cri-dockerd. However, to install cri-dockerd, you need to install Go first, as cri-dockerd is developed using this programming language.

1. To install Go, you need to download it from the official website :

marijan$ wget https://go.dev/dl/go1.22.4.linux-amd64.tar.gz

2. Once it is downloaded, you need to extract the file from the archive :

marijan$ tar -xzf go1.22.4.linux-amd64.tar.gz -C /usr/share/

3. Then, add Go to the PATH environment variable :

marijan$ export PATH=$PATH:/usr/local/go/bin

4. Finally, verify if everything works correctly by running the following command :

marijan$ go version
go version go1.22.4 linux/amd64

Now that we've installed Go, we can set up cri-dockerd.

1. First, you need to install Git Tools if it's not already installed :

marijan$ apt-get install git-all -y

2. Then, clone the repository from GitHub :

marijan$ git clone https://github.com/Mirantis/cri-dockerd.git

3. Once cloned, create a bin folder and build CRI using Go :

marijan$ cd cri-dockerd
marijan:~/cri-dockerd$ mkdir bin
marijan:~/cri-dockerd$ go build -o bin/cri-dockerd

4. Create another bin folder under the directory /usr/local :

marijan$ mkdir /usr/local/bin/cri-dockerd

5. Install cri-dockerd by executing the following commands :

marijan:~/cri-dockerd/bin$ install -o root -g root -m 0755 cri-dockerd /usr/local/bin
marijan:~/cri-dockerd$ install packaging/systemd/* /etc/systemd/system
marijan:~/cri-dockerd$ sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service

6. Finally, reload the daemon and enable the plugin :

marijan$ systemctl daemon-reload
marijan$ systemctl enable --now cri-docker.socket

6. You can ensure everything works well by running :

marijan$ systemctl status cri-docker.socket

Kubeadm, Kubelet and Kubeclt

Now that our container runtime has been set up, we can install all the tools needed to configure and manage our Kubernetes cluster.

1. First, install the packages needed to use the Kubernetes apt repository :

marijan$ apt-get install -y apt-transport-https ca-certificates curl gpg

2. Then, download the public signing key for Kubernetes package repositories :

marijan$ curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

3. Add the appropriate Kubernetes apt repository :

marijan$ echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

4. Update the apt package index then install and pin kubelet, kubeadm, and kubectl ::

marijan$ apt-get update
marijan$ apt-get install -y kubelet kubeadm kubectl
marijan$ apt-mark hold kubelet kubeadm kubectl

5. Enable the kubelet service before running kubeadm :

marijan$ systemctl enable --now kubelet

6. Finally, you have to disable the swap on the system. For Debian, comment the line about the swap in the folder /etc/fstab and reboot your system :

marijan$ vim /etc/fstab

# swap was on /dev/sda5 during installation
# UUID=e5c9c68a-3429-49b9-8747-b3af73000bc5 none            swap    sw              0       0
# /dev/sr0        /media/cdrom0   udf,iso9660 user,noauto     0       0

marijan$ reboot

Configuration

The configuration process is different between the Master and the Node machines.

Setting up the Network (Calico)

To set up a network in your Kubernetes cluster, there are multiple add-ons available, each with different functionalities. This information can be found here. In our case, we will use Calico.

This section specifically for the Master. Calico should not be configured on Node machines.

1. Run the following command to initiate your pods network with the CIDR required by Calico. Specify the CRI socket as well, which is cri-dockerd in this case :

marijan$ kubeadm init --pod-network-cidr=192.168.0.0/16 --cri-socket=unix:///var/run/cri-dockerd.sock

2. Now that it has been initiated, configure kubectl :

marijan$ mkdir -p $HOME/.kube
marijan$ cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
marijan$ chown $(id -u):$(id -g) $HOME/.kube/config

3. Then, download the Calico networking manifest :

marijan$ curl https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml -O

4. Apply the manifest once it is downloaded :

marijan$ kubectl apply -f calico.yaml

5. Finally, check if Calico status is set to Running :

marijan$ kubectl get pods -A

NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-564985c589-qf9cs   1/1     Running   0          21m
kube-system   calico-node-qtvz2                          1/1     Running   0          21m
kube-system   coredns-7db6d8ff4d-6m78v                   1/1     Running   0          94m
kube-system   coredns-7db6d8ff4d-bzxk9                   1/1     Running   0          94m
kube-system   etcd-deb-vm-01                             1/1     Running   0          94m
kube-system   kube-apiserver-deb-vm-01                   1/1     Running   0          94m
kube-system   kube-controller-manager-deb-vm-01          1/1     Running   0          94m
kube-system   kube-proxy-k96tw                           1/1     Running   0          94m
kube-system   kube-scheduler-deb-vm-01                   1/1     Running   0          94m

Connecting Nodes

Once the network is set up, you can connect your Nodes to the cluster.

1. Firstly, you need to generate a token and the public key on the Master :

marijan$ kubeadm token create
5didvk.d09sbcov8ph2amjw

marijan$ openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \
> openssl dgst -sha256 -hex | sed 's/^.* //'

8cb2de97839780a412b93877f8507ad6c94f73add17d5d7058e91741c9d5ec78

2. Then, execute the join command on the Node machine:

marijan$ kubeadm join --token 5didvk.d09sbcov8ph2amjw 172.16.0.10:6443 --discovery-token-ca-cert-hash sha256:8cb2de97839780a412b93877f8507ad6c94f73add17d5d7058e91741c9d5ec78 --cri-socket=unix:///var/run/cri-dockerd.sock

• --token : token generate before
• HOST:PORT : (6443 is the default port for kubeadmin)
• --discovery-token-ca-cert-hash sha256: : public key generate before
• --cri-socket : Container Runtime used

Pods

When you deploy an application in a container, Kubernetes automatically creates a Pod, and the application is encapsulated inside it.

If you need to set up multiple instances of an application due to an increase in the number of users, you typically don't encapsulate the application in the same Pod, even if it's the same application running. Instead, you should create a new Pod with a new instance of the application.

However, you can have multiple containers in the same Pod if they are different types but complement each other.

Fichier:Kubernetes Pod.png

For example, you might associate Nginx and MySQL containers within the same Pod. The Pod handles network connectivity, shared volumes, monitoring of application states, and more. Without these applications in your Pod, you would need to establish all these parameters manually.

Another benefit of using a Pod is that if you need to delete, for example, the web server, the associated database will be deleted as well.

• With the kubectl command, you can list all of the Pods existing on the nodes, along with information such as their status, names, and more :

marijan$ kubectl get pods

NAME        READY   STATUS    RESTARTS   AGE
myapp-pod   1/1     Running   0          5m58s

• To get more information about a specific pod, you can run the following command, specifying the type of object (pod) and the name of the pod :

marijan$ kubectl describe pod myapp-pod

Name:             myapp-pod
Namespace:        default
Priority:         0
Service Account:  default
Node:             deb-vm-02/172.16.0.11
Start Time:       Mon, 10 Jun 2024 14:22:20 +0200
Labels:           app=myapp
                  type=front-end
Annotations:      cni.projectcalico.org/containerID: f12c3f7ba719ce8be72a6598d16a53e0d1abab90948d210e35b324c96ed1ed5e
                  cni.projectcalico.org/podIP: 192.168.98.65/32
                  cni.projectcalico.org/podIPs: 192.168.98.65/32
Status:           Running
IP:               192.168.98.65
IPs:
  IP:  192.168.98.65
Containers:
  nginx-container:
    Container ID:   docker://eb1d10777beb8dfff77aafb22b05859bbf19ebc6c5e37929c2aefea7f2a4453a
    Image:          nginx
    Image ID:       docker-pullable://nginx@sha256:0f04e4f646a3f14bf31d8bc8d885b6c951fdcf42589d06845f64d18aec6a3c4d
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Mon, 10 Jun 2024 14:22:40 +0200
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-sbnqm (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  kube-api-access-sbnqm:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  25s   default-scheduler  Successfully assigned default/myapp-pod to deb-vm-02
  Normal  Pulling    24s   kubelet            Pulling image "nginx"
  Normal  Pulled     12s   kubelet            Successfully pulled image "nginx" in 11.704s (11.704s including waiting). Image size: 187667860 bytes.
  Normal  Created    6s    kubelet            Created container nginx-container
  Normal  Started    6s    kubelet            Started container nginx-container

• To perform an action on a Pod or any other object (ReplicaSet, ReplicationController, and more), you need to run the kubectl command followed by the action, the type of object, and the name of the object :

marijan$ kubectl delete pod myapp-pod

• To get all Pods, ReplicaSet, Deployment and more running, you can run the following command :

marijan$ kubectl get all
NAME                         READY   STATUS    RESTARTS   AGE
pod/myapp-replicaset-4j8wk   1/1     Running   0          18h

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   4d21h

NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/myapp-replicaset   1         1         0       18h

Yet Another Markup Language (YAML)

Yet Another Markup Language (YAML) files are used as input for the creation of objects such as Pods, Deployments, Services, and more.

In Kubernetes, a .yml file begins with these four main sections, which are required in the configuration file :

• apiVersion : This specifies the version of the Kubernetes API to use. Each type of object has its own version, which could be different from others.
• kind : This specifies the type of object you are going to create.
• metadata : This section contains data about the object being deployed, such as names, labels, and application-specific information. All the data specified under metadata should be written as a dictionary. They will be the children of the metadata section, so proper indentation is required.
• spec : This section specifies the desired state of the object, such as the containers in a Pod. Similar to metadata, the data here is written as a dictionary. The dash "-" indicates the beginning of the first item in a list.

marijan$ cat pod-definition.yml

apiVersion: v1
kind: Pod
metadata:
    name: myapp-pod
    labels:
        app: myapp 
        type: front-end
spec:
    containers:
        - name: nginx-container
          image: nginx

• Once the file is ready to be deployed, you can run the following command to create the YAML configuration and create the resources in Kubernetes :

marijan$ kubectl create -f pod-definition.yml

• If you have a running pod and you don't have the .yml file, you can extract it to make modifications :

marijan$ kubectl get pod myapp-pod -o yaml > pod-definition.yaml

Replica Sets and Replication Controllers

A Replica Set ensures high availability of Pods in our cluster. It manages the replacement of failed Pods, load-balancing, and scaling in response to an increase in the number of users.

You can set up a Replica Set in different ways. First, if you have a single node with one Pod and that Pod fails, it will automatically be replaced.

Another possibility, as mentioned earlier, is that if the number of users increases, the Replica Set can manage load-balancing and scaling by duplicating the Pod or creating identical Pods on other nodes.

This setup is not only for load-balancing and scaling but also ensures high availability by replacing failed Pods.

The Replica Set has a predecessor called the Replication Controller. The Replication Controller is older and has been replaced by the Replica Set because it lacks the flexibility that the Replica Set offers.

To set up Replica Set, you have to implant that on your .yml file :

marijan$ cat replication-definition.yml

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: myapp-ReplicaSet
  labels:
    app: myapp
    type: front-end
spec:
  template:
    metadata:
      name: myapp-pod
      labels:
        app: myapp
        type: front-end
    spec:
      containers:
      - name: nginx-container
        image: nginx
  replicas: 1
  selector:
    matchLabels:
      type: front-end

As we done in the previous section about YAML, the four main sections are present: apiVersion, kind, metadata, and spec. In this section, we will change the apiVersion and kind. Instead of deploying a Pod, we will deploy a ReplicaSet.

We will add a template section, where we will provide the content of our Pod, including its name, labels, and containers in the spec section. Additionally, we will add replicas to specify the number of replicas we want to have.

Furthermore, we will add a selector. This is a major difference between a ReplicationController and a ReplicaSet, as the previous one does not provide this option.

The selector is used to avoid duplicating an existing Pod. Here is a scenario :

1. You already have a running Nginx pod.

marijan$ kubectl get pods
NAME        READY   STATUS    RESTARTS   AGE
nginx-pod   1/1     Running   0          2m43s

2. You then set up a ReplicationController file without a selector because this option is not available in it. Inside this file, you have the exact same pod configuration as the existing and running Pod. If you start this ReplicationController, it will run the pod without checking if it already exists. If the original Nginx Pod crashes, it will not be recreated because another Pod was already created :

marijan$ cat replicationcontroller-definition.yml

apiVersion: v1
kind: ReplicationController
metadata:
  name: myapp-ReplicationController
  labels:
    app: myapp
    type: front-end
spec:
  template:
    metadata:
      name: nginx-pod
      labels:
        app: nginx
        type: front-end
    spec:
      containers:
      - name: nginx-container
        image: nginx
  replicas: 1

marijan$ kubectl get pods
NAME                                READY   STATUS    RESTARTS   AGE
nginx-pod                           1/1     Running   0          2m43s
myapp-ReplicationController-4j8wk   1/1     Running   0          2m43s

3. With the selector included in a ReplicaSet, when you execute the YAML file, it will check if a pod with this label already exists. If it does, it will complete the number of Pods if any are missing, and if not, it will maintain the already existing ones.

To perform a scaling of your ReplicaSet without having to shut down pods, you can run the following command:

marijan$ kubectl replace -f replicaset-definition.yml

It is possible to modify a specific part of the .yml file, for example, to increase the number of replicas :

marijan$ kubectl scale --replicas=6 -f replicaset-defintion.yml

Deployments

The advantage of setting up a Deployment file on Kubernetes is that it automates various processes such as deployment, upgrading instances, undoing recent changes, and making multiple changes to the application.

To fully benefit from using a Deployment, you should have several Pods running. Because to avoid to interruput the service, if you have multiple Pods running, the Deployment will do it one by one.

In fact, within a Deployment, you will find a ReplicaSet, which is responsible for running all the containers and ensure if one of them crash. The Deployment sits above the ReplicaSet to automate various processes listed before.

Just like with a ReplicaSet, you need to set up a .yml file for a Deployment. The sections are exactly the same as those in a ReplicaSet, except for the kind :

marijan$ cat replication-definition.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-deployment
  labels:
    app: myapp
    type: front-end
spec:
  template:
    metadata:
      name: myapp-pod
      labels:
        app: myapp
        type: front-end
    spec:
      containers:
      - name: nginx-container
        image: nginx
  replicas: 3
  selector:
    matchLabels:
      type: front-end