Arch Linux; How to install on-premise Kubernetes using kubeadm

Table of Contents

TL;DR #

# Install
# used `paru`, wrapper of `pacman`
paru -Sy kubeadm
paru -Sy kubelet
paru -S kubectl

# Edit or Create files

# /etc/modules-load.d/k8s.conf
overlay
br_netfilter

# /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1

# check
lsmod | grep br_netfilter
lsmod | grep overlay

# check if its set to 1
sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward

# Install container runtime
paru -S containerd # must be installed with kubectl
sudo systemctl enable containerd

# Disable Swap Mem
sudo swapoff -a
sudo swapoff -a # temporary
sudo vim /etc/fstab # comment the line start with 'swap'
systemctl --type swap # check

# Choose cgroup driver

# If default config.toml not exists
sudo containerd config default > /etc/containerd/config.toml

# Edit /etc/containerd/config.toml like below 
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  ...
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

# need to start kubelet before kubeadm
sudo systemctl start kubelet # will fail before configuration is done
sudo systemctl enable kubelet

# start kubeadm
# --cri-socket is deprecated
# need to run with sudo
sudo kubeadm init --node-name k8s-node-a --pod-network-cidr='10.90.0.0/16' --control-plane-endpoint jaehong21.iptime.org:6443

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

export KUBECONFIG=/etc/kubernetes/admin.conf

# Install CNI

# ------ Flannel ------

kubectl create ns kube-flannel
kubectl label --overwrite ns kube-flannel pod-security.kubernetes.io/enforce=privileged

helm repo add flannel https://flannel-io.github.io/flannel/
helm install flannel --set podCidr="10.90.0.0/16" --namespace kube-flannel flannel/flannel

sudo systemctl restart containerd
sudo systemctl restart kubelet # must be done

# ------ Calico ------

# when NetworkManager is present
# Create file /etc/NetworkManager/conf.d/calico.conf
[keyfile]
unmanaged-devices=interface-name:cali*;interface-name:tunl*;interface-name:vxlan.calico;interface-name:vxlan-v6.calico;interface-name:wireguard.cali;interface-name:wg-v6.cali

# Tigera Calico operator and CRD

# Edit namespace `tiger-operator` to `calico-system`
# %s/namespace: tigera-operator/namespace: calico-system
kubectl create ns calico-system
curl https://raw.githubusercontent.com/projectcalico/calico/v3.27.2/manifests/tigera-operator.yaml > tigera-operator.yaml
kubectl apply -f tigera-operator.yaml -n calico-system

# creating the necessary CR
curl https://raw.githubusercontent.com/projectcalico/calico/v3.27.2/manifests/custom-resources.yaml > custom-resources.yaml
# edit custom-resources.yaml 
spec.calicoNetwork.ipPools.cidr: 10.90.0.0/16 # just like `pod-network-cidr`
kubectl apply -f custom-resources.yaml -n calico-system

# check until all pods to be READY
watch kubectl get pods -n calico-system
---
NAME                                      READY   STATUS    RESTARTS   AGE
calico-kube-controllers-b567b6fc7-srdn2   1/1     Running   0          2m44s
calico-node-4mpk4                         1/1     Running   0          2m35s
calico-typha-84fddcc7b8-nk9m6             1/1     Running   0          2m44s
csi-node-driver-tgpzq                     2/2     Running   0          2m44s
tigera-operator-748c69cf45-s56fb          1/1     Running   0          8m4s
---

# Remove the taints on the control plane to schedule pods on it
kubectl taint nodes --all node-role.kubernetes.io/control-plane-  
kubectl taint nodes --all node-role.kubernetes.io/master-

# ------

kubectl get node -o wide # now node need to be READY
---
NAME        STATUS   ROLES           AGE   VERSION
archlinux   Ready    control-plane   33m   v1.29.3

sudo systemctl restart containerd
sudo systemctl restart kubelet

# check valid time for certs
openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -text | grep Not

# install argocd
helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd --namespace argocd argo/argo-cd

Access from outside #

https://devopscube.com/kubernetes-kubeconfig-file/

Installation #

paru -Sy kubeadm
paru -Sy kubelet

paru -S kubectl

Prerequisites #

Create and Edit files like below

# /etc/modules-load.d/k8s.conf

overlay
br_netfilter

# /etc/sysctl.d/k8s.conf

net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1

sudo sysctl --system

# check
lsmod | grep br_netfilter
lsmod | grep overlay

# check if its set to 1
sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward

Install containerd #

paru -S containerd # must be installed with kubectl
sudo systemctl enable containerd

Configuration #

All nodes in a cluster (control-plane and worker) require a running instance of kubelet.service All provided systemd services accept CLI overrides in environment files:

kubelet.service: /etc/kubernetes/kubelet.env
kube-apiserver.service: /etc/kubernetes/kube-apiserver.env
kube-controller-manager.service: /etc/kubernetes/kube-controller-manager.env
kube-proxy.service: /etc/kubernetes/kube-proxy.env
kube-scheduler.service: /etc/kubernetes/kube-scheduler.env

Disable swap #

sudo swapoff -a
systemctl --type swap

Choose cgroup driver #

Remember Arch Linux use systemd as its init system (no matter you systemd-boot or GRUB as bootloader), so you need to choose systemd cgroup driver before deploying the control plane.

If etc/containerd/config.toml doesn’t exist, create the file with default configuration

sudo containerd config default > /etc/containerd/config.toml

To use the systemd cgroup driver in /etc/containerd/config.toml with runc, set

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  ...
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

Remember to restart containerd.service to make the change take effect. → sudo systemctl restart containerd

Choose container runtime interface (CRI) #

A container runtime has to be configured and started, before kubelet.service can make use of it. You will pass flag --cri-socket with the container runtime interface endpoint to kubeadm init or kubeadm join in order to create or join a cluster.

For example, if you choose containerd as CRI runtime, the flag --cri-socket will be:

kubeadm init --cri-socket /run/containerd/containerd.sock \
	--pod-network-cidr='10.90.0.0/16'

Containerd #

Before Kubernetes version 1.27.4, when using containerd as container runtime, it is required to provide kubeadm init or kubeadm join with its CRI endpoint. To do so, specify their flag --cri-socket to /run/containerd/containerd.sock[4].

kubeadm join --cri-socket=/run/containerd/containerd.sock

After Kubernetes version 1.27.4, kubeadm will auto detect this CRI for you, flag --cri-socket is only needed when you installed multiple CRI.

Choose cluster network parameter #

(Optional) Choose alternative node network proxy provider #

Node proxy provider like kube-proxy is a network proxy that runs on each node in your cluster, maintaining network rules on nodes to allow network communication to your Pods from network sessions inside or outside of your cluster.

By default kubeadm choose kube-proxy as the node proxy that runs on each node in your cluster. Container Network Interface (CNI) plugins like cilium offer a complete replacement for kube-proxy.

If you want to use cilium’s implementation of node network proxy to fully leverage cilium’s network policy feature, you should pass flag --skip-phases=addon/kube-proxy to kubeadm init to skip the install of kube-proxy.

Cilium will install a full replacement during its installation. See this[5] for details.

Create Cluster #

Before creating a new kubernetes cluster with kubeadm start and enable kubelet.service.

Initialize control-plane #

To initialize control-plane, you need pass the following necessary flags to kubeadm init

If run successfully, kubeadm init will have generated configurations for the kubelet and various control-plane components below /etc/kubernetes/ and /var/lib/kubelet/.

Finally, it will output commands ready to be copied and pasted to setup kubectl and make a worker node join the cluster (based on a token, valid for 24 hours).

To use kubectl with the freshly created control-plane node, setup the configuration (either as root or as a normal user):

Installing CNI plugins (pod network addon) #

Note: You must deploy a Container Network Interface (CNI) based Pod network add-on so that your Pods can communicate with each other. Cluster DNS (CoreDNS) will not start up before a network is installed.

Pod network add-on (CNI plugins) implements the Kubernetes network model[6] differently from simple solutions like flannel to more complicated solutions like calico

An increasingly adopted advanced CNI plugin is cilium, which achieves impressive performance with eBPF[7]. To install cilium as CNI plugin, use cilium-cli:

cilium-cli install

For more details on pod network, see this[8] official document.

Control plane node isolation #

By default, your cluster will not schedule Pods on the control plane nodes for security reasons. If you want to be able to schedule Pods on the control plane nodes, for example for a single machine Kubernetes cluster, run:

kubectl taint nodes --all node-role.kubernetes.io/control-plane-

Reset Kubeadm #

https://velog.io/@chan9708/Kubeadm-%EA%B9%94%EB%81%94%ED%95%98%EA%B2%8C-Reset

sudo kubeadm reset 
# unmount every configuration with kubeadm

# ---
The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the iptables command.

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.
# ---

sudo systemctl restart kubelet
sudo reboot

Troubleshooting #

failed to get cpu utilizations in kubeadm #

In kubeadm, metrics-server is not installed in default. So, errors like below can happen in hpa resources.

failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)

Reference: https://github.com/kubernetes-sigs/metrics-server/issues/196

In addition, while downloading the metrics-server , to make the pod to be READY, you must pass the --kubelet-insecure-tls to the argument. And it will work. But, this we need to make sure that this isn’t the safest way in kubeadm. It will just make it work. So, to prevent such issues, visit here for more: https://github.com/kubernetes-sigs/metrics-server/issues/196#issuecomment-835818605

Pod pending and stuck; “had untolerated taint {node.kubernetes.io/disk-pressure: }” #

Reference:

~~If actual disk is still idle just try, kubectl taint nodes archlinux node.kubernetes.io/disk-pressure:NoSchedule-. If doesn’t work, sudo systemctl restart kubelet.service~~

containerd’s container image and details about executed containers are stored at /var/lib/mysql at default. You can change this folder location to elsewhere that have sufficient free space.

sudo vim /lib/systemd/system/containerd.service
# add --data-root to ExecStart
# ExecStart=/usr/bin/containerd --data-root="/mnt/nfs/desktop/containerd"

# or
sudo vim /etc/containerd/config.toml
# root = "/mnt/nfs/desktop/containerd"