↓Skip to main content
  1. Posts/

[K3s-04] K3s Service LoadBalancer Configuration (Klipper and MetalLB)

·711 words
K3s
Table of Contents

As on-premise kubernetes cannot create its own operational LoadBalancer (ClusterIP and NodePort is still available). LoadBalancer type in kubernetes was originally designed to be operate externally from the cluster side. So we need to set additional configurations to use LoadBalancer type in kubernetes cluster.

In k3s, Traefik is is installed default as ingress controller. It deploys LoadBalancer on port 80 and 443. These ports should not be exposed with other HostPort or NodePort.

To use other LoadBalancer other than ServiceLB (such as MetalLB), configure all servers in the cluster with the --disable=servicelb flag.

From k3s Networking docs

Traefik’s inital config file is located at /var/lib/rancher/k3s/server/manifests/traefik.yaml. As it is an initial setting, you should not edit manually! Instead, you should customize Traefik by creating an additional HelmChartConfig manifest in /var/lib/rancher/k3s/server/manifests. For more details and an example see Customizing Packaged Components with HelmChartConfig. For more information on the possible configuration values, refer to the official Traefik Helm Configuration Parameters..

1) Klipper, default k3s Service LB #

In the first place it is good to set External IP for k3s-agent nodes (https://docs.k3s.io/cli/agent). The value below is the IP address of the node itself just for an example for myself.

  • control plane: 13.209.245.101
  • agent node: 43.202.94.188, 43.203.19.201
# at k3s-node-b, (43.202.94.188)
sudo k3s agent --node-external-ip 43.202.94.188 --server https://13.209.245.101:6443 --token XXX

# at k3s-node-c (43.203.19.201)
sudo k3s agent --node-external-ip 43.203.19.201 --server https://13.209.245.101:6443 --token XXX

# Needs a minute to apply on External IPs
kubectl get node -A -o wide

If the ServiceLB Pod runs on a node that has an external IP configured, the node’s external IP can be populated into the Service’s status.loadBalancer.ingress address list. Otherwise, the node’s internal IP is used.

To select specific node for ServiceLB, add svccontroller.k3s.cattle.io/enablelb=true label to one or more nodes. By default, nodes are not labeled. As long as all nodes remain unlabeled, all nodes with ports available will be used by ServiceLB. (+ Also, can label as group as Creating ServiceLB Node Pools)

Problem: When there are single nodes, or want to use running nodes to be LoadBalancer itself, there are some problems. LoadBalancers are hosted nodes where free hosts exist. But, all nodes have traefik in default with port 80 and 443. Then, how can we host LoadBalancer on k3s nodes?

Answer: Disabling traefik on k3s nodes, aren’t actually a valid solution. Because, when traefik is disabled using --disable traefik parameter, the web requests are not even handled.

# https://github.com/k3s-io/k3s/issues/2403#issuecomment-956220875
# Create directory
mkdir -p /var/lib/rancher/k3s/server/manifests/

# Create traefik-config.yaml
cat <<EOF > /var/lib/rancher/k3s/server/manifests/traefik-config.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    ports:
       web:
         exposedPort: 8088

       websecure:
         port: 8443
         expose: true
         exposedPort: 8443

EOF

# Install k3s
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--https-listen-port 7443" sh -

By changing the port of traefik, we can use port 80 and 443 for LoadBalancer. But, to change k3s default configuration, it must be done before installing k3s.

2) MetalLB configuration (WIP) #

To disable ServiceLB, configure all servers in the cluster with the --disable=servicelb flag. This is necessary if you wish to run a different LB, such as MetalLB.

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.12/config/manifests/metallb-native.yaml

will create metallb-system namespace

  • metallb-system/controller: controller that handles IP address assignment
  • metallb-system/speaker: daemonset that handle protocols
  • Service accounts and RBAC permissions

Two modes on MetalLB https://velog.io/@youwins/MetalLB

  • L2 mode: k3s node broadcast and accepts ARP requests to make k3s nodes act as LoadBalancer
  • BGP mode: To put an external LoadBalancer at the outside of the cluster, and connect by BGP advertisement.

+) Q. What is FRR mode in MetalLB?

The most simple way to expose k3s service to outside is using Traefik Ingress. As Traefik and its Ingress is configured default in k3s. By just simply, creating Ingress resource, and write the name of the service(in this case, nginx-deployment) to expose, it will be exposed to outside.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: traefik-ingress
spec:
  ingressClassName: traefik
  rules:
    - host: k3s.jaehong21.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nginx-deployment
                port:
                  number: 80

In case of attaching TLS certificate to the ingress, refer to the next article

Reference #