[K3s-03] How to Inject Docker Credentials to K3s

Table of Contents

In AWS EKS, it has its AWS ECR credentials in default. But, in case of on-premise environemnt, it needs to inject docker crendentials directly (to pull images from private docker registry). Basic idea is to fetch ECR Authentication token from AWS with cronjob that creates an secret (dockerconfigjson) named regcred (you can change the name, regcred). Let’s set our example namespace as practice

apiVersion: v1
kind: Secret
metadata:
  name: ecr-registry-helper-secrets
  namespace: practice
stringData:
  AWS_SECRET_ACCESS_KEY: "XXX" # Replace with your AWS secret access key
  AWS_ACCESS_KEY_ID: "XXX" # Replace with your AWS access key ID
  AWS_ACCOUNT: "XXX" # Replace with your AWS account ID
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: ecr-registry-helper-cm
  namespace: practice
data:
  AWS_REGION: "ap-northeast-2" # Replace with your ECR region
  DOCKER_SECRET_NAME: regcred # Replace with your desired ECR token secret name
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-default
  namespace: default

Now, create a Role and a RoleBinding that grants CronJob to delete and create secrets.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: practice
  name: role-full-access-to-secrets
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["regcred"] # Replace with your desired ECR token secret name
    verbs: ["delete"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["create"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: default-role-binding
  namespace: practice
subjects:
  - kind: ServiceAccount
    name: sa-default # Replace with your service account name if different
    namespace: practice
    apiGroup: ""
roleRef:
  kind: Role
  name: role-full-access-to-secrets # Replace with your role name if different
  apiGroup: ""

Lastly, create a CronJob. Image omarxs/awskctl have been configured to use aws command-line inside the container. Also, don’t forget to set the namespace.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: ecr-registry-helper
  namespace: default
spec:
  schedule: "0 */10 * * *" # Replace with your desired schedule
  successfulJobsHistoryLimit: 2
  suspend: false
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: sa-default # Replace with your service account name if different
          containers:
            - name: ecr-registry-helper
              image: omarxs/awskctl:v1.0
              imagePullPolicy: IfNotPresent
              envFrom:
                - secretRef:
                    name: ecr-registry-helper-secrets # Replace with your secret name if different
                - configMapRef:
                    name: ecr-registry-helper-cm # Replace with your configmap name if different
              command:
                - /bin/bash
                - -c
                - |-
                  ECR_TOKEN="$(aws ecr get-login-password --region ${AWS_REGION})"
                  NAMESPACE_NAME=practice # Replace with your desired namespace
                  kubectl delete secret --ignore-not-found $DOCKER_SECRET_NAME -n $NAMESPACE_NAME
                  kubectl create secret docker-registry $DOCKER_SECRET_NAME --docker-server=https://${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com --docker-username=AWS --docker-password=${ECR_TOKEN} --namespace=$NAMESPACE_NAME
                  echo "Secret was successfully updated at $(date)"                  
          restartPolicy: Never

This cronjob will run every 10 hours. As, AWS ECR token is valid for 12 hours, it will be refreshed before it expires. Additionally, how to execute the cronjob manually

kubectl create job --from=cronjob/ecr-registry-helper ecr-registry-helper-manual -n practice
# Replace with your CronJob name if different

Check whether it shows this kind of message when looking at the logs: Secret was successfully updated at Sat Jan 1 00:00:00 UTC 2023

# delete manual job
kubectl delete job ecr-registry-helper-manual -n practice # Replace with your job name and namespace if different

Now, all you need to do is put imagePullSecrets on the Pod spec.

# Example Pod
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
  namespace: practice
spec:
  containers:
  - name: test-container
    image: <your-account>.dkr.ecr.<your-region>.amazonaws.com/<your-image>:<your-tag> # Replace with your ECR image
    imagePullPolicy: Always
  imagePullSecrets:
  - name: regcred # Replace with your secret name if different

TL;DR #

apiVersion: v1
kind: Secret
metadata:
  name: ecr-registry-helper-secrets
  namespace: default
stringData:
  AWS_SECRET_ACCESS_KEY: "" # Replace with your AWS secret access key
  AWS_ACCESS_KEY_ID: "" # Replace with your AWS access key ID
  AWS_ACCOUNT: "" # Replace with your AWS account ID
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: ecr-registry-helper-cm
  namespace: default
data:
  AWS_REGION: "eu-central-1" # Replace with your ECR region
  DOCKER_SECRET_NAME: regcred # Replace with your desired ECR token secret name
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-default
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: role-full-access-to-secrets
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["regcred"] # Replace with your desired ECR token secret name
    verbs: ["delete"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["create"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: default-role-binding
  namespace: default
subjects:
  - kind: ServiceAccount
    name: sa-default # Replace with your service account name if different
    namespace: default
    apiGroup: ""
roleRef:
  kind: Role
  name: role-full-access-to-secrets # Replace with your role name if different
  apiGroup: ""
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: ecr-registry-helper
  namespace: default
spec:
  schedule: "0 */10 * * *" # Replace with your desired schedule
  successfulJobsHistoryLimit: 2
  suspend: false
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: sa-default # Replace with your service account name if different
          containers:
            - name: ecr-registry-helper
              image: omarxs/awskctl:v1.0
              imagePullPolicy: IfNotPresent
              envFrom:
                - secretRef:
                    name: ecr-registry-helper-secrets # Replace with your secret name if different
                - configMapRef:
                    name: ecr-registry-helper-cm # Replace with your configmap name if different
              command:
                - /bin/bash
                - -c
                - |-
                  ECR_TOKEN="$(aws ecr get-login-password --region ${AWS_REGION})"
                  NAMESPACE_NAME=default # Replace with your desired namespace
                  kubectl delete secret --ignore-not-found $DOCKER_SECRET_NAME -n $NAMESPACE_NAME
                  kubectl create secret docker-registry $DOCKER_SECRET_NAME --docker-server=https://${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com --docker-username=AWS --docker-password=${ECR_TOKEN} --namespace=$NAMESPACE_NAME
                  echo "Secret was successfully updated at $(date)"                  
          restartPolicy: Never

registries.yaml #

In case of K3s, it supports additional private registry configuration. Upon startup, K3s will check to see if /etc/rancher/k3s/registries.yaml exists. If so, the registry configuration contained in this file is used when generating the containerd configuration.

So, when its credentials are correctly congirued in the yaml file, adding imagePullSecrets on the Deployments in kuberenetes are unecessary anymore.

mirrors:
  <REGISTRY>:
    endpoint:
      - https://<REGISTRY>/v2
configs:
  <REGISTRY>:
    auth:
      username: <BASIC AUTH USERNAME>
      password: <BASIC AUTH PASSWORD>
      token: <BEARER TOKEN>
    tls:
      ca_file: <PATH TO SERVER CA>
      cert_file: <PATH TO CLIENT CERT>
      key_file: <PATH TO CLIENT KEY>
      insecure_skip_verify: <SKIP TLS CERT VERIFICATION BOOLEAN>

TL;DR #

registries.yaml #

Reference #