[K3s-03] How to Inject Docker Credentials to K3s
Table of Contents
In AWS EKS, it has its AWS ECR credentials in default. But, in case of on-premise environemnt, it needs to inject docker crendentials directly (to pull images from private docker registry). Basic idea is to fetch ECR Authentication token from AWS with cronjob that creates an secret (dockerconfigjson) named regcred
(you can change the name, regcred
).
Let’s set our example namespace as practice
apiVersion: v1
kind: Secret
metadata:
name: ecr-registry-helper-secrets
namespace: practice
stringData:
AWS_SECRET_ACCESS_KEY: "XXX" # Replace with your AWS secret access key
AWS_ACCESS_KEY_ID: "XXX" # Replace with your AWS access key ID
AWS_ACCOUNT: "XXX" # Replace with your AWS account ID
---
apiVersion: v1
kind: ConfigMap
metadata:
name: ecr-registry-helper-cm
namespace: practice
data:
AWS_REGION: "ap-northeast-2" # Replace with your ECR region
DOCKER_SECRET_NAME: regcred # Replace with your desired ECR token secret name
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: sa-default
namespace: default
Now, create a Role
and a RoleBinding
that grants CronJob
to delete and create secrets.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: practice
name: role-full-access-to-secrets
rules:
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["regcred"] # Replace with your desired ECR token secret name
verbs: ["delete"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: default-role-binding
namespace: practice
subjects:
- kind: ServiceAccount
name: sa-default # Replace with your service account name if different
namespace: practice
apiGroup: ""
roleRef:
kind: Role
name: role-full-access-to-secrets # Replace with your role name if different
apiGroup: ""
Lastly, create a CronJob
. Image omarxs/awskctl
have been configured to use aws command-line inside the container. Also, don’t forget to set the namespace.
apiVersion: batch/v1
kind: CronJob
metadata:
name: ecr-registry-helper
namespace: default
spec:
schedule: "0 */10 * * *" # Replace with your desired schedule
successfulJobsHistoryLimit: 2
suspend: false
jobTemplate:
spec:
template:
spec:
serviceAccountName: sa-default # Replace with your service account name if different
containers:
- name: ecr-registry-helper
image: omarxs/awskctl:v1.0
imagePullPolicy: IfNotPresent
envFrom:
- secretRef:
name: ecr-registry-helper-secrets # Replace with your secret name if different
- configMapRef:
name: ecr-registry-helper-cm # Replace with your configmap name if different
command:
- /bin/bash
- -c
- |-
ECR_TOKEN="$(aws ecr get-login-password --region ${AWS_REGION})"
NAMESPACE_NAME=practice # Replace with your desired namespace
kubectl delete secret --ignore-not-found $DOCKER_SECRET_NAME -n $NAMESPACE_NAME
kubectl create secret docker-registry $DOCKER_SECRET_NAME --docker-server=https://${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com --docker-username=AWS --docker-password=${ECR_TOKEN} --namespace=$NAMESPACE_NAME
echo "Secret was successfully updated at $(date)"
restartPolicy: Never
This cronjob will run every 10 hours. As, AWS ECR token is valid for 12 hours, it will be refreshed before it expires. Additionally, how to execute the cronjob manually
kubectl create job --from=cronjob/ecr-registry-helper ecr-registry-helper-manual -n practice
# Replace with your CronJob name if different
Check whether it shows this kind of message when looking at the logs: Secret was successfully updated at Sat Jan 1 00:00:00 UTC 2023
# delete manual job
kubectl delete job ecr-registry-helper-manual -n practice # Replace with your job name and namespace if different
Now, all you need to do is put imagePullSecrets
on the Pod spec.
# Example Pod
apiVersion: v1
kind: Pod
metadata:
name: test-pod
namespace: practice
spec:
containers:
- name: test-container
image: <your-account>.dkr.ecr.<your-region>.amazonaws.com/<your-image>:<your-tag> # Replace with your ECR image
imagePullPolicy: Always
imagePullSecrets:
- name: regcred # Replace with your secret name if different
TL;DR #
apiVersion: v1
kind: Secret
metadata:
name: ecr-registry-helper-secrets
namespace: default
stringData:
AWS_SECRET_ACCESS_KEY: "" # Replace with your AWS secret access key
AWS_ACCESS_KEY_ID: "" # Replace with your AWS access key ID
AWS_ACCOUNT: "" # Replace with your AWS account ID
---
apiVersion: v1
kind: ConfigMap
metadata:
name: ecr-registry-helper-cm
namespace: default
data:
AWS_REGION: "eu-central-1" # Replace with your ECR region
DOCKER_SECRET_NAME: regcred # Replace with your desired ECR token secret name
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: sa-default
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: default
name: role-full-access-to-secrets
rules:
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["regcred"] # Replace with your desired ECR token secret name
verbs: ["delete"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: default-role-binding
namespace: default
subjects:
- kind: ServiceAccount
name: sa-default # Replace with your service account name if different
namespace: default
apiGroup: ""
roleRef:
kind: Role
name: role-full-access-to-secrets # Replace with your role name if different
apiGroup: ""
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: ecr-registry-helper
namespace: default
spec:
schedule: "0 */10 * * *" # Replace with your desired schedule
successfulJobsHistoryLimit: 2
suspend: false
jobTemplate:
spec:
template:
spec:
serviceAccountName: sa-default # Replace with your service account name if different
containers:
- name: ecr-registry-helper
image: omarxs/awskctl:v1.0
imagePullPolicy: IfNotPresent
envFrom:
- secretRef:
name: ecr-registry-helper-secrets # Replace with your secret name if different
- configMapRef:
name: ecr-registry-helper-cm # Replace with your configmap name if different
command:
- /bin/bash
- -c
- |-
ECR_TOKEN="$(aws ecr get-login-password --region ${AWS_REGION})"
NAMESPACE_NAME=default # Replace with your desired namespace
kubectl delete secret --ignore-not-found $DOCKER_SECRET_NAME -n $NAMESPACE_NAME
kubectl create secret docker-registry $DOCKER_SECRET_NAME --docker-server=https://${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com --docker-username=AWS --docker-password=${ECR_TOKEN} --namespace=$NAMESPACE_NAME
echo "Secret was successfully updated at $(date)"
restartPolicy: Never
registries.yaml #
In case of K3s, it supports additional private registry configuration. Upon startup, K3s will check to see if /etc/rancher/k3s/registries.yaml
exists. If so, the registry configuration contained in this file is used when generating the containerd configuration.
So, when its credentials are correctly congirued in the yaml file, adding imagePullSecrets
on the Deployments in kuberenetes are unecessary anymore.
mirrors:
<REGISTRY>:
endpoint:
- https://<REGISTRY>/v2
configs:
<REGISTRY>:
auth:
username: <BASIC AUTH USERNAME>
password: <BASIC AUTH PASSWORD>
token: <BEARER TOKEN>
tls:
ca_file: <PATH TO SERVER CA>
cert_file: <PATH TO CLIENT CERT>
key_file: <PATH TO CLIENT KEY>
insecure_skip_verify: <SKIP TLS CERT VERIFICATION BOOLEAN>