Kluctl Blog

• Managing many Helm Charts with Kluctl
• Introducing the Template Controller and building GitOps Preview Environments
• Templating in Kustomize with Kluctl style deploy and diff
• Multiple Environments with Flux and Kluctl
• Rethinking Kubernetes Configuration Management
• Kluctl and Flux
• Hello kluctl fans!

Managing many Helm Charts with Kluctl

If you are a DevOps engineer, SRA, a Homelab enthusiast or in any other comparable role that needs to manage Kubernetes deployments, you’ll probably have made the same journey that many others have gone. You started with plain YAML manifests deployed via kubectl and eventually found out about Helm and Kustomize.

Whatever tool you decide to go for, the next realisation is usually that Helm and Kustomize itself need to be managed as well. The naive solution is to create shell scripts that perform the required helm install and helm upgrade commands, but these easily turn out to be unmanageable and quite limited.

There are multiple solutions available that allow you to manage your (and all third-party) Helm Charts and Kustomize deployments. Examples are Flux, Argo CD, Helmfile and Kluctl. This blog post will concentrate on Kluctl and at the end shortly compare it with the other options.

The final project that results from this post can be found in the kluctl-examples repository .

What to learn from this post?

I hope this post allows its readers to learn how easy it can be to manage multiple Helm Charts from one deployment project while leveraging the following features and advantages of Kluctl:

1. Keep full control over deployment order (e.g. cert-manager after Cilium and apps after all base deployments)
2. Organise your deployment project in a way that feels natural (a straightforward folder structure)
3. Use the Kluctl CLI to diff and deploy and always be confident about the consequences of your deployment

What to deploy?

We will deploy two categories of deployments. The first category is a collection of “base” deployments, namely the following Helm Charts:

1. Cilium
2. cert-manager
3. ingress-nginx

The second category contains “apps”, or the podinfo microservice to be specific. It should be easy to apply the same ideas you learn in this post in case you want to add more apps to the deployment.

The reason for this split is that it allows us to leverage Kluctl’s abilities to do dependency management and ordered deployments. It will also allow us to split up the project into multiple projects later (in a future blog post).

You might have noticed that we also deploy Cilium, which is usually unnecessary when deploying to a local cluster. I decided to still do this in this tutorial, as I believe that we can learn a lot from this, especially in regard to deployment order and dependency management (e.g., without a CNI, nothing else will work).

Setting up a local cluster

We’ll use kind for the local cluster. As we need to disable a few things (CNI and kube-proxy), and also expose some ports (for ingress-nginx), let’s first create a proper kind configuration under ./kind-config.yaml:

# three node (two workers) cluster config
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    # this is required for ingress-nginx to work
    kubeadmConfigPatches:
      - |
        kind: InitConfiguration
        nodeRegistration:
          kubeletExtraArgs:
            node-labels: "ingress-ready=true"        
    extraPortMappings:
      - containerPort: 80
        hostPort: 8080
        listenAddress: 127.0.0.1
        protocol: TCP
      - containerPort: 443
        hostPort: 4443
        listenAddress: 127.0.0.1
        protocol: TCP
  - role: worker
  - role: worker
networking:
  # we use Cilium, so let's disable the default CNI
  disableDefaultCNI: true
  kubeProxyMode: none

Now let’s create the kind cluster:

$ kind create cluster --name kluctl-tutorial --config kind-config.yaml
Creating cluster "kluctl-tutorial" ...
 ✓ Ensuring node image (kindest/node:v1.25.3) 🖼 
 ✓ Preparing nodes 📦 📦 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing StorageClass 💾 
 ✓ Joining worker nodes 🚜 
Set kubectl context to "kind-kluctl-tutorial"
You can now use your cluster with:

kubectl cluster-info --context kind-kluctl-tutorial

Thanks for using kind! 😊

If you check the PODs of the cluster now, you’ll see:

$ kubectl get pod -A
kube-system          coredns-565d847f94-hcf4m                                0/1     Pending   0          4m37s
kube-system          coredns-565d847f94-md625                                0/1     Pending   0          4m37s
kube-system          etcd-kluctl-tutorial-control-plane                      1/1     Running   0          4m51s
kube-system          kube-apiserver-kluctl-tutorial-control-plane            1/1     Running   0          4m50s
kube-system          kube-controller-manager-kluctl-tutorial-control-plane   1/1     Running   0          4m51s
kube-system          kube-scheduler-kluctl-tutorial-control-plane            1/1     Running   0          4m51s
local-path-storage   local-path-provisioner-684f458cdd-zl8xj                 0/1     Pending   0          4m37s

Please note that multiple PODs are in pending state, which is because there is no CNI deployed. We’ll fix this as the first step now.

Setting up the basic project structure

Let’s first create some basic project structure. A Kluctl deployment project starts with a root deployment.yaml which usually just includes more deployment projects and items .

A deployment item is a collection of Kubernetes manifests, either directly provided or generated from Kustomize or Helm.

Our root deployment yaml (./deployment.yaml) should look like this:

deployments:
  - include: base

This tells Kluctl that a sub-deployment should be included from the path ./base.

Now create ./base/deployment.yaml:

deployments:
  - path: cilium

This tells Kluctl that a deployment item at the location ./base/cilium should be deployed. The referenced folder can contain plain manifests, a Kustomize deploymnet or a reference to a Helm Chart and the necessary Helm Values.

In this case, it will be a Helm Chart.

Adding the first Helm Chart (Cilium)

A Helm based deployment item can be specified by putting a helm-chart.yaml file into the item’s directory. This means, we have to add ./base/cilium/helm-chart.yaml with the following content:

helmChart:
  repo: https://helm.cilium.io/
  chartName: cilium
  chartVersion: 1.13.0
  skipPrePull: true
  releaseName: cilium
  namespace: "kube-system"

If you’re used to Helm already, you’ll probably recognise most of the parameters for the helmChart already. You basically provide the repository URL, the chart name and version and some information about the release to be deployed.

The configuration/values for the Helm Release are passed via the helm-values.yaml file, which must be put along the helm-chart.yaml, so it must be at ./base/cilium/helm-values.yaml. For our Cilium inside Kind example, these are good values:

nodeinit:
  enabled: true
kubeProxyReplacement: strict
hostServices:
  enabled: false
externalIPs:
  enabled: true
nodePort:
  enabled: true
hostPort:
  enabled: true
bpf:
  masquerade: false
image:
  pullPolicy: IfNotPresent
ipam:
  mode: kubernetes

# See https://medium.com/@charled.breteche/kind-cluster-with-cilium-and-no-kube-proxy-c6f4d84b5a9d for details
k8sServiceHost: kluctl-tutorial-control-plane
k8sServicePort: 6443

The above values are loosely based on the Kind cluster with Cilium and no kube-proxy .

Let’s deploy it!

That’s all we need for a working and minimalistic Kluctl deployment. Let’s actually deploy it now (press y when asked!):

$ kluctl deploy
✓ Loading kluctl project
✓ Initializing k8s client
✓ Rendering templates
✓ Pulling Helm Chart cilium with version 1.13.0
✓ Rendering Helm Charts
✓ Building kustomize objects
✓ Postprocessing objects
⚠ No discriminator configured. Orphan object detection will not work
⚠ No discriminator configured for target, retrieval of remote objects will be slow.
✓ Getting 16 additional remote objects
✓ Getting namespaces
✓ cilium: Applied 16 objects.
✓ Running server-side apply for all objects

Warnings:
  no discriminator configured. Orphan object detection will not work

New objects:
  ClusterRole/cilium
  ClusterRole/cilium-operator
  ClusterRoleBinding/cilium
  ClusterRoleBinding/cilium-operator
  kube-system/ConfigMap/cilium-config
  kube-system/DaemonSet/cilium
  kube-system/DaemonSet/cilium-node-init
  kube-system/Deployment/cilium-operator
  kube-system/Role/cilium-config-agent
  kube-system/RoleBinding/cilium-config-agent
  kube-system/Secret/cilium-ca
  kube-system/Secret/hubble-ca-secret
  kube-system/Secret/hubble-server-certs
  kube-system/Service/hubble-peer
  kube-system/ServiceAccount/cilium
  kube-system/ServiceAccount/cilium-operator
? The diff succeeded, do you want to proceed? (y/N) y
✓ cilium: Applied 16 objects.
✓ Running server-side apply for all objects

New objects:
  ClusterRole/cilium
  ClusterRole/cilium-operator
  ClusterRoleBinding/cilium
  ClusterRoleBinding/cilium-operator
  kube-system/ConfigMap/cilium-config
  kube-system/DaemonSet/cilium
  kube-system/DaemonSet/cilium-node-init
  kube-system/Deployment/cilium-operator
  kube-system/Role/cilium-config-agent
  kube-system/RoleBinding/cilium-config-agent
  kube-system/Secret/cilium-ca
  kube-system/Secret/hubble-ca-secret
  kube-system/Secret/hubble-server-certs
  kube-system/Service/hubble-peer
  kube-system/ServiceAccount/cilium
  kube-system/ServiceAccount/cilium-operator

This should be pretty fast as it does not wait for readiness of Cilium. After a few seconds or minutes, you should see:

$ kubectl get pod -A 
NAMESPACE            NAME                                                    READY   STATUS    RESTARTS   AGE
kube-system          cilium-g46vp                                            1/1     Running   0          2m11s
kube-system          cilium-n5ccg                                            1/1     Running   0          2m11s
kube-system          cilium-node-init-6gdcz                                  1/1     Running   0          2m11s
kube-system          cilium-node-init-8p6jj                                  1/1     Running   0          2m11s
kube-system          cilium-node-init-vswr8                                  1/1     Running   0          2m11s
kube-system          cilium-operator-74fc7f9d76-hc7fl                        1/1     Running   0          2m11s
kube-system          cilium-operator-74fc7f9d76-sdw9x                        1/1     Running   0          2m11s
kube-system          cilium-sqs26                                            1/1     Running   0          2m11s
kube-system          coredns-565d847f94-hcf4m                                1/1     Running   0          134m
kube-system          coredns-565d847f94-md625                                1/1     Running   0          134m
kube-system          etcd-kluctl-tutorial-control-plane                      1/1     Running   0          134m
kube-system          kube-apiserver-kluctl-tutorial-control-plane            1/1     Running   0          134m
kube-system          kube-controller-manager-kluctl-tutorial-control-plane   1/1     Running   0          134m
kube-system          kube-scheduler-kluctl-tutorial-control-plane            1/1     Running   0          134m
local-path-storage   local-path-provisioner-684f458cdd-zl8xj                 1/1     Running   0          134m

This shows us that the CNI has initialised properly and all PODs are running now.

A few words about deployment order

I assume it’s clear that whatever we’re going to add to the deployment project from now on, must be deployed AFTER Cilium got deployed. The reasons for this are simple: Nearly nothing can run without a CNI.

Deployment order is something that can be easily controlled in Kluctl deployment projects. By default, Kluctl deploys all deployment items in parallel to get as much performance as possible. Consider the following example deployment.yaml:

deployments:
  - path: a
  - path: b
  - path: c

This example will cause a, b and c to be deployed in parallel. If you want b and c to be deployed after a has been deployed, simply put a barrier between a and b:

deployments:
  - path: a
  - barrier: true
  - path: b
  - path: c

This will cause Kluctl to stop and wait at the barrier and only continue when everything before (in this case just a) got deployed. The same works with include and any other deployment item. We’ll leverage the barriers in the next section.

Adding the cert-manager Helm Chart

Adding the cert-manager Helm Chart is basically the same as we’ve seen with the Cilium Helm Chart. Open the ./base/deployment.yaml file and add a new deployment item:

deployments:
  - path: cilium
  - barrier: true
  - path: cert-manager

The barrier is important here, as noted in the previous section. It will ensure that all Cilium components are deployed before Kluctl continues with cert-manager.

Now, add ./base/cert-manager/helm-chart.yaml with the following content:

helmChart:
  repo: https://charts.jetstack.io
  chartName: cert-manager
  chartVersion: v1.11.0
  releaseName: cert-manager
  namespace: "kube-system"
  skipPrePull: true

cert-manager is a little special when it comes to installing the CRDs. It skips doing this by default and thus requires the following ./base/cert-manager/helm-values.yaml:

installCRDs: true

Deploy it again

Now perform another deployment. You’ll notice that it will first show you what will change and then ask for confirmation again. Press y when asked for confirmation:

$ kluctl deploy
✓ Loading kluctl project
✓ Initializing k8s client
✓ Rendering templates
✓ Pulling Helm Chart cilium with version 1.13.0
✓ Rendering Helm Charts
✓ Building kustomize objects
✓ Postprocessing objects
⚠ No discriminator configured. Orphan object detection will not work
⚠ No discriminator configured for target, retrieval of remote objects will be slow.
✓ Getting 67 additional remote objects
✓ Getting namespaces
✓ cilium: Applied 18 objects.
✓ Finished waiting
✓ cert-manager: Applied 49 objects. Applied 4 hooks.
✓ Running server-side apply for all objects

Warnings:
  no discriminator configured. Orphan object detection will not work

New objects:
  ClusterRole/cert-manager-cainjector
  ...snip...
  kube-system/ServiceAccount/cert-manager-webhook

Applied hooks:
  kube-system/ServiceAccount/cert-manager-startupapicheck
  kube-system/Role/cert-manager-startupapicheck:create-cert
  kube-system/RoleBinding/cert-manager-startupapicheck:create-cert
  kube-system/Job/cert-manager-startupapicheck
? The diff succeeded, do you want to proceed? (y/N) y
✓ cilium: Applied 18 objects.
✓ Finished waiting
✓ cert-manager: Applied 49 objects. Applied 4 hooks. Deleted 4 hooks.
✓ Running server-side apply for all objects

New objects:
  ClusterRole/cert-manager-cainjector
  ...snip...
  kube-system/ServiceAccount/cert-manager-webhook

Applied hooks:
  kube-system/ServiceAccount/cert-manager-startupapicheck
  kube-system/Role/cert-manager-startupapicheck:create-cert
  kube-system/RoleBinding/cert-manager-startupapicheck:create-cert
  kube-system/Job/cert-manager-startupapicheck

After a few seconds or minutes you should see cert-manager up and running in your cluster. We won’t further use cert-manager as it would be out of scope for this tutorial. I assume you can imagine how you would proceed with deployments that need to create cert-manager issuers and certificates (you’d use barriers and an appropriate project structure).

Add the ingress-nginx Helm Chart

Now let’s add the ingress-nginx Helm Chart. It’s important that you have used the kind configuration specified here , as otherwise the ingress controller won’t be accessible from your machine.

First, update ./base/deployment.yaml to include two new deployment items:

deployments:
  - path: namespaces
  - barrier: true
  - path: cilium
  - barrier: true
  - path: cert-manager
  - path: ingress-nginx
    waitReadiness: true

The first new deployment item is namespaces, which should contain a single file named ingress-nginx.yaml with the following content:

apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx

This deployment item and the barrier entry ensure that the required ingress-nginx namespace is created before we try to deploy the ingress-nginx Helm Chart. This is a pattern that you’ll encounter in nearly all Kluctl based deployments. It is also one of the best and easiest examples why ordering is important: You can’t deploy namespaced objects before the target namespace is deployed.

Next, create the file ./base/ingress-nginx/helm-chart.yaml with the following content:

helmChart:
  repo: https://kubernetes.github.io/ingress-nginx
  chartName: ingress-nginx
  # Please don't try to manually upgrade this to the latest version yet. We will use this to demonstrate an upgrade later!
  chartVersion: 4.5.0
  skipPrePull: true
  releaseName: ingress-nginx
  namespace: ingress-nginx

For ./base/ingress-nginx/helm-values.yaml, you must use:

controller:
  hostNetwork: true
  service:
    type: ClusterIP
  nodeSelector:
    ingress-ready: "true"
  tolerations:
    - key: "node-role.kubernetes.io/control-plane"
      effect: "NoSchedule"
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate

This configuration ensures that the ingress controller runs on the control-plane nodes and uses the host network of these. The extraPortMappings of the kind configuration will then allow you to access the ingress controller via localhost, or via http://my-ingress.127.0.0.1.nip.io:8080 (see https://nip.io ).

The waitReadiness: true in the deployment.yaml means that Kluctl should not just deploy the resources of the deployment item, but also wait for readiness of each resource.

Deploy it again

Simply run kluctl deploy again and confirm it with y. You should see how Kluctl deploys all resources, runs hooks and waits for readiness.

Add the podinfo application

We now have a somewhat functional cluster up and running locally. We can now also deploy the podinfo Helm Chart and expose it via ingress-nginx.

Let’s however first extend the project structure a bit. Edit the root deployment at ./deployment.yaml and change it to:

deployments:
  - include: base
  - barrier: true
  - include: apps

This will ensure that everything from base is deployed before it starts to deploy anything from apps. The apps sub-deployment can now be used to deploy all kinds of apps, for example the podinfo “microservice”.

Now create the file ./apps/deployment.yaml with the following content:

deployments:
  - path: podinfo

And ./apps/podinfo/helm-chart.yaml with the following content:

helmChart:
  repo: oci://ghcr.io/stefanprodan/charts/podinfo
  chartVersion: 6.3.4
  releaseName: podinfo
  namespace: "podinfo"
  skipPrePull: true

This time, we will create the necessary namespace in the same deployment item as the Helm Chart itself. This can be accomplished by adding a ./apps/podinfo/kustomization.yaml with the following content:

resources:
  - namespace.yaml
  - helm-rendered.yaml
  - ingress.yaml

Kluctl will respect the order of resources found in Kustomize deployments, and thus deploy the namespace before anything else. The helm-rendered.yaml is an auto-generated file containing the templated Helm Chart, this file however will never physically appear in your project.

To add an ingress, add ./apps/podinfo/ingress.yaml (which is also specified in the kustomization.yaml) with the following content:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: podinfo
  namespace: podinfo
spec:
  ingressClassName: nginx
  rules:
    - host: podinfo.127.0.0.1.nip.io
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: podinfo
                port:
                  number: 9898

The above ingress uses the previously deployed ingress-nginx controller with the class name nginx. We use nip.io to get a simple dummy domain name for the ingress. This domain will resolve to 127.0.0.1 and thus let you access the ingress through the extraPortMappings of the kind cluster .

Deploy podinfo

Now run kluctl deploy again and wait for it to finish. Also wait for all PODs to become ready in the cluster and then try to access podinfo by opening http://podinfo.127.0.0.1.nip.io:8080/ in your browser.

You should be greeted by this sweet little fellow :)

Tabula Rasa

Now let’s prove that the order of deployments was actually honored. Right now, you have added new stuff incrementally, so in theory the deployment could be in a state where it just worked by accident. We however want to make sure and prove that the whole deployment can be performed in one go.

Let’s delete the kind cluster, re-create it and then re-deploy:

$ kind delete cluster --name kluctl-tutorial
$ kind create cluster --name kluctl-tutorial --config kind-config.yaml
$ kluctl deploy

Things should be up and running again after a few minutes, without the need to retry the deployment.

Upgrade the ingress-nginx Helm Chart

If you look at ./base/ingress-nginx/helm-chart.yaml again, you’ll see that version 4.5.0 was used. This version is already outdated and should be upgraded. Simply change the version to 4.5.2 and then run kluctl deploy again.

When you do this, Kluctl will first perform a dry-run and then print a diff that shows you what would happen if you confirm the deployment. The changes seen in the diff should look like this:

$ kluctl deploy
✓ Loading kluctl project
...snip...
✓ Running server-side apply for all objects

Warnings:
  no discriminator configured. Orphan object detection will not work

Changed objects:
  ClusterRole/ingress-nginx
  ...snip...
  ingress-nginx/ServiceAccount/ingress-nginx

Diff for object ClusterRole/ingress-nginx
+----------------------------------------------+----------------------+
| Path                                         | Diff                 |
+----------------------------------------------+----------------------+
| metadata.labels["app.kubernetes.io/version"] | -1.6.3               |
|                                              | +1.6.4               |
+----------------------------------------------+----------------------+
| metadata.labels["helm.sh/chart"]             | -ingress-nginx-4.5.0 |
|                                              | +ingress-nginx-4.5.2 |
+----------------------------------------------+----------------------+

Diff for object ClusterRoleBinding/ingress-nginx
+----------------------------------------------+----------------------+
| Path                                         | Diff                 |
+----------------------------------------------+----------------------+
| metadata.labels["app.kubernetes.io/version"] | -1.6.3               |
|                                              | +1.6.4               |
+----------------------------------------------+----------------------+
| metadata.labels["helm.sh/chart"]             | -ingress-nginx-4.5.0 |
|                                              | +ingress-nginx-4.5.2 |
+----------------------------------------------+----------------------+
...snip...

? The diff succeeded, do you want to proceed? (y/N) 

If you confirm the deployment, it will apply the changes seen before.

Templating and Kluctl Targets

What you have seen so far is just showcasing how Kluctl deployment projects are structured and how Helm is integrated. This alone is already very powerful, as it allows you to structure large and complex deployments in a way that mirrors your intent, without too much complexity or boilerplate.

The next step would be to introduce templating and Kluctl targets . This will then allow you to make your deployment multi-environment and multi-cluster capable.

These topics will however require another post, as it would be too much to be covered in this post as well.

GitOps with Kluctl

This post has shown the “push” flow of Kluctl by simply calling Kluctl from your local machine. The same flow can be adapted to a pipeline based CI/CD system, e.g. Github Actions or Gitlab CI.

Kluctl also offers a pull based flow, implemented through the Kluctl controller (GitHub repository is located here ). This is comparable to how Flux does it, you deploy a KluctlDeployment custom resource and the controller handles reconciliation via periodic deployments.

Kluctl vs Flux/ArgoCD/Helmfile

As mentioned at the top of this post, there are multiple solutions available that can all be used to tackle the same set of problems. All of these do it with different approaches and I will try to compare these approaches with the way Kluctl does it.

Flux

Flux allows to define Kustomizations and HelmReleases , which are both Kubernetes Custom Resources . These Custom Resources are then handled by different Flux Controllers which perform the actual reconciliation.

This in turn means that you make yourself dependent on the Custom Resources being applied on the cluster and the controller processing them. This also means, you can’t easily try out things locally, as the only available command to do this is flux diff kustomization which is much harder to use then kluctl diff and unable to handle Helm Charts.

You will also never be able to mix GitOps with non-GitOps flows, e.g. “deploy from your laptop” for development environments and GitOps for “real” environments. Being able to mix these flows would allow you to move extremely fast while testing out and researching new stuff, without the need to constantly push to Git and then wait for the reconciliation to succeed/fail.

Dependency management in Flux is handled via dependsOn fields, which can however not be mixed with different resource types.

ArgoCD

I’m not very experienced with ArgoCD , so take my words with a grain of salt. But from everything I’ve seen so far, the same things I wrote about Flux also apply to ArgoCD. The biggest difference is that you have a UI that visualises changes and drift.

Dependency management is handled via Sync Waves , which allows you to give resources a priority which is then used to determine deployment order.

Helmfile

Helmfile is probably the tool that can be best compared to Kluctl. There are however some important differences. Helmfile concentrates on Helm as being the underlying building block for all deployments. This means, it will internally convert plain manifests and Kustomize deployments into Helm Charts and install them via the Helm CLI.

Kluctl on the other hand uses Kustomize as the base building block and instead converts Helm Charts into Kustomize deployments by simply rendering the Charts. This also means, that Kluctl will not use helm install but instead rely on its own server-side-apply based implementation, giving it much better conflict detection, conflict resolution and diffing capabilities.

Helmfile currently has no GitOps style controller available. From what I can find online, people tend to use helmfile template to simply generate plain manifests which they then commit and push to Git.

What next?

The project that we created in this post is only partially useful, as it is not ready for multi-environment and multi-cluster deployments. In an upcoming post, I will describe how to add targets and use templating to allow different flavors to be deployed. I will also describe how to properly split up projects, for example so that you have a cluster-base deployment with Cilium and ingress-nginx and multiple application specific deployments.

Introducing the Template Controller and building GitOps Preview Environments

This blog post serves two purposes. The first one is to announce and present the Template Controller ( Source ). The second purpose is to demonstrate it by setting up a simple GitOps based Kubernetes deployment with dynamic preview environments.

The Template Controller

The template-controller is a Kubernetes controller that is able to create arbitrary objects based on dynamic templates and arbitrary input objects. It is inspired by ArgoCD’s ApplicationSet , which is able to create dynamic ArgoCD Applications from a list of generators (e.g. Git) and an Application template.

The Template Controller uses a different approach, making it more flexible and independent of the GitOps system being used. It uses arbitrary Kubernetes objects as inputs and allows to create templated objects of any kind (e.g. a Flux Helm Release or a KluctlDeployment ). This makes the controller very extensible, as any type of input can be implemented with the help of additional controllers which are not necessarily part of the project.

When specifying the input objects, you’d also specify which part of the object to use as input. This is done by specifying a JSON Path that select the subfield of the object to use, e.g. status.result for a GitProjector or data for a ConfigMap.

The Template Controller implements this functionality through the ObjectTemplate CRD. As the name implies, it also uses a templating engine, which is identical to the one used in Kluctl , with the ObjectTemplate's input matrix available as global variables.

Preparation

To try the examples provided in this blog post, you’ll need to have a running cluster ready. You could for example use a local kind cluster:

$ kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.25.3) 🖼 
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Thanks for using kind! 😊

Then, you will need the Template Controller installed into this cluster:

$ helm repo add kluctl http://kluctl.github.io/charts
"kluctl" has been added to your repositories
$ helm install -n kluctl-system --create-namespace template-controller kluctl/template-controller
NAME: template-controller
LAST DEPLOYED: Wed Dec 28 17:24:47 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1

You will also need the Flux Kluctl Controller installed for the examples. If you decide to use plain Flux deployments, you will need to install Flux instead.

$ # we assume that you have the Helm repository installed already
$ helm install -n kluctl-system --create-namespace flux-kluctl-controller kluctl/flux-kluctl-controller 
NAME: flux-kluctl-controller
LAST DEPLOYED: Wed Dec 28 17:27:53 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1

A few words on security

The Template Controller can access and create any kind of Kubernetes resource. This makes it very powerful but also very dangerous. The Template Controller needs to run with a quite privileged service account but at the same time uses user impersonation to downgrade permissions to less privileged service accounts while processing ObjectTemplates.

By default, the controller will use the default service account of the namespace that the ObjectTemplate is deployed to. This will by default limit permissions to basically nothing, unless you explicitly bind roles to the default service account. A better way is to create a dedicated service account instead and bind limited roles with the required permissions to that dedicated service account. This service account can then be specified in the ObjectTemplate via spec.serviceAccountName.

The following RBAC rules can be used when going through the examples in this blog post, it is however suggested to not blindly reuse them in your real deployments. You should carefully asses which permissions are really needed and limit the roles appropriately. Also pay attention when using the cluster-admin role or any other ClusterRole, as it easily allows privilege escalation and at least allows to deploy into other namespaces than the ObjectTemplate's namespace.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: template-controller
  namespace: default
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: template-controller
  namespace: default
rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["*"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: template-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: template-controller
subjects:
  - kind: ServiceAccount
    name: template-controller
    namespace: default

Dynamic Preview Environments

The initial use case and the main reason the Template Controller was created was to implement preview environments in a GitOps setup. Preview environments are dynamic environments that are spinned up and down on demand, for example when a pull request introduces changes that need to be tested before these are merged into the main branch.

In a GitOps-based setup, one would need to create the relevant custom resources per preview environment, for example a Flux Kustomization , Flux HelmRelease or a or KluctlDeployment . The underlying GitOps controller would then take over and perform the actual deployment.

In the following examples we will concentrate on using KluctlDeployments. Changing it to use Kustomizations or HelmReleases should be self-explanatory, as the Template Controller treats all resource kinds equally.

There are multiple options on how to define the desired pre-conditions and configuration of a preview environment, which will be described in the next chapters.

Linking Preview Environments to Branches

You can, for example, link Git branches to preview environments, so that for each new branch a preview environment is created, with configuration being read from a yaml file inside the branch. This can be achieved by using a GitProjector , which will periodically clone the configured Git repository, scan for matching branches and files and project the result into the GitProjector status. The status is then available as matrix input inside the ObjectTemplate.

Example GitProjector:

apiVersion: templates.kluctl.io/v1alpha1
kind: GitProjector
metadata:
  name: preview
  namespace: default
spec:
  interval: 1m
  # You'll need to fork this repository and update the url to actually test what happens when you create new branches
  url: https://github.com/kluctl/kluctl-examples.git
  # In case you use a private repository:
  # secretRef:
  #   name: git-credentials
  ref:
    # let's only take preview- branches into account
    branch: preview-.*

The following ObjectTemplate can then use the GitProjetor's status.result field to create one KluctlDeployment per branch.

apiVersion: templates.kluctl.io/v1alpha1
kind: ObjectTemplate
metadata:
  name: preview
  namespace: default
spec:
  # this is the service account described at the top
  serviceAccountName: template-controller
  prune: true
  matrix:
    - name: git
      object:
        ref:
          apiVersion: templates.kluctl.io/v1alpha1
          kind: GitProjector
          name: preview
        jsonPath: status.result
        # status.result is a list of matching branches. Without expandLists being true, the matrix input would treat
        # that list as a single input, but we actually want the list items being one input per item
        expandLists: true
  templates:
    - object:
        apiVersion: flux.kluctl.io/v1alpha1
        kind: KluctlDeployment
        metadata:
          # We can use the branch name as basis for the object name
          name: "{{ matrix.git.ref.branch | slugify }}"
          namespace: default
        spec:
          interval: 1m
          deployInterval: "never"
          timeout: 5m
          source:
            url: https://github.com/kluctl/kluctl-examples.git
            path: simple-helm
            ref:
              # ensure we deploy from the correct branch
              branch: "{{ matrix.git.ref.branch }}"
            # secretRef:
            #  name: git-credentials
          target: "simple-helm"
          context: default
          args:
            # Look into the simple-helm example to figure out what the environment arg does. It basically sets the
            # namespace to use, but could theoretically do much more.
            environment: "{{ matrix.git.ref.branch | slugify }}"
          prune: true

Please note the use of templating variables and filters inside the actual template under spec.templates. Each template will be rendered once per matrix input, which in this case means once per branch. The templates can use the current matrix input in the form of a variable, accessible via matrix.<name>, e.g. matrix.git in this case. Please read the documentation of GitProjector to figure out what is available in matrix.git, which is basically just a copy of the individual status.result list items.

One Preview Environment per pull requests

Another option is to use pull requests instead of the underlying Git branches to create preview environments. This might be useful if you want to report the status of your preview environment to the pull request, e.g. by updating the commit status when the deployment turns green or red. One might also want to post complex status comments, for example the result of the deployment in the form of structured and beautiful diff.

To achieve this, you can use the ListGithubPullRequests or the ListGitlabMergeRequests custom resource, which are also provided by the Template Controller.

Consider the following example:

apiVersion: templates.kluctl.io/v1alpha1
kind: ListGithubPullRequests
metadata:
  name: preview
  namespace: default
spec:
  interval: 1m
  owner: kluctl
  repo: kluctl-examples
  state: open
  base: main
  head: kluctl:preview-.*
  # in case you forked the repo into a private repo
  #tokenRef:
  #  secretName: git-credentials
  #  key: github-token

The above example will regularly (1m interval) query the GitHub API for pull requests inside the kluctl-examples repository. It will filter for pull requests which are in the state “open” and are targeted against the “main” branch. The result of the query is then stored in the status.pullRequests field of the custom resource. The content of the pullRequests field basically matches what GitHub would return via the Pulls API (with some fields omitted to reduce the size).

You can now use the result in an ObjectTemplate:

apiVersion: templates.kluctl.io/v1alpha1
kind: ObjectTemplate
metadata:
  name: pr-preview
  namespace: default
spec:
  # this is the service account described at the top
  serviceAccountName: template-controller
  prune: true
  matrix:
    - name: pr
      object:
        ref:
          apiVersion: templates.kluctl.io/v1alpha1
          kind: ListGithubPullRequests
          name: preview
        jsonPath: status.pullRequests
        # status.result is a list of matching branches. Without expandLists being true, the matrix input would treat
        # that list as a single input, but we actually want the list items being one input per item
        expandLists: true
  templates:
    - object:
        apiVersion: flux.kluctl.io/v1alpha1
        kind: KluctlDeployment
        metadata:
          # We can use the head branch name as basis for the object name
          name: pr-{{ matrix.pr.head.ref | slugify }}
          namespace: default
        spec:
          interval: 1m
          deployInterval: "never"
          timeout: 5m
          source:
            url: https://github.com/kluctl/kluctl-examples.git
            path: simple-helm
            ref:
              # ensure we deploy from the correct branch
              branch: "{{ matrix.pr.head.ref }}"
            # secretRef:
            #  name: git-credentials
          target: "simple-helm"
          context: default
          args:
            # Look into the simple-helm example to figure out what the environment arg does. It basically sets the
            # namespace to use, but could theoretically do much more.
            environment: "pr-{{ matrix.pr.head.ref | slugify }}"
          prune: true

Using Flux instead of Kluctl

The above examples can easily be changed to work with Flux resources, e.g. Kustomization and HelmRelease. Simply replace the template KluctlDeployment with the appropriate resources and use the same template variables where needed.

Shutting down preview environments

All the above examples will result in pruning the whole deployment when the branch gets deleted. This works because after deletion of the branch, the GitProjector will not contain that branch in the status.result after the next reconciliation. This will lead to the ObjectTemplate not having the matrix input anymore, causing a prune of the KluctlDeployment object, which in turn causes deletion of all resources deployed by the Kluctl controller.

For the pull request based examples, it will also prune the deployments when the pull request gets closed or merged, simply because the ListGithubPullRequests filters for open pull requests and thus will not list the pull requests anymore after they have been closed/merged.

Another use case: Transformation of objects

The following is a very simple example of an ObjectTemplate that uses a Secret as input and simply transforms it into another secret.

This can turn out to be quite useful if you need to re-use the same secret value multiple times but in different forms. For example, if you have a secret that stores database credentials, you might also need the same username and password inside a JDBC url. Typically, you’d have to store the secret twice in both required forms.

This is however not an option if the secret is generated after the deployment, e.g. by the Zalando Postgres operator or by the AWS RDS operator. Using the Mittwald Secret Generator is also something that I can recommend, as it removes the need to pre-generate secrets. With the ObjectTemplate based transformation, the generated secrets can then be used in many new ways.

Consider the following input Secret:

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: input-secret
  namespace: default
stringData:
  password: my-user
  username: secret-password

And the following ObjectTemplate:

apiVersion: templates.kluctl.io/v1alpha1
kind: ObjectTemplate
metadata:
  name: transformer-template
  namespace: default
spec:
  serviceAccountName: template-controller
  prune: true
  matrix:
    - name: secret
      object:
        ref:
          apiVersion: v1
          kind: Secret
          name: input-secret
  templates:
  - object:
      apiVersion: v1
      kind: Secret
      metadata:
        name: "transformed-secret"
      stringData:
        jdbc_url: "jdbc:postgresql://db-host:5432/db?user={{ matrix.secret.data.username | b64decode }}&password={{ matrix.secret.data.password | b64decode }}"
        # sometimes the key names inside a secret are not what another component requires, so we can simply use different names if we want
        username_with_different_key: "{{ matrix.secret.data.username | b64decode }}"
        password_with_different_key: "{{ matrix.secret.data.password | b64decode }}"

I hope the example above is self-explanatory. It simply transforms one secret into another, which is then in a form that can be consumed by a Java application for example that can only work with JDBC urls.

What’s next?

I believe that there are much more use cases for the Template Controller, and I’m absolutely convinced that the community will invent completely new ones and maybe share them by posting examples and ideas. Due to the flexible nature of the matrix inputs and template definition, a lot is possible. Think big! 😸

The Template Controller currently comes with a few additional custom resources (e.g. GitProjector and ListGithubPullRequests) that are meant to be used as matrix inputs. I can imagine that other custom resources might be candidates to be included in the controller as well, and I’m also open for ideas.

Templating in Kustomize with Kluctl style deploy and diff

Kustomize is currently one of the most used tools to organise Kubernetes manifests and the resulting deployments. As of the Kustomize website, “Kustomize introduces a template-free way to customize application configuration that simplifies the use of off-the-shelf applications.”

This says it very clear: Kustomize is template-free. The reasoning why Kustomize does not leverage templating and also will never do so is very reasonable and easy to understand. It tries to avoid the potential overload and complexity that comes with templating.

In my opinion however, the feared complexity is only a problem if one tries to suit the needs of everyone. This is an issue that popular Helm Charts do have for example. If you try to make everyone happy, your templates must make everything configurable, which eventually leads to Kubernetes manifests having more templating code than the actual YAML.

Kustomize in that regard, has the advantage that the re-used manifests do not have to take external customization into account. You, as the “customizer”, can decide what needs to be configurable and can achieve this with overlays and patches.

Looking at Kustomize from that perspective, it is of course very reasonable to keep it fully template-free. There is however also another perspective that you might want to consider.

What if I don’t care about the needs of others?

That sounds a bit selfish, doesn’t it? :) But let’s be honest to our self, many times you just need to create a deployment that suits your own needs right now. This deployment might re-use other components which it needs to customize, but it doesn’t need to be customized by someone else.

Thus, the level of customization that you need to implement is minimal. Maybe you just want to allow changing the target namespace or some replica count depending on the target environment. Maybe you want to have some components enabled in one environment and other components disabled in other environments. If you think more about it, you might also realise that it’s not about “customization” anymore but actually about “configuration”.

In that case, templating does not bring the risks that come with components that are meant to be re-used and customized. This is because it is very clear what level of configuration is required and thus the use of templating can be reduced to exactly that. In the end, you’ll only need a few places with something like {{ my_service.replicas }} and maybe some conditional blocks with {% if my_service.enabled %}...{% endif %}.

On the other hand, using plain Kustomize to achieve the same result can easily become unnecessarily complex. For example, the use of bases, overlays and patches tends to create project structures that are hard to grasp when they grow. It can also force you to change your project structure in “unnatural” (at least that is how it feels for me) ways, because you have to adapt to the way overlays work. Templating would allow much simpler solutions in the above case.

But…Kustomize doesn’t support templating, right?

Bringing templating to Kustomize

Kluctl builds a large set of its features and promised advantages on top of templating . The Kustomize integration also allows templating in all involved resources, including the kustomization.yaml itself and all referenced manifests.

Configuration can be provided in different ways:

1. Via CLI arguments , e.g. --arg or --args-from-file.
2. Via Targets , meaning that you can define named targets with fixed args.
3. Via vars in Kluctl deployments.
4. Via Environment variables (through KLUCTL_ARG_XXX).

In this blog post, we’ll focus on the first option for simplicity. The second and third options are much more powerful, but require more boilerplate to set up a Kluctl project and Kluctl deployments . The first option also works with plain Kustomize deployments, which is what I’m going to demonstrate.

Whatever option is used, all “args” are then available in every place by simply using Jinja2 variable expressions , e.g. {{ args.my_arg }}.

A simple example

We will use the podtato-head project, specifically the Kustomize delivery scenario as an example . But first, we’ll need a test cluster. I suggest to simply use Kind and create a fresh cluster:

$ kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.25.3) 🖼 
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Thanks for using kind! 😊

Ensure that you are on the correct Kubernetes context by calling kubectl config current-context and verify that it points to kind-kind. Without using Kluctl Targets with fixed contexts, Kluctl behaves the same as any other tool in the Kubernetes space when it comes to the current context, meaning that you have to watch out to not deploy to prod by accident! :)

Then, clone the example project and enter the delivery directory:

$ git clone https://github.com/podtato-head/podtato-head.git
$ cd podtato-head/delivery/kustomize/base
$ ls -lah
total 120
drwxr-xr-x  17 user  wheel   544B Nov 16 16:28 .
drwxr-xr-x   6 user  wheel   192B Nov 16 16:28 ..
-rw-r--r--   1 user  wheel   558B Nov 16 16:28 configmap-discovery.yaml
-rw-r--r--   1 user  wheel   1.5K Nov 16 16:28 deployment-entry.yaml
-rw-r--r--   1 user  wheel   1.1K Nov 16 16:28 deployment-hat.yaml
-rw-r--r--   1 user  wheel   1.1K Nov 16 16:28 deployment-left-arm.yaml
-rw-r--r--   1 user  wheel   1.1K Nov 16 16:28 deployment-left-leg.yaml
-rw-r--r--   1 user  wheel   1.1K Nov 16 16:28 deployment-right-arm.yaml
-rw-r--r--   1 user  wheel   1.1K Nov 16 16:28 deployment-right-leg.yaml
-rw-r--r--   1 user  wheel   474B Nov 16 16:28 kustomization.yaml
-rw-r--r--   1 user  wheel   447B Nov 16 16:28 service-entry.yaml
-rw-r--r--   1 user  wheel   438B Nov 16 16:28 service-hat.yaml
-rw-r--r--   1 user  wheel   454B Nov 16 16:28 service-left-arm.yaml
-rw-r--r--   1 user  wheel   453B Nov 16 16:28 service-left-leg.yaml
-rw-r--r--   1 user  wheel   456B Nov 16 16:28 service-right-arm.yaml
-rw-r--r--   1 user  wheel   456B Nov 16 16:28 service-right-leg.yaml
-rw-r--r--   1 user  wheel   281B Nov 16 16:28 serviceaccount.yaml

As you can see, this is a simple Kustomize deployment, not using any bases or overlays. Let’s start using Kluctl by doing a vanilla deployment first:

$ kluctl deploy
✓ Loading kluctl project
✓ Initializing k8s client
✓ Rendering templates
✓ Rendering Helm Charts
✓ Building kustomize objects
✓ Postprocessing objects
✓ Getting namespaces
✓ .: Applied 14 objects.
✓ Running server-side apply for all objects

New objects:
  default/ConfigMap/podtato-head-service-discovery
  default/Deployment/podtato-head-entry
  default/Deployment/podtato-head-hat
  default/Deployment/podtato-head-left-arm
  default/Deployment/podtato-head-left-leg
  default/Deployment/podtato-head-right-arm
  default/Deployment/podtato-head-right-leg
  default/Service/podtato-head-entry
  default/Service/podtato-head-hat
  default/Service/podtato-head-left-arm
  default/Service/podtato-head-left-leg
  default/Service/podtato-head-right-arm
  default/Service/podtato-head-right-leg
  default/ServiceAccount/podtato-head
? The diff succeeded, do you want to proceed? (y/N) y
✓ .: Applied 14 objects.
✓ Running server-side apply for all objects

New objects:
  default/ConfigMap/podtato-head-service-discovery
  default/Deployment/podtato-head-entry
  default/Deployment/podtato-head-hat
  default/Deployment/podtato-head-left-arm
  default/Deployment/podtato-head-left-leg
  default/Deployment/podtato-head-right-arm
  default/Deployment/podtato-head-right-leg
  default/Service/podtato-head-entry
  default/Service/podtato-head-hat
  default/Service/podtato-head-left-arm
  default/Service/podtato-head-left-leg
  default/Service/podtato-head-right-arm
  default/Service/podtato-head-right-leg
  default/ServiceAccount/podtato-head

kluctl shows you the diff it will apply, asks for confirmation, applies the changes and then shows you the applied result. Verify that it got deployed:

$ kubect get pod
NAME                                      READY   STATUS    RESTARTS   AGE
podtato-head-entry-7dfd8cdd6d-6mtxd       1/1     Running   0          95s
podtato-head-hat-6bcbf5f957-mfc6r         1/1     Running   0          95s
podtato-head-left-arm-7d9db78544-689tx    1/1     Running   0          95s
podtato-head-left-leg-59f45ffc4-grcjc     1/1     Running   0          95s
podtato-head-right-arm-5444b48b85-427w7   1/1     Running   0          95s
podtato-head-right-leg-f68df999f-g27nz    1/1     Running   0          95s

There are multiple things that you might have noticed already:

1. Kustomize does not handle deployments, it just “builds” them and then let’s you handle the actual deployment via kubectl apply -f. Kluctl also handles the actual deployment for you. The advantages of this will be clear in a few minutes.
2. Kluctl showed a diff (very simple one in this case, just new objects), then asked for confirmation and then showed what it did (identical to what the diff showed). The power of the diff feature will become much clearer in a few minutes.

Introducing some templating

Now let’s introduce some templating into the example deployment. For example, let’s edit deployment-entry.yaml and change the replicas field to:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: podtato-head-entry
...
spec:
  replicas: {{ args.entry_replicas }}
...

If you’d try to re-deploy this now, it will throw an error about args.entry_replicas being undefined. So let’s call Kluctl with the now required arg:

$ kluctl deploy -a entry_replicas=3
...

Changed objects:
  default/Deployment/podtato-head-entry

Diff for object default/Deployment/podtato-head-entry
+---------------+--------------------------------------------------------------+
| Path          | Diff                                                         |
+---------------+--------------------------------------------------------------+
| spec.replicas | -1                                                           |
|               | +3                                                           |
+---------------+--------------------------------------------------------------+
? The diff succeeded, do you want to proceed? (y/N)

You’ll notice that Kluctl again stops and asks for confirmation. But this time, it will actually show you some meaningful diff. It allows you to verify that Kluctl will apply the intended changes. The diff that you see is NOT a simple file based diff, but a diff after performing a full-blown server-side apply in dry-run mode. This means, what you see is what you’ll get, no surprises in-between!

Let’s confirm with y:

? The diff succeeded, do you want to proceed? (y/N) y
✓ .: Applied 14 objects.
✓ Running server-side apply for all objects

Changed objects:
  default/Deployment/podtato-head-entry

Diff for object default/Deployment/podtato-head-entry
+---------------+--------------------------------------------------------------+
| Path          | Diff                                                         |
+---------------+--------------------------------------------------------------+
| spec.replicas | -1                                                           |
|               | +3                                                           |
+---------------+--------------------------------------------------------------+

The actual deployment is performed and the result printed to the user. The result should always be identical to the diff you saw before.

Let’s make something conditional

Now let’s make the “hat” of the podtate-head optional. However, as you have previously deployed the project already, you’ll need to delete the hat deployment manually:

$ kubectl delete deployment.apps/podtato-head-hat service/podtato-head-hat

Edit kustomization.yaml and put an if/endif around the hat resource:

...
resources:
- configmap-discovery.yaml
{% if args.hat_enabled | default(true) %}
- deployment-hat.yaml
- service-hat.yaml # also remove the original entry from the bottom of the file
{% endif %}
- deployment-left-arm.yaml
...

You can now deploy with Kluctl while having the hat disabled:

$ kluctl deploy -a entry_replicas=3 -a hat_enabled=false

If you deploy with a hat and then with hat_enabled=false, Kluctl will not delete/prune the previously deployed hat. If you want to have pruning support, you must create a Kluctl deployment with commonLabels enabled, so that Kluctl knows how to identify related objects.

Using vars files instead of arguments

args can also be passed via vars files, which are arbitrary structured YAML files. This is a comparable to helm value files. Consider the following examples.

test-args.yaml:

entry_replicas: 2
hat_enabled: false

prod-args.yaml:

entry_replicas: 3

These values can be used with --args-from-file:

$ kluctl diff --args-from-file=test-args.yaml
$ kluctl diff --args-from-file=prod-args.yaml

Based on that, you can easily implement multi-environment deployments. This is however still a poor-mans solution to multi-environment deployments, with the use of Kluctl projects being the better solution. Said Kluctl projects allow you to define named targets which are fixed in their configuration, so that you only have to invoke kluctl deploy -t test without needing to know what the internal details are.

What’s next?

This article has shown how Kluctl can be used on simple/plain Kustomize deployments. The next thing you should consider is using Kluctl projects and Kluctl deployments around your Kustomize deployments. It will allow you to have much more flexible and powerful ways of configuration management. It will also allow you to use the GitOps style flux-kluctl-controller .

Upcoming blog posts will show why the Helm Integration is a good thing with many advantages and also describe why one would choose Kluctl over plain Helm.

Multiple Environments with Flux and Kluctl

Most projects that have server-side components usually need to be deployed multiple times, at least if you don’t want to break things for your users all the time. This means, that there is not only one single “prod” environment, but also something like “test” or “staging” environments. So usually, there is a minimum of two different environments.

Each of these environments has a defined role to play. The “prod” environment is obviously the environment that is used in production. A “test” or “staging” environment is usually the one where new versions are deployed so that they can be tested before being promoted to “prod”. A “dev” environment might be the one where even high-risk deployments are allowed where breaking everything is daily business. You could even have multiple dev environments, e.g. one per developer.

In the Kubernetes world, it is perfectly viable and good practice to run multiple environments on a single cluster. Depending on the requirements of your project, you might want to separate prod from non-prod environments, but you could still have “prod” and “staging” on the same cluster while “test”, “dev”, “uat” exist on a non-prod cluster.

Environment Parity

If you want to make this work in a way that does not cause too much pain, you must practice environment parity as much as possible. The best way to achieve this is to have full automation and everything as code. The same “code” that was used to deploy “prod” should also be used to deploy “staging”, “test” and all other environments. The only difference in the deployments should be a defined set of configurations.

Such configurations can for example be the target cluster and/or namespace, the resource allocations (e.g. 1 replica in “dev”, 3 replicas in “prod”), external systems (e.g. databases) and ingress configuration (e.g. DNS and certificates). Configuration can also enable/disable conditional parts of the deployment, for example to disable advanced monitoring on “dev” environments and enable mocking services as replacements for real external systems.

Tooling

There are multiple tools available that allow you to implement a multi-env/multi-cluster deployment that is completely automated and completely “as code”. Helm and Kustomize are currently the first tools that will pop up when you try to look for such tools. As written in my previous blog post , I believe that these tools are the best option for the things that they do very good, but a sub-optimal choice when it comes to configuration management.

Kluctl is the preferred solution for me right now. Not only because I built it, but also because so far I did not find a solution that is as easy to learn and use and so flexible at the same time.

Fully working multi-env example

I suggest to open the microservices demo in another tab and look into it at least briefly (especially the third part). I will from now on pick stuff from this tutorial as examples in this blog post.

Targets in Kluctl

Kluctl works with the concept of “targets”. A target is a named configuration that acts as the entry point for every further configuration required for your environment. As an example, look at the targets from .kluctl.yaml of the microservices demo:

targets:
  - name: local
    context: kind-kind
    args:
      env_type: local
  - name: test
    context: kind-kind
    args:
      env_type: real
  - name: prod
    context: kind-kind
    args:
      env_type: real

Based on these, the same deployment can be configured differently depending on the target, or actually the args passed via the target. In the microservices demo, env_type (name can be chosen by you) is used to include further configuration inside the deployment.yaml:

...
vars:
  - file: ./vars/{{ args.env_type }}.yml
...

At the same time, Kluctl makes the target configuration itself available to the deployment, making things like this possible:

apiVersion: v1
kind: Namespace
metadata:
  name: ms-demo-{{ target.name }}

You can also conditionally enable/disable parts of your deployment depending on the configuration loaded via the vars block above:

deployments:
  - path: adservice
  - path: cartservice
  - path: checkoutservice
  - path: currencyservice
  {% if services.emailservice.enabled %}
  - path: emailservice
  {% endif %}
  - path: frontend
  {% if services.loadgenerator.enabled %}
  - path: loadgenerator
  {% endif %}
  - path: paymentservice
  - path: productcatalogservice
  - path: recommendationservice
  - path: shippingservice

I hope the above snippets give you a feeling about how multi-env deplyoments can be solved via Kluctl. As already mentioned, I suggest to read through the microservices demo tutorial to get an even better understanding. The first two parts will describe some Kluctl basics while the third part enters multi-env deployments.

GitOps and Flux

Kluctl is designed in a way that allows seamless co-existence of CLI based workflows, classical CI/CD and GitOps. This means, that even if you decide to perform prod deployments only via GitOps, you can still perform exactly the same deployment to other environments through your CLI. All this without any struggle (you really only need access to the cluster) and 100% compatible to how the same deployment would be performed via GitOps or CI/CD.

This allows you to adapt your workflow depending on what your current goal is. For example, a developer testing out risky and bleeding-edge changes in his personal dev environment can deploy from his local machine, avoiding the painful “modify -> push -> wait -> error -> repeat” cycles seen too often in pure GitOps and CI/CD setups. When the developer is done with the changes, GitOps can take over on the another (e.g. “test” and later “prod”) environment.

Even for “prod”, which in the above scenario is GitOps managed, can benefit from the possibility to run Kluctl from your local machine. Running a “kluctl diff -t prod” before promoting to “prod” can prevent some scary surprises.

Kluctl implements GitOps via the flux-kluctl-controller . It allows to create KluctlDeployment objects which refer to your Kluctl project (which relies in Git) and the target to be deployed.

Installing flux-kluctl-controller

Before being able to create KluctlDeployment objects, the flux-kluctl-controller needs to be installed first. Please navigate to the installation documentation and follow the instructions found there.

Microservices Demo and Flux

Deploying the microservices demo via Flux is quite easy. First, we’ll need a GitRepository object that refers to the Kluctl project:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: microservices-demo
spec:
  interval: 1m
  url: https://github.com/kluctl/kluctl-examples.git
  ref:
    branch: main

This will cause Flux to pull the source whenever it changes. A KluctlDeployment can then refer to the GitRepository:

apiVersion: flux.kluctl.io/v1alpha1
kind: KluctlDeployment
metadata:
  name: microservices-demo-test
spec:
  interval: 5m
  path: "./microservices-demo/3-templating-and-multi-env/"
  sourceRef:
    kind: GitRepository
    name: microservices-demo
  timeout: 2m
  target: test
  prune: true
  # kluctl targets specify the expected context name, which does not necessarily match the context name
  # found while it is deployed via the controller. This means we must pass a kubeconfig to kluctl that has the
  # context renamed to the one that it expects.
  renameContexts:
    - oldContext: default
      newContext: kind-kind

The above example will cause the controller to deploy the “test” target/environment to the same cluster where the controller runs on. The same deployment can also be deployed to “prod” with a slightly different KluctlDeployment:

apiVersion: flux.kluctl.io/v1alpha1
kind: KluctlDeployment
metadata:
  name: microservices-demo-prod
spec:
  interval: 5m
  path: "./microservices-demo/3-templating-and-multi-env/"
  sourceRef:
    kind: GitRepository
    name: microservices-demo
  timeout: 2m
  target: prod
  prune: true
  # kluctl targets specify the expected context name, which does not necessarily match the context name
  # found while it is deployed via the controller. This means we must pass a kubeconfig to kluctl that has the
  # context renamed to the one that it expects.
  renameContexts:
    - oldContext: default
      newContext: kind-kind

Multiple Clusters

To make things easy for now, the above examples stick with a single cluster. The microservices demo project is deploying all targets to different namespaces already, so this is enough to showcase the Flux support. To make it work with multiple clusters, simply install the controller on another cluster and create the appropriate KluctlDeployment objects per cluster.

As an alternative, you can have a central Flux (+flux-kluctl-controller) installation that deploys to multiple clusters. This can be achieved with the help of the spec.kubeconfig and spec.serviceAccountName field of the KluctlDeployment object.

Also, as the examples stem from the microservices demo , they use the kind-kind context names. In a more realistic setup, you would use the real cluster/context names here. This also assumes that all developers will then use the same context names to refer to the same clusters. If this is honored, you gain 100% compatibility between the GitOps based deployments and CLI based deployments.

What’s next?

The flux-kluctl-controller and Kluctl itself already support dynamic feature/review environments, meaning that the controller can create new targets/deployments dynamically. The next blog article will go into the details of these feature/review environments.

Rethinking Kubernetes Configuration Management

One of the big advantages of Kubernetes is its declarative nature when it comes to deployed resources. You define what state you expect, apply it to Kubernetes and then let it figure out how to get to that state.

At first, this sounds like a big win, which it actually is. It however doesn’t take long for beginners to realize that simply writing yamls and applying them via “kubectl apply” is not enough.

The beginner will realize pretty early that configuration between different manifests must be synchronized, e.g. a port defined in a Service must match the port used when accessing the service from another Deployment. Hardcoding these configuration values will work till some degree, but get messy when your deployment projects get larger.

It will get completely unmanageable when the same resources need to get deployed multiple times, e.g. for multiple environments (local, test, uat, prod, …) and/or clusters. Copying and modifying all resources is probably the worst solution one could come up with at this stage.

A poor-man’s solution that works a little bit better is to introduce glue bash code, that performs substitutions and decides which manifests to actually apply. As with most bash scripting, this tends to start with “just a few lines” and ends up with unmaintainable spaghetti code.

Spoiler: This blog post is meant to introduce and promote Kluctl, which is a new tool and approach to configuration management in Kubernetes. Skip the next sections about existing tools if you don’t want to read about stuff you probably already know well enough.

Multiple tools to the rescue

Luckily, there are multiple solutions available that try to solve the described situation from above. Each of these tools has a different approach with its pros and cons. Many of the pros/cons are a matter of personal preference, e.g. text based templating is preferred by some and avoided by others.

Based on the last sentence, I’ll try to list a few tools that I know of and what my personal opinion/preference/experience with these is.

Kustomize

Kustomize advertises itself as a “template-free” solution for configuration management. Instead of templating, it relies on “bases and overlays” for composition and configuration of different environments.

Overlays include bases while performing some basic modifications to these (e.g. add a name prefix). More complex modifications to bases require the use of strategic or json6902 patches.

This approach usually starts clean and is easy to understand and maintain in the beginning. It can however easily get complicated and messy due to the unnatural project structure. It can become hard to follow the manifests and how/where modifications/patches got introduced.

The biggest pro point for Kustomize is its ease of use and the flat learning curve. There is nearly no need to learn anything new, as everything is plain YAML and looks very familiar if you already know how to write and read Kubernetes manifests. Kustomize also offers some very convenient tools, like secret and configmap generators. Kustomize is also integrated into kubectl and thus does not need any extra installation.

An interesting and revealing observation I made is that even though Kustomize does not offer templating on its own, other tools that leverage Kustomize (e.g. Flux or ArgoCD) offer features like substitutions to fulfill a recurring need of their users, loosening the “template-free” nature of Kustomize and signaling a clear demand for templating with Kustomize.

Helm

The first thing you see on helm.sh is: “The package manager for Kubernetes”. This already reveals what it’s very good for: Package management. If you want to pick a ready-to-use, pre-packaged and well-configurable application from a huge library of “Charts”, simply go to hub.helm.sh and look for what you need.

Installation is as simple as two command line calls, “helm repo add …” and “helm install …”. Other commands then allow you to manage the installation, e.g. upgrade or delete it.

What it’s not good at is composition and configuration management for your infrastructure and applications. Helm Charts require quite some boilerplate templating if you want to get to the flexibility that is offered by many of the publicly available charts. At the same time, you usually don’t need that flexibility and end up with templates that only have very few configuration options.

If you still decide to use Helm for composition and configuration management, you might end up with all the overhead of Helm without any of the advantages of it. For example, you would have to provide and maintain a Helm repository and actually perform releases into it, even if such release management doesn’t make sense for you.

Composition of Helm Charts requires the use of “Umbrella Charts”. These are charts that don’t by itself define resources but instead refer to multiple dependent charts. Configuration of these sub-charts is passed down from the umbrella chart. The umbrella chart, as all other charts, will also require you to perform release management and maintenance of a repository.

In my personal opinion, umbrella charts do not feel “natural”. It feels like a solution that only exists because it is technically possible to do it this way.

Helm and Kustomize can not be used together without some outer glue, e.g. scripting or special tooling. Many solutions that glue together Helm and Kustomize will however lose some crucial Helm features in-between, e.g. Helm Hooks and release management.

Using both tools in combination can still be very powerful. For example, it allows you to customize existing public Helm Charts in ways not offered by the chart maintainers. For example, if a chart does not offer configuration to add nodeSelectors, you could use a Kustomize patch to add the desired modifications to the resources in question.

(Kustomize has some experimental Helm integration, but the Kustomize developers strongly discourage use of it in production).

Flux CD

Flux allows you to implement GitOps style continuous delivery of your infrastructure and applications. It internally heavily relies on Kustomize and Helm to provide the actual resources.

Flux can also be used for composition and configuration management. It is for example possible to have a single Kustomization deployment that actually glues together multiple other Kustomizations and HelmReleases.

Flux also “enhances” Kustomize and Helm by allowing different kinds of preprocessing operations, e.g. substitutions for Kustomize and patches for Helm Charts. These enhancements are what make configuration management much easier with Flux.

The biggest advantage of Flux (and GitOps in general) is however also one of its biggest disadvantages, at least in my humble opinion. You become completely dependent on the Flux infrastructure, even for testing of simple changes to your deployments. I assume every DevOps engineer knows the pain of recurring “modify -> push -> wait -> error -> repeat” cycles. Such trial and error sessions can become quite time-consuming and frustrating.

It would be much easier to test out your changes locally (e.g. via kind or minikube) and push to Git only when you feel confident enough that your changes are ready. This is not easily possible to do in a reliable way with Flux based configuration management. Even a dedicated environment/playground stays painful as it does not remove the described cycle.

Plain Kustomize and Helm would allow you to run the tools from your local machine, targeted against a local cluster or remote playground environment, but this would require you to replicate some of the features offered by Flux so that your testing stays compatible with what is later deployed via Flux.

Argo CD

Argo CD is another GitOps implementation that is similar to Flux. The most visible difference is that it also offers some very good UI to visualize your deployments and the state of these. I do not have enough experience with ArgoCD to go into detail, but I assume that many of the things I wrote about Flux also apply for ArgoCD.

Rethinking Kubernetes Configuration Management

I propose to rethink how the available tooling is used in daily business. Basically, I suggest that the previously described tools keep doing the parts that they are best at and introduce new tooling to solve the configuration management problem.

Basically, we need a tool that acts as “glue” between Kustomize, Helm and GitOps. Something that allows to perform declarative composition of self-written and third-party Kustomize deployments, third-party Helm Charts and a powerful but easy to learn mechanism surrounding it to synchronize configuration between all components.

It would get even better if a unified CLI would manage all deployments the exact same way, no matter how large or complex. A CLI that does not interfere with GitOps style continues delivery and allows friendly co-existence.

Introducing Kluctl

My proposed solution for all this is Kluctl. It’s “the missing glue” (hence the name, glue with k and without e). Kluctl was born out of the need to create a unified solution that removes the need for all other glue around Kubernetes deployments. Development of it happened based on large real-life production environments inside a corporate environment with practicability being one of the highest priorities.

It provides a powerful but easy to learn project structure and templating system that feels natural when used daily. The project structure follows a simple inclusion hierarchy and gives you full freedom on how to organize your project.

The templating engine allows you to load configuration from multiple sources (usually arbitrary structured yaml files) and use the configuration in all components of your project.

The project is then deployed to “targets”, which can represent an environment and/or cluster. Multiple targets can exist on the same cluster or on different clusters, it’s all up to you. The “target” acts as the entry point for your configuration, which then implicitly defines what a target means.

The Kluctl CLI allows to diff, deploy, prune and delete targets, all via a unified CLI. If you remember how to do a “deploy”, you will easily figure out how to diff or prune a target. If you remember how to do it for project X, you will also know how to do it for project Y. It’s always the same from the CLI perspective, no matter how simple or complex a deployment actually is.

Kluctl will perform a dry-run and show a diff before a deployment is actually performed. Only if you agree with what would be changed, the changes are actually applied. After the deployment, Kluctl will show another diff that visualizes what has been changed.

This means, you always know what will happen and what has happened. It allows you to trust your deployments, even if you somehow lost track about the state of a target environment.

In addition, Kluctl works according to the premise: Everything can, nothing must. That’s why it works equally well for small and large deployment projects compared to the overhead some other tools impose.

Flux integration is possible via the flux-kluctl-controller and allows friendly co-existence of classical DevOps and GitOps. This means you can deploy to a playground environment from your local machine and let GitOps handle all other deployments, adhering to the GitOps principles.

Learning Kluctl

If you want to learn about Kluctl, go to kluctl.io and read the documentation. Best is actually to start with the Microservices Demo Tutorial, as it tries to introduce you to the Kluctl concepts step by step.

After you’ve finished the tutorial, you should be able to understand how projects are structured, how Kustomize and Helm is integrated and how targets are used to implement multi-env and multi-cluster deployments.

Based on that, you should be able to imagine how Kluctl could play a role in your own projects and maybe you decide to give it a try.

More tutorials, documentation and blog posts will follow soon.

Kluctl and Flux

We’re very happy to announce that Kluctl can from now on be used together with Flux . This will allow you to combine the workflows and features advertised by Kluctl with GitOps style continuous delivery.

GitOps vs Kluctl

One of the first questions that we usually get when introducing Kluctl to someone is something like: “Why not GitOps?” or “Why not Flux?”. There seems to be a common misunderstanding that arises in many people when trying to understand Kluctl on first sight, which is to believe that Kluctl is an alternative or competitor to GitOps and Flux (or even ArgoCD).

This is not the case. If one wants to compare Kluctl with something else, then it’s more appropriate to compare it to Helm, Kustomize or Helmfile. It should be clear that Kustomize for example is not an alternative/competitor for Flux, but instead an essential tool and building block to make it work.

Kluctl can be looked at from the same perspective when it comes to Flux. Flux implements Helm and Kustomize support via different controllers, namely the kustomize-controller and the helm-controller . With Kluctl, the same style of controller can be implemented and integrated into the Flux ecosystem.

Introducing the Kluctl Flux Controller

An alpha version of the Kluctl Flux Controller has just been released. It allows to create KluctlDeployment objects which are reconciled in a similar fashion as Kustomizations .

Each KluctlDeployment specifies a source object (e.g. a GitRepository ), the target to be deployed and some information on how to handle kubeconfigs. The controller then regularly reconciles the deployment, meaning that it will invoke kluctl deploy whenever a change is detected in the deployment.

Sounds great? Then take a look at this very simple example

Kustomize/Helm vs Kluctl

If you’ve already read through the Kluctl documentation , you’ve probably noticed that Kluctl internally uses Kustomize and Helm extensively.

This might raise the question: Why not use plain Kustomize and/or Helm if Flux is already involved? Good question!. Lets take a look:

Kluctl Projects/Deployments

If you prefer the way Kluctl organizes and structures projects and deployments, then using the Flux Kluctl Controller is obviously the best choice. Kluctl allows you to easily glue together what belongs together. If for example, a redis database is required to make your application work, you can manage the redis Helm Release and your application in the same deployment, including the necessary configuration to let them talk to each other.

To see how different a Kluctl deployment is compared to classic Kustomize/Helm + Flux, you can compare the flux2-kustomize-helm-example and the Kluctl Microservices Demo ( here is tutorial for the demo).

Native multi-env support

Kluctl allows you to natively create deployment projects that can be deployed multiple times to different environments/targets. You can for example have one target that is solely meant for local (e.g. Kind based) deployments, one that targets the test environment and one for prod. You can then use templating to influence deployments in whatever way you like. For example, you could change the local target to set all replicas to 1 and skip resource hungry support applications (e.g. monitoring infrastructure).

This is possible in plain Kustomize as well, but requires you to solve it without the concept of targets and without templating. In Kustomize, multi-env deployments must be solved with overlays , which does not necessary align with how you prefer your project structure.

Mix DevOps and GitOps

The core idea of GitOps is that Git becomes the single source of truth for the desired cluster state. This is something that is extremely valuable with many advantages compared to other approaches. There are however still situations where diverging from GitOps is very valuable as well.

For example, when you start a new deployment project, you’re usually in a state of frequent changes inside the deployment project. These frequent changes need frequent deployments and testing until you get to a point where things are stable enough. If you’re forced to adhere to GitOps in that situation, you end up with very noisy Git histories and plenty of trial-and-error deployment cycles. This is a major productivity killer and we believe there has to be a better way.

With Kluctl, you can start developing locally and deploying from your local machine, with the guarantee that what you see is what will also happen later when GitOps is introduced for the same deployment. When you’re ready, push to Git, create the appropriate KluctlDeployment resource and let GitOps/Flux do its magic.

You can also use dedicated targets for development purposes and only deploy to them from your local machine, while other targets are deployed via GitOps/Flux.

See it in action

If you want to see the Flux Kluctl Controller in action, check out template-cluster-k3s-kluctl , which is a fork of k8s@home template-cluster-k3s with all HelmReleases and Kustomizations ported to KluctlDeployments.

What now?

More documentation, guides, tutorials and examples will follow in the next few days and weeks.

Hello kluctl fans!

As can be seen, we’ve launched https://kluctl.io , which will be the future home of all kluctl related documentation, tutorials, blog articles and much more.

We’re looking forward to what is coming for kluctl!