Templating

• 1: Predefined Variables
• 2: Variable Sources
• 3: Filters
• 4: Functions

1 - Predefined Variables

There are multiple variables available which are pre-defined by kluctl. These are:

args

This is a dictionary of arguments given via command line. It contains every argument defined in deployment args.

target

This is the target definition of the currently processed target. It contains all values found in the target definition, for example target.name.

images

This global object provides the dynamic images features described in images.

2 - Variable Sources

There are multiple places in deployment projects (deployment.yaml) where additional variables can be loaded into future Jinja2 contexts.

The first place where vars can be specified is the deployment root, as documented here. These vars are visible for all deployments inside the deployment project, including sub-deployments from includes.

The second place to specify variables is in the deployment items, as documented here.

The variables loaded for each entry in vars are not available inside the deployment.yaml file itself. However, each entry in vars can use all variables defined before that specific entry is processed. Consider the following example.

vars:
- file: vars1.yaml
- file: vars2.yaml
- file: optional-vars.yaml
  ignoreMissing: true
- file: default-vars.yaml
  noOverride: true
- file: vars3.yaml
  when: some.var == "value"
- file: vars3.yaml
  sensitive: true
- file: vars4.yaml
  targetPath: my.target.path

vars2.yaml can now use variables that are defined in vars1.yaml. A special case is the use of previously defined variables inside values vars sources. Please see the documentation of values for details.

At all times, variables defined by parents of the current sub-deployment project can be used in the current vars source.

The following properties can be set on all variable sources:

ignoreMissing

Each variable source can have the optional field ignoreMissing set to true, causing Kluctl to ignore if the source can not be found.

noOverride

When specifying noOverride: true, Kluctl will not override variables from the previously loaded variables. This is useful if you want to load default values for variables.

when

Variables can also be loaded conditionally by specifying a condition via when: <condition>. The condition must be in the same format as described in conditional deployment items

sensitive

Specifying sensitive: true causes the Webui to redact the underlying variables for non-admin users. This will be set to true by default for all variable sources that usually load sensitive data, including sops encrypted files and Kubernetes secrets.

targetPath

Specifies a JSON path to be used as the target path in the new templating context.

Only simple pathes are supported that do not contain wildcards or lists.

For some variable sources, targetPath will become mandatory when the resulting variable is not a dictionary.

Variable source types

Different types of vars entries are possible:

file

This loads variables from a yaml file. Assume the following yaml file with the name vars1.yaml:

my_vars:
  a: 1
  b: "b"
  c:
    - l1
    - l2

This file can be loaded via:

vars:
  - file: vars1.yaml

After which all included deployments and sub-deployments can use the jinja2 variables from vars1.yaml.

Kluctl also supports variable files encrypted with SOPS. See the sops integration integration for more details.

values

An inline definition of variables. Example:

vars:
  - values:
      a: 1
      b: c

These variables can then be used in all deployments and sub-deployments.

In case you need to use variables defined in previous vars sources, the values var source needs some special handling in regard to templating. It’s important to understand that the deployment project is rendered BEFORE any vars source processing is performed, which means that it will fail to render when you use previously defined variables in a values vars source. To still use previously defined variables, surround the values vars source with {% raw %} and {% endraw %}. In addition, the template expressions must be wrapped with ", as otherwise the loading of the deployment project will fail shortly after rendering due to YAML parsing errors.

vars:
  - values:
      a: 1
      b: c
{% raw %}
  - values:
      c: "{{ a }}"
{% endraw %}

An alternative syntax is to use a template expression that itself outputs a template expression:

vars:
  - values:
      a: 1
      b: c
  - values:
      c: {{ '{{ a }}' }}

The advantage of the second method is that the type (number) of a is preserved, while the first method would convert it into a string.

git

This loads variables from a file inside a git repository. Example:

vars:
  - git:
      url: ssh://git@github.com/example/repo.git
      ref:
        branch: my-branch
      path: path/to/vars.yaml

The ref field has the same format at found in Git includes

Kluctl also supports variable files encrypted with SOPS. See the sops integration integration for more details.

gitFiles

This loads multiple branches/tags and its contents from a git repository. The branches/tags can be filtered via regex and the files to load can be filtered via globs. Files can also be parsed and interpreted as yaml. Providing targrtPath is mandatory for this variables source.

Example:

vars:
  - gitFiles:
      url: ssh://git@github.com/example/repo.git
      ref:
        branch: preview-env-.*
      files:
        - glob: preview-info.yaml
          parseYaml: true
    targetPath: previewEnvs

The following fields are supported for gitFiles.

url

Specified the Git url.

ref

Specifies the ref to match. The ref field has the same format at found in Git includes, with the addition that branches and tags can specify regular expressions.

files

Specifies a list of file filters. Each entry can have the following fields:

gitFiles result

The above example will put the result into the variable previewEnvs. The result is a list of matching branches/tags with each entry having the following form:

previewEnvs:
- ref:
    branch: preview-env-1
  refStr: refs/heads/preview-env-1
  files:
  - path: preview-info.yaml
    size: 1234
    content: |
      some:
        arbitrary:
          yamlContent: 42      
    parsed:
      some:
        arbitrary:
          yamlContent: 42
    # this is a copy of the original `gitFiles.files` entry that caused this match
    file:
      glob: preview-info.yaml
      parseYaml: true
  # this is a flat dict with each entry being a copy of what is found in `files` for that same entry
  # it is indexed by the relative path of each file
  filesByPath:
    preview-info.yaml:
      path: preview-info.yaml
      content: ...
    dir1/sub-dir/file.yaml:
      path: dir1/sub-dir/file.yaml
      content: ...
  # this is a nested dict that follows the directory structure
  filesTree:
    preview-info.yaml:
      path: preview-info.yaml
      content: ...
    dir1:
      sub-dir:
        file.yaml:
          path: dir1/sub-dir/file.yaml
          content: ...
- ref:
    branch: preview-env-2
  ...

Each file entry, as found in files, filesByPath and filesTree has the following fields:

clusterConfigMap

Loads a configmap from the target’s cluster and loads the specified key’s value into the templating context. The value is treated and loaded as YAML and thus can either be a simple value or a complex nested structure. In case of a simple value (e.g. a number), you must also specify targetPath.

The referred ConfigMap must already exist while the Kluctl project is loaded, meaning that it is not possible to use a ConfigMap that is deployed as part of the Kluctl project itself.

Assume the following ConfigMap to be already deployed to the target cluster:

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-vars
  namespace: my-namespace
data:
  vars: |
    a: 1
    b: "b"
    c:
      - l1
      - l2    

This ConfigMap can be loaded via:

vars:
  - clusterConfigMap:
      name: my-vars
      namespace: my-namespace
      key: vars

The following example uses a simple value:

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-vars
  namespace: my-namespace
data:
  value: 123

This ConfigMap can be loaded via:

vars:
  - clusterConfigMap:
      name: my-vars
      namespace: my-namespace
      key: value
    targetPath: deep.nested.path

clusterSecret

Same as clusterConfigMap, but for secrets.

clusterObject

Retrieves an arbitrary Kubernetes object from the target’s cluster and loads the specified content under path into the templating context. The content can either be interpreted as is or interpreted and loaded as yaml text. In both cases, rendering with the current context (without the newly introduced variables) can also be enabled.

targetPath must also be specified to configure under which sub-keys the new variables should be loaded.

The referred Kubernetes object must already exist while the Kluctl project is loaded, meaning that it is not possible to use an object that is deployed as part of the Kluctl project itself. The exception to this is when you use ignoreMissing: true and properly handle the missing case inside your templating (an example can be found further down).

Objects can either be referred to by name or by labels. In case of labels, Kluctl assumes that only a single object matches. If multiple object are expected to match, list: true must also be passed, in which case the result loaded into targetPath will be a list of objects instead of a single object.

Assume the following object to be already deployed to the target cluster:

apiVersion: some.group/v1
kind: SomeObject
metadata:
  name: my-object
  namespace: my-namespace
spec:
  ...
status:
  my-status: all-good

This object can be loaded via:

vars:
  - clusterObject:
      kind: SomeObject
      name: my-object
      namespace: my-namespace
      path: status
    targetPath: my.custom.object.status

The following properties are supported for clusterObject sources:

kind (required)

The object kind. Kluctl will try to find the matching Kubernetes resource for this kind, which might either be a native API resource or a custom resource. If multiple resources match, apiVersion must also be specified.

apiVersion (optional)

The apiVersion of the object. This field is only required if kind is not enough to identify the underlying API resource.

namespace (required)

The namespace from which to load the object.

name (optional)

The name of the object. If specified, the object with the given name must exist (ignoreMissing: true can override this).

Can be omitted when labels is specified.

labels (optional)

Specifies one or multiple labels to match. If specified, name is not allowed.

By default, assumes and requires (unless ignoreMissing: true is set) that only one object matches. If multiple objects are assumed to match, set list: true as well, in which case the result will be a list as well.

list (optional)

If set to true, the result will be a list with one or more elements.

path (required)

Specifies a JSON path to be used to load a sub-key from the matching object(s). Use $ to load the whole object. To load a single field, use something like status.my.field. To load a whole sub-dict/sub-object or sub-list, use something like status.conditions.

The specified JSON path is only allowed to result in a single match.

render (optional)

If set to true, Kluctl will render the resulting object(s) with the current templating context (excluding the newly loaded variables). Rendering happens on the values of individual fields of the resulting object(s). When parseYaml: true is specified as well, rendering happens before parsing the YAML string.

parseYaml (optional)

Instructs Kluctl to treat the value found at path as a YAML string. The value must be of type string. Kluctl will parse the string as YAML and use the resulting YAML value (which can be a simple int/float/bool or a complex list/dict) as the result and store it in targetPath. When render: true is specified as well, the YAML string is rendered before parsing happens.

http

The http variables source allows to load variables from an arbitrary HTTP resource by performing a GET (or any other configured HTTP method) on the URL. Example:

vars:
  - http:
      url: https://example.com/path/to/my/vars

The above source will load a variables file from the given URL. The file is expected to be in yaml or json format.

The following additional properties are supported for http sources:

method

Specifies the HTTP method to be used when requesting the given resource. Defaults to GET.

body

The body to send along with the request. If not specified, nothing is sent.

headers

A map of key/values pairs representing the header entries to be added to the request. If not specified, nothing is added.

jsonPath

Can be used to select a nested element from the yaml/json document returned by the HTTP request. This is useful in case some REST api is used which does not directly return the variables file. Example:

vars:
  - http:
      url: https://example.com/path/to/my/vars
      jsonPath: $[0].data

The above example would successfully use the following json document as variables source:

[{"data": {"vars": {"var1": "value1"}}}]

Authentication

Kluctl currently supports BASIC and NTLM authentication. It will prompt for credentials when needed.

awsSecretsManager

AWS Secrets Manager integration. Loads a variables YAML from an AWS Secrets Manager secret. The secret can either be specified via an ARN or via a secretName and region combination. An existing AWS config profile can also be specified.

The secrets stored in AWS Secrets manager must contain a valid yaml or json file.

Example using an ARN:

vars:
  - awsSecretsManager:
      secretName: arn:aws:secretsmanager:eu-central-1:12345678:secret:secret-name-XYZ
      profile: my-prod-profile

Example using a secret name and region:

vars:
  - awsSecretsManager:
      secretName: secret-name
      region: eu-central-1
      profile: my-prod-profile

The advantage of the latter is that the auto-generated suffix in the ARN (which might not be known at the time of writing the configuration) doesn’t have to be specified.

gcpSecretManager

Google Secret Manager integration. Loads a variables YAML from a Google Secrets Manager secret. The secret name should be specified in projects/*/secrets/*/versions/* format.

The secrets stored in Google Secrets manager must contain a valid yaml or json file.

Example:

vars:
  - gcpSecretManager:
      secretName: "projects/my-project/secrets/secret/versions/latest"

It is recommended to use workload identity when you are using kluctl controller. You will need to annotate kluctl controller service account with service account name created in your google project:

    args:
      controller_service_account_annotations:
        iam.gke.io/gcp-service-account: kluctl-controller@PROJECT-NAME.iam.gserviceaccount.com

substitute PROJECT-NAME with your real project name in google. Service account in your google project should have role roles/secretmanager.secretAccessor to access secrets.

To run kluctl locally with gcpSecretManager enabled refer to setting local development environment article.

azureKeyVault

Azure Key Vault integration. Loads a variables YAML from an Azure Key Vault.

Example

vars:
  - azureKeyVault:
      vaultUri: "https://example.vault.azure.net/"
      secretName: kluctl

SDK azure-sdk-for-go supports az login or Environment Variables

$ export AZURE_CLIENT_ID="__CLIENT_ID__"
$ export AZURE_CLIENT_SECRET="__CLIENT_SECRET__"
$ export AZURE_TENANT_ID="__TENANT_ID__"
$ export AZURE_SUBSCRIPTION_ID="__SUBSCRIPTION_ID__"

vault

Vault by HashiCorp with Tokens authentication integration. The address and the path to the secret can be configured. The implementation was tested with KV Secrets Engine.

Example using vault:

vars:
  - vault:
      address: http://localhost:8200
      path: secret/data/simple

Before deploying please make sure that you have access to vault. You can do this for example by setting the environment variable VAULT_TOKEN.

systemEnvVars

Load variables from environment variables. Children of systemEnvVars can be arbitrary yaml, e.g. dictionaries or lists. The leaf values are used to get a value from the system environment.

Example:

vars:
- systemEnvVars:
    var1: ENV_VAR_NAME1
    someDict:
      var2: ENV_VAR_NAME2
    someList:
      - var3: ENV_VAR_NAME3

The above example will make 3 variables available: var1, someDict.var2 and someList[0].var3, each having the values of the environment variables specified by the leaf values.

All specified environment variables must be set before calling kluctl unless a default value is set. Default values can be set by using the ENV_VAR_NAME:default-value form.

Example:

vars:
- systemEnvVars:
    var1: ENV_VAR_NAME4:defaultValue

The above example will set the variable var1 to defaultValue in case ENV_VAR_NAME4 is not set.

All values retrieved from environment variables (or specified as default values) will be treated as YAML, meaning that integers and booleans will be treated as integers/booleans. If you want to enforce strings, encapsulate the values in quotes.

Example:

vars:
- systemEnvVars:
    var1: ENV_VAR_NAME5:'true'

The above example will treat true as a string instead of a boolean. When the environment variable is set outside kluctl, it should also contain the quotes. Please note that your shell might require escaping to properly pass quotes.

3 - Filters

In addition to the builtin Jinja2 filters, kluctl provides a few additional filters:

b64encode

Encodes the input value as base64. Example: {{ "test" | b64encode }} will result in dGVzdA==.

b64decode

Decodes an input base64 encoded string. Example {{ my.source.var | b64decode }}.

from_yaml

Parses a yaml string and returns an object. Please note that json is valid yaml, meaning that you can also use this filter to parse json.

to_yaml

Converts a variable/object into its yaml representation. Please note that in most cases the resulting string will not be properly indented, which will require you to also use the indent filter. Example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-config
data:
  config.yaml: |
    {{ my_config | to_yaml | indent(4) }}    

to_json

Same as to_yaml, but with json as output. Please note that json is always valid yaml, meaning that you can also use to_json in yaml files. Consider the following example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  template:
    spec:
      containers:
      - name: c1
        image: my-image
        env: {{ my_list_of_env_entries | to_json }}

This would render json into a yaml file, which is still a valid yaml file. Compare this to how this would have to be solved with to_yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  template:
    spec:
      containers:
      - name: c1
        image: my-image
        env:
          {{ my_list_of_env_entries | to_yaml | indent(10) }}

The required indention filter is the part that makes this error-prone and hard to maintain. Consider using to_json whenever you can.

render

Same as the global render function, but deprecated now. render being a filter turned out to not work well with local variables, as these are not accessible in filters. Please only use the global function.

sha256(digest_len)

Calculates the sha256 digest of the input string. Example:

{{ "some-string" | sha256 }}

digest_len is an optional parameter that allows to limit the length of the returned hex digest. Example:

{{ "some-string" | sha256(6) }}

slugify

Slugify a string based on python-slugify.

4 - Functions

In addition to the provided builtin global functions, kluctl also provides a few global functions:

load_template(file)

Loads the given file into memory, renders it with the current Jinja2 context and then returns it as a string. Example:

{% set a=load_template('file.yaml') %}
{{ a }}

load_template uses the same path searching rules as described in includes/imports.

Please note that there is a limitation in this (and other) functions in regard to loop variables. You can currently not use loop variables directly as they are not accessible inside Jinja2 extensions/filters. There is an open issue in that regard here. For a workaround, perform the same as in get_var.

load_sha256(file, digest_len)

Loads the given file into memory, renders it and calculates the sha256 hash of the result.

The filename given to load_sha256 is treated the same as in load_template. Recursive loading/calculating of hashes is allowed and is solved by replacing load_sha256 invocations with currently loaded templates with dummy strings. This also allows to calculate the hash of the currently rendered template, for example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-config-{{ load_sha256("configmap.yaml") }}
data:

digest_len is an optional parameter that allows to limit the length of the returned hex digest.

load_base64(file, width)

Loads the given file into memory and returns the base64 representation of the binary data. The width parameter is optional and causes load_base64 to wrap the base64 string into a multiline string.

The filename given to load_base64 is treated the same as in load_template.

This function is useful if you need to include binary data in your deployment. For example:

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
data:
  binarySecret: "{{ load_base64("secret.bin") }}"

To use wrapped base64, use:

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
data:
  binarySecret: |
    "{{ load_base64("large-secret.bin") | indent(4) }}"

get_var(field_path, default)

Convenience method to navigate through the current context variables via a JSON Path. Let’s assume you currently have these variables defined (e.g. via vars):

my:
  deep:
    var: value

Then {{ get_var('my.deep.var', 'my-default') }} would return value. When any of the elements inside the field path are non-existent, the given default value is returned instead.

The field_path parameter can also be a list of pathes, which are then tried one after the another, returning the first result that gives a value that is not None. For example, {{ get_var(['non.existing.var', my.deep.var'], 'my-default') }} would also return value.

Please note that there is a limitation in this (and other) functions in regard to loop variables. You can currently not use loop variables directly as they are not accessible inside Jinja2 global functions or filters. There is an open issue in that regard here. For a workaround, assign the loop variable to a local variable:

{% set list=[{x: "a"}, {x: "b"}, {x: "c"}] %}
{% for e in list %}
{% set e=e %} <-- this is the workaround
{{ get_var('e.x') }}
{% endfor %}

merge_dict(d1, d2)

Clones d1 and then recursively merges d2 into it and returns the result. Values inside d2 will override values in d1.

update_dict(d1, d2)

Same as merge_dict, but merging is performed in-place into d1.

raise(msg)

Raises a python exception with the given message. This causes the current command to abort.

render(template)

Renders the input string with the current Jinja2 context. Example:

{% set a="{{ my_var }}" %}
{{ render(a) }}

Please note that there is a limitation in this (and other) functions in regard to loop variables. You can currently not use loop variables directly as they are not accessible inside Jinja2 global functions or filters. There is an open issue in that regard here. For a workaround, perform the same as in get_var.

debug_print(msg)

Prints a line to stderr.

time.now()

Returns the current time. The returned object has the following members:

time.utcnow()

Returns the current time in UTC. The object has the same members as described in time.now().

time.parse_iso(iso_time_str)

Parse the given string and return a time object. The string must be in ISO time. The object has the same members as described in time.now().

time.second, time.minute, time.hour

Represents a time delta to be used with t + delta and t - delta. Example

{{ time.now() + time.minute * 10 }}