GitOps with ArgoCD and Tanka

GitOps with ArgoCD and Tanka

3 weeks ago New!

9 min read


GitOps is becoming the standard of doing continuous delivery. Define your state in Git, eutomatically update and change the state when pull requests are merged. Within the Kubernetes ecosystem two tools have become very popular for doing this FluxCD and ArgoCD. Different in their own way, but both have the same goal - managing your continuous delivery life cycle.

Tanka is a configuration management tool for Kubernetes using the Jsonnet language. The Jsonnet language is an extension of JSON which provides an easy way to manipulate the JSON data. It works very well when applying configuration to Kubernetes due to the k8s-alpha Jsonnet Kubernetes library which is generated for each Kubernetes version. It removes much of the boiler plating that comes with the configuration of Kubernetes resources. My subjective opinion on the benefits with Jsonnet and Tanka are:

  • A good but getting even better OSS community around Jsonnet with projects/libraries as k8s-alpha, kube-prometheus, Grafana Jsonnet library, kube-thanos amongst other
  • Easy manipulation of JSON objects with Jsonnet, check out a introduction at Tanka
  • Monitoring-mixins, reuse OSS Grafana dashboards and Prometheus rules. The maintainers of projects set guidelines on what to monitor and alert on and you just extend them and change them easily using Jsonnet. Examples of mixins are:

    Check out monitoring-mixins website for more mixins.

  • An awesome CLI which easily allows to check diffs using 'tk diff'. It also has resource targeting which functions similarly to terraform -target, tk diff -t Deployment/example.

  • Tanka provides documentation and guidelines on how to work with Jsonnet, using the Jsonnet-bundler and how to setup your folder structure amongst other things. This is one downside I find with Jsonnet, the language is hard to get started with and is lacking documentation. Tanka is solving some of these problems.

ArgoCD has a feature called plugins which in theory should provide a way to use any configuration management tool with ArgoCD. FluxCD does not have the same feature parity. ArgoCD allows me to:

  • Not have to autogenerate YAML files and push them to Git so that the CD tools syncs the yaml. Just as with e.g JS/SCSS, we want to avoid pushing statically generated files to Git.
  • Create PRs where the only diff is the Jsonnet code that been added, not 100s of lines of YAML. This makes the PRs much more readable.
  • Apply my Tanka workflow automatically and mirror what I do locally to what is automatically applied to the Kubernetes cluster.

If your not familiar with ArgoCD, Tanka or Jsonnet feel free to check them out before proceeding.

Initializing Tanka

Note: A full demo of the code can be found at adinhodovic/tanka-argocd-demo.

Create a directory for your Tanka configs and run tk init. Tanka will create the following directory structure:

.
├── environments
│  └── default
│     ├── main.jsonnet
│     └── spec.json
├── jsonnetfile.json
├── jsonnetfile.lock.json
├── lib
│  └── k.libsonnet
└── vendor
   ├── github.com
   │  ├── grafana
   │  │  └── jsonnet-libs
   │  │     └── ksonnet-util
   │  │        └── kausal.libsonnet
   │  └── ksonnet
   │     └── ksonnet-lib
   │        └── ksonnet.beta.4
   │           ├── k.libsonnet
   │           └── k8s.libsonnet
   ├── ksonnet-util
   └── ksonnet.beta.4

By default you'll have an environment called default which is sufficient for this demo. A couple of jsonnet libraries for creating and managing Kubernetes resources will also be added. The vendor directory will have all the vendored dependensies and the lib directory will have a k.libsonnet Jsonnet file which imports the K8s library and any extensions you'll want. Now that Tanka is initialized, we'll want to create our first ArgoCD project so that any future resources are automatically added using GitOps.

Deploying ArgoCD with Tanka and Jsonnet-bundler support

We'll deploy ArgoCD using helm, and I'll do it using HelmRelease CRDs that come with the helm-operator. Note: Tanka has recently released support for Helm charts, however the feature is experimental and I won't use it in this demo. Let's create our helm release for ArgoCD in a new file called argo-cd.jsonnet, we'll start of with the chart spec and metadata:

  local _config = {
    name: 'argo-cd',
    repo_server_service_account: 'argo-cd-repo-server',
  },

  argo_cd: {

    helm_release: {
      kind: 'HelmRelease',
      apiVersion: 'helm.fluxcd.io/v1',
      metadata: {
        name: _config.name,
      },
      spec: {
        chart: {
          repository: 'https://argoproj.github.io/argo-helm',
          version: '2.9.3',
          name: _config.name,
        },
        releaseName: _config.name,
        values: {
        ...
        ...

The above snippet is the just a regular helm install --name argo-cd argo/argo-cd defined declaratively using the HelmRelease CRD. We'll extend the above snippet with values for the helm chart. ArgoCD has multiple services and we'll need to tweak two of them -- the api server and the repo controller. The api server handles the application life cycle amongst other things and the repo controller manages the git repository generating manifests. We'll add the following values to our api server:

  • A list of Git repositories we'll use along with any secrets that are neccassary in case the repository is private.
  • Our config management plugin which in this case is Tanka and which arguments are the defaults.
  • We'll also disable auth for this demo.
        values: {
          server: {
            extraArgs: [
              '--disable-auth',
            ],
            config: {
              repositories: std.manifestYamlDoc(
                [
                  {
                    url: 'https://github.com/my-org/my-repo',
                    passwordSecret: {
                      name: 'argo-cd-git',
                      key: 'password',
                    },
                    usernameSecret: {
                      name: 'argo-cd-git',
                      key: 'username',
                    },
                  },
                ],
              ),
              configManagementPlugins: std.manifestYamlDoc(
                [
                  {
                    name: 'tanka',
                    init: {
                      command: [
                        'sh',
                        '-c',
                      ],
                      args: [
                        'jb install',
                      ],
                    },
                    generate: {
                      command: [
                        'sh',
                        '-c',
                      ],
                      args: [
                        'tk show environments/${TK_ENV} --dangerous-allow-redirect ${EXTRA_ARGS}',
                      ],
                    },
                  },
                ],
              ),
            },
          },

ArgoCD requires a service account with RBAC privileges to manage yor Kubernetes resources. We can start of by creating the service account. Note: The below service account give ArgoCD full API access, you will need to add security constraints according to your needs!

    local _config = {
        name: 'argo-cd',
        repo_server_service_account: 'argo-cd-demo-repo-server',
    },

    local serviceAccount = $.core.v1.serviceAccount,
    service_account:
      serviceAccount.new(_config.repo_server_service_account),

    local clusterRole = $.rbac.v1.clusterRole,
    local policyRule = $.rbac.v1beta1.policyRule,
    cluster_role:
      clusterRole.new() +
      clusterRole.mixin.metadata.withName(_config.repo_server_service_account) +
      clusterRole.withRulesMixin([
        policyRule.new() +
        policyRule.withApiGroups('*') +
        policyRule.withResources(['*']) +
        policyRule.withVerbs(['*']),
      ]),

    local clusterRoleBinding = $.rbac.v1.clusterRoleBinding,
    cluster_role_binding:
      clusterRoleBinding.new() +
      clusterRoleBinding.mixin.metadata.withName(_config.repo_server_service_account) +
      clusterRoleBinding.mixin.roleRef.withApiGroup('rbac.authorization.k8s.io') +
      clusterRoleBinding.mixin.roleRef.withKind('ClusterRole') +
      clusterRoleBinding.mixin.roleRef.withName(_config.repo_server_service_account) +
      clusterRoleBinding.withSubjectsMixin({
        kind: 'ServiceAccount',
        name: _config.repo_server_service_account,
        namespace: 'default',
      }),

We'll reference the service account when configuring the repoServer in the Helm values. We'll also adjust the repoServer by installing the Tanka binary and the Jsonnet-bundler to fetch all dependencies pre manifest generation.

          repoServer: {
            serviceAccount: {
              name: _config.repo_server_service_account,
            },
            volumes: [
              {
                name: 'custom-tools',
                emptyDir: {},
              },
            ],
            initContainers: [
              {
                name: 'download-tools',
                image: 'curlimages/curl',
                command: [
                  'sh',
                  '-c',
                ],
                args: [
                  'curl -Lo /custom-tools/jb https://github.com/jsonnet-bundler/jsonnet-bundler/releases/latest/download/jb-linux-amd64 \
                  && curl -Lo /custom-tools/tk https://github.com/grafana/tanka/releases/download/v0.12.0/tk-linux-amd64 \
                  && chmod +x /custom-tools/tk && chmod +x /custom-tools/jb',
                ],
                volumeMounts: [
                  {
                    mountPath: '/custom-tools',
                    name: 'custom-tools',
                  },
                ],
              },
            ],
            volumeMounts: [
              {
                mountPath: '/usr/local/bin/jb',
                name: 'custom-tools',
                subPath: 'jb',
              },
              {
                mountPath: '/usr/local/bin/tk',
                name: 'custom-tools',
                subPath: 'tk',
              },
            ],
          },

We are using curl as an initContainer to install the Jsonnet-bundler and Tanka and making them both executable and we mount both the binaries. ArgoCD should have all dependencies to run Tanka and the Jsonnet-bundler.

Creating your first ArgoCD Application and AppProject

An AppProject is a logical grouping of Applications, which controles where and which Kubernetes resources can be deployed. An Application is a Kubernetes resources which is used to deploy Kubernetes resources from a Git source and also uses a destination source which is a Kubernetes server. The ArgoCD documentation explains Applications here and AppProjects here.

We now need to add our first AppProject. We'll keep the naming consistent to the Tanka environment which is the environment called default. The below snippet is very lenient allowing any resources, destinations and sources, you can tweak this to any security demands you want to impose.

    default_project: {
      apiVersion: 'argoproj.io/v1alpha1',
      kind: 'AppProject',
      metadata: {
        name: 'default',
        finalizers: [
          'resources-finalizer.argocd.argoproj.io',
        ],
      },
      spec: {
        description: 'MyOrg Default AppProject',
        sourceRepos: [
          '*',
        ],
        clusterResourceWhitelist: [
          {
            group: '*',
            kind: '*',
          },
        ],
        destinations: [
          {
            namespace: '*',
            server: '*',
          },
        ],
      },
    },

Lastly we need our actual ArgoCD Application. The application will refer to the same repository as the AppProject we created above and it will also refer to that AppProject. We will use the plugin tanka and pass the environment default in the variable TK_ENV. The changes will be deployed in cluster and we wil allow ArgoCD to prune resources. This means if you delete something in Git ArgoCD will delete it in your cluster.

    default_application: {
      apiVersion: 'argoproj.io/v1alpha1',
      kind: 'Application',
      metadata: {
        name: 'default',
      },
      spec: {
        project: 'default',
        source: {
          repoURL: 'https://github.com/myOrg/default',
          path: 'tanka',
          targetRevision: 'HEAD',
          plugin: {
            name: 'tanka',
            env: [
              {
                name: 'TK_ENV',
                value: 'default',
              },
            ],
          },
        },
        destination: {
          server: 'https://kubernetes.default.svc',
        },
        syncPolicy: {
          automated: {
            prune: true,
            selfHeal: true,
          },
        },
      },
    },

Let's now apply our changes and deploy ArgoCD with the Tanka command tk apply .. We should now have ArgoCD deployed as well as an AppProject and an Application. We can port forward the ArgoCD server to ensure that it is functioning as it should. Use kubectl to port forward the server service kubectl port-forward -n default svc/argo-cd-argo-cd-demo-server 8080:80. You should now be able to access the server UI at localhost:8080. On the root page of the ArgoCD server you should see our default Application we deployed. You can view all the details of the application as Git repository, environment variables. Press the application and view if it has synced. Since we use a single Tanka environment and ArgoCD Application, ArgoCD will manage it self using GitOps, so any changes to your ArgoCD helm deployment will be automatically synced. However, if the sync would fail then you would need to manually fix it so that ArgoCD works and can sync again. You can go around this by excluding ArgoCD from the Tanka environment or use a different environment for various applications.

Now that we have ArgoCD running and it can apply manifests using Tanka we can start creating our first application using Jsonnet and Tanka.

Creating your first deployment using Tanka & ArgoCD

Our first application will be a simple echo container. We will be using the Kubernetes jsonnet library which provides functions that removes boilerplate. We'll start of by creating our container in a new file called echo.jsonnet:

local container = $.core.v1.container,
local containerPort = $.core.v1.containerPort,
echo_container::
  container.new('echo', 'k8s.gcr.io/echoserver:1.4') +
  container.withPorts(containerPort.new('http', 8080)),

The container uses the echoserver image and exposes the port 8080. Now we'll add a deployment for the container and a service for the deployment. The service will be created using serviceFor that comes from Grafana's ksonnet-util library. To use the ksonnet-util library you can install it with the follwing command jb install github.com/grafana/jsonnet-libs/ksonnet-util. You can then import it in your main Jsonnet file.

  local deployment = $.apps.v1.deployment,
  echo_deployment:
    deployment.new('echo', 1, [self.echo_container]),

  oauth2_proxy_service:
    $.util.serviceFor(self.echo_deployment),

Now we will import our echo.jsonnet files in our main Jsonnet file.

(import 'ksonnet-util/kausal.libsonnet') +
(import 'argo-cd.jsonnet') +
(import 'echo.jsonnet')

We can compare diffs between the cluster and our added jsonnet code. We'll use Tanka's diff function with the target flag to only show diff for the deployment tk diff . -t deployment/echo. Tanka will display the diff and you can introspect it to ensure that you are content with the changes. However, we do not want to apply the changes, we want to push them to git and ArgoCD will take over from there. Push them and head over to localhost:8080 where the ArgoCD server is port forwarded. ArgoCD syncs every 5 minutes by default so it might take a minute or two for your changes to be synced. You can add webhooks or lower the sync time to remove the delay. When ArgoCD has synced your changes a DAG should be generated for the Application displaying the echo Service, Deployment, ReplicaSet and Pod.

ArgoCD Tanka

Now you've applied the GitOps workflow, using the continuous delivery tool ArgoCD which synced our changes from Git. On top of that we got to do it with Tanka and Jsonnet without any YAML template generation.

A full demo of the code can be found at adinhodovic/tanka-argocd-demo.

Next up is secret injection when using Tanka and ArgoCD with Hashicorp's Vault.


Similar Posts

3 weeks ago New!
infrastructure argo-cd ci-cd devops

Migrating Kubernetes Resources between ArgoCD Applications

1 min read

I've been using ArgoCD for a while now, and as time went by I started to splitting my Kubernetes resources into smaller ArgoCD Applications. However, I could not figure out clear guidelines on how to …


1 year ago
mailgun statuscake terraform cloudflare devops s3 rds django

Kickstarting Infrastructure for Django Applications with Terraform

8 min read

When creating Django applications or using cookiecutters as Django Cookiecutter you will have by default a number of dependencies that will be needed to be created as a S3 bucket, a Postgres Database and a …


1 year ago
devops gitlab-ci kaniko automation ci/cd

Creating templates for Gitlab CI Jobs

4 min read

Writing Gitlab CI templates becomes repetitive when you have similar applications running the same jobs. If a change to a job is needed it will be most likely needed to do the same change in …