GitOps with ArgoCD and Tanka

GitOps with ArgoCD and Tanka

1 month ago

9 min read


GitOps is becoming the standard of doing continuous delivery. Define your state in Git, automatically update and change the state when pull requests are merged. Within the Kubernetes ecosystem two tools have become very popular for doing this - FluxCD and ArgoCD. Different in their own way, but both have the same goal - managing your continuous delivery life cycle.

Tanka is a configuration management tool for Kubernetes using the Jsonnet language. The Jsonnet language is an extension of JSON which provides an easy way to manipulate the JSON data. It works very well when applying configuration to Kubernetes due to the k8s-alpha Jsonnet Kubernetes library which is generated for each Kubernetes version. It removes much of the boiler plating that comes with the configuration of Kubernetes resources. My thoughts on the benefits with Jsonnet and Tanka are:

  • A good but getting even better OSS community around Jsonnet with projects/libraries as k8s-alpha, kube-prometheus, Grafana Jsonnet library, kube-thanos amongst others.
  • Easy manipulation of JSON objects with Jsonnet, check out an introduction at Tanka.
  • Monitoring-mixins, reuse OSS Grafana dashboards and Prometheus rules. The maintainers of projects set guidelines on what to monitor and alert on and you just extend them and adjust them to your needs easily using Jsonnet. Examples of mixins are:

    Check out monitoring-mixins website for more mixins.

  • An awesome CLI which easily allows to check diffs using tk diff. It also has resource targeting which functions similarly to terraform -target, example tk diff -t Deploymenubernetes jsonnetexample.

  • Tanka provides documentation and guidelines on how to work with Jsonnet, how to use use the Jsonnet-bundler and how to setup your folder structure amongst other things. This is one downside I find with Jsonnet, the language is hard to get started with and is lacking documentation. Tanka is solving some of these problems.

ArgoCD has a feature called plugins which in theory should provide a way to use any configuration management tool with ArgoCD. FluxCD does not have the same feature parity. ArgoCD allows me to:

  • Not have to generate YAML files and push them to Git so that the CD tools syncs the YAML manifests. Just as with e.g JS/SCSS, we want to avoid pushing generated files to Git.
  • Create pull requests where the only diff is the Jsonnet code that been added, not 100s of lines of YAML. This makes the pull requests much more readable.
  • Apply my Tanka workflow automatically and mirror what I do locally. The Kubernetes resources are automatically applied to the Kubernetes cluster.

If your not familiar with ArgoCD, Tanka or Jsonnet feel free to check them out before proceeding.

Initializing Tanka

Note: A full demo of the code can be found at adinhodovic/tanka-argocd-demo.

Create a directory for your Tanka configs and run tk init. Tanka will create the following directory structure:

.
├── environments
│  └── default
│     ├── main.jsonnet
│     └── spec.json
├── jsonnetfile.json
├── jsonnetfile.lock.json
├── lib
│  └── k.libsonnet
└── vendor
   ├── github.com
   │  ├── grafana
   │  │  └── jsonnet-libs
   │  │     └── ksonnet-util
   │  │        └── kausal.libsonnet
   │  └── ksonnet
   │     └── ksonnet-lib
   │        └── ksonnet.beta.4
   │           ├── k.libsonnet
   │           └── k8s.libsonnet
   ├── ksonnet-util
   └── ksonnet.beta.4

By default you'll have an environment called default which is sufficient for this demo. A couple of Jsonnet libraries for creating and managing Kubernetes resources will also be added. The vendor directory will have all the vendor dependencies and the lib directory will have a k.libsonnet Jsonnet file which imports the Kubernetes jsonnet library and any extensions you'll want. Now that Tanka is initialized, we'll want to create our first ArgoCD project so that any future resources are automatically added using GitOps.

Deploying ArgoCD with Tanka and Jsonnet-bundler support

We'll deploy ArgoCD using Helm, and I'll do it using the HelmRelease CustomResourceDefinition(CRD) that comes with the helm-operator. Note: Tanka has recently released support for Helm charts, however the feature is experimental and I won't use it in this demo. Let's create our Helm release for ArgoCD in a new file called argo-cd.jsonnet, we'll start of with the chart specification and metadata:

  local _config = {
    name: 'argo-cd',
    repo_server_service_account: 'argo-cd-repo-server',
  },

  argo_cd: {

    helm_release: {
      kind: 'HelmRelease',
      apiVersion: 'helm.fluxcd.io/v1',
      metadata: {
        name: _config.name,
      },
      spec: {
        chart: {
          repository: 'https://argoproj.github.io/argo-helm',
          version: '2.9.3',
          name: _config.name,
        },
        releaseName: _config.name,
        values: {
        ...
        ...

The above snippet is the just a regular helm install --name argo-cd argo/argo-cd defined declaratively using the HelmRelease CRD. We'll extend the above snippet with values for the Helm chart. ArgoCD has multiple services and we'll need to tweak two of them -- the API server and the repository controller. The API server handles the application life cycle and the repository controller manages the git repository - generating and applying manifests. We'll add the following values to our API server specification:

  • A list of Git repositories we'll use along with any secrets that are neccessary in the case of the repository being private.
  • We'll also disable auth for this demo.
  • Installation of the Jsonnet packages using the Jsonnet-bundler.
  • Our config management plugin which in this case is Tanka and which arguments are the defaults.
        values: {
          server: {
            extraArgs: [
              '--disable-auth', // disable auth
            ],
            config: {
              repositories: std.manifestYamlDoc( // add our repository which we'll sync from
                [
                  {
                    url: 'https://github.com/my-org/my-repo',
                    passwordSecret: {
                      name: 'argo-cd-git',
                      key: 'password',
                    },
                    usernameSecret: {
                      name: 'argo-cd-git',
                      key: 'username',
                    },
                  },
                ],
              ),
              configManagementPlugins: std.manifestYamlDoc( // Add Tanka as a configuration management plugin
                [
                  {
                    name: 'tanka',
                    init: {
                      command: [
                        'sh',
                        '-c',
                      ],
                      args: [
                        'jb install',
                      ],
                    },
                    generate: {
                      command: [
                        'sh',
                        '-c',
                      ],
                      args: [
                        'tk show environments/${TK_ENV} --dangerous-allow-redirect ${EXTRA_ARGS}',
                      ],
                    },
                  },
                ],
              ),
            },
          },

The ArgoCD repository controller requires a service account with RBAC privileges to manage yor Kubernetes resources. We can start of by creating the service account. Note: The below service account give ArgoCD full API access, you will need to add security constraints according to your needs!

    local _config = {
        name: 'argo-cd',
        repo_server_service_account: 'argo-cd-demo-repo-server',
    },

    local serviceAccount = $.core.v1.serviceAccount,
    service_account:
      serviceAccount.new(_config.repo_server_service_account),

    local clusterRole = $.rbac.v1.clusterRole,
    local policyRule = $.rbac.v1beta1.policyRule,
    cluster_role:
      clusterRole.new() +
      clusterRole.mixin.metadata.withName(_config.repo_server_service_account) +
      clusterRole.withRulesMixin([
        policyRule.new() +
        policyRule.withApiGroups('*') +
        policyRule.withResources(['*']) +
        policyRule.withVerbs(['*']),
      ]),

    local clusterRoleBinding = $.rbac.v1.clusterRoleBinding,
    cluster_role_binding:
      clusterRoleBinding.new() +
      clusterRoleBinding.mixin.metadata.withName(_config.repo_server_service_account) +
      clusterRoleBinding.mixin.roleRef.withApiGroup('rbac.authorization.k8s.io') +
      clusterRoleBinding.mixin.roleRef.withKind('ClusterRole') +
      clusterRoleBinding.mixin.roleRef.withName(_config.repo_server_service_account) +
      clusterRoleBinding.withSubjectsMixin({
        kind: 'ServiceAccount',
        name: _config.repo_server_service_account,
        namespace: 'default',
      }),

We'll reference the service account when configuring the repoServer in the Helm values. We'll also adjust the repoServer by installing the Tanka binary and the Jsonnet-bundler to fetch all dependencies pre manifest generation.

          repoServer: {
            serviceAccount: {
              name: _config.repo_server_service_account,
            },
            volumes: [
              {
                name: 'custom-tools',
                emptyDir: {},
              },
            ],
            initContainers: [
              {
                name: 'download-tools',
                image: 'curlimages/curl',
                command: [
                  'sh',
                  '-c',
                ],
                args: [ // Using curl to fetch both Tanka and Jsonnet-bundler
                  'curl -Lo /custom-tools/jb https://github.com/jsonnet-bundler/jsonnet-bundler/releases/latest/download/jb-linux-amd64 \
                  && curl -Lo /custom-tools/tk https://github.com/grafana/tanka/releases/download/v0.12.0/tk-linux-amd64 \
                  && chmod +x /custom-tools/tk && chmod +x /custom-tools/jb',
                ],
                volumeMounts: [
                  {
                    mountPath: '/custom-tools',
                    name: 'custom-tools',
                  },
                ],
              },
            ],
            volumeMounts: [
              {
                mountPath: '/usr/local/bin/jb', // Mount jb
                name: 'custom-tools',
                subPath: 'jb',
              },
              {
                mountPath: '/usr/local/bin/tk', // Mount tk
                name: 'custom-tools',
                subPath: 'tk',
              },
            ],
          },

We are using curl as an initContainer to install the Jsonnet-bundler and Tanka and making them both executable and we mount both the binaries. ArgoCD should have all dependencies to run Tanka and the Jsonnet-bundler.

Creating your first ArgoCD Application and AppProject

An AppProject is a logical grouping of ArgoCD Applications, which control where and which Kubernetes resources can be deployed. An Application is a Kubernetes resources which is used to deploy Kubernetes resources from a Git source and also uses a destination source which is a Kubernetes server.

The ArgoCD documentation explains Applications here and AppProjects here.

We now need to add our first AppProject. We'll keep the naming consistent to the Tanka environment which is the environment called default. The below snippet is very lenient allowing any resources, destinations and sources, you can tweak this to any security demands you want to impose.

    default_project: {
      apiVersion: 'argoproj.io/v1alpha1',
      kind: 'AppProject',
      metadata: {
        name: 'default',
        finalizers: [
          'resources-finalizer.argocd.argoproj.io',
        ],
      },
      spec: {
        description: 'MyOrg Default AppProject',
        sourceRepos: [
          '*',
        ],
        clusterResourceWhitelist: [
          {
            group: '*',
            kind: '*',
          },
        ],
        destinations: [
          {
            namespace: '*',
            server: '*',
          },
        ],
      },
    },

Lastly we need our actual ArgoCD Application. The application will refer to the same Git repository as the AppProject we created above and it will also refer to that AppProject. We will use the plugin tanka and pass the environment default in the variable TK_ENV. The changes will be deployed in cluster and we wil allow ArgoCD to prune resources. This means if you delete something in Git ArgoCD will delete it in your cluster. We'll also enable self healing which means that if any Kubernetes resource is changed and deviates from our definition in Git, it will automatically self heal restoring the resource definition to the one in Git.

    default_application: {
      apiVersion: 'argoproj.io/v1alpha1',
      kind: 'Application',
      metadata: {
        name: 'default',
      },
      spec: {
        project: 'default',
        source: {
          repoURL: 'https://github.com/<my_org>/default',
          path: 'tanka',
          targetRevision: 'HEAD',
          plugin: {
            name: 'tanka',
            env: [
              {
                name: 'TK_ENV',
                value: 'default',
              },
            ],
          },
        },
        destination: {
          server: 'https://kubernetes.default.svc',
        },
        syncPolicy: {
          automated: {
            prune: true,
            selfHeal: true,
          },
        },
      },
    },

Let's now apply our changes and deploy ArgoCD with the Tanka command tk apply .. We should now have ArgoCD deployed as well as an AppProject and an Application. We can port forward the ArgoCD server to ensure that it is functioning as it should. Use kubectl to port forward the server service kubectl port-forward -n default svc/argo-cd-argo-cd-demo-server 8080:80. You should now be able to access the server UI at localhost:8080. On the root page of the ArgoCD server you should see our default Application we deployed. You can view all the details of the application as the Git repository and the environment variables. You can now push your repository to Git and ArgoCD will succefully sync the repostitory. Since we use a single Tanka environment and ArgoCD Application, ArgoCD will manage itself using GitOps, so any changes to your ArgoCD helm deployment will be automatically synced. However, if the sync would fail then you would need to manually fix it so that ArgoCD works and can sync again. You can go around this by excluding ArgoCD from the Tanka environment or use a different environment for various applications.

Now that we have ArgoCD running and it can apply manifests using Tanka we can start creating our first application using Jsonnet and Tanka.

Creating your first deployment using Tanka & ArgoCD

Our first application will be a simple echo container. We will be using the Kubernetes Jsonnet library which provides functions that removes boilerplate. We'll start of by creating our container in a new file called echo.jsonnet:

local container = $.core.v1.container,
local containerPort = $.core.v1.containerPort,
echo_container::
  container.new('echo', 'k8s.gcr.io/echoserver:1.4') +
  container.withPorts(containerPort.new('http', 8080)),

The container uses the echoserver image and exposes the port 8080. Now we'll add a deployment for the container and a service for the deployment. The service will be created using serviceFor that comes from Grafana's ksonnet-util library. To use the ksonnet-util library you can install it with the Jsonnet-bundler using the following command jb install github.com/grafana/jsonnet-libs/ksonnet-util. You can then import it in your main Jsonnet file.

  local deployment = $.apps.v1.deployment,
  echo_deployment:
    deployment.new('echo', 1, [self.echo_container]),

  oauth2_proxy_service:
    $.util.serviceFor(self.echo_deployment),

Now we will import our echo.jsonnet files in our main Jsonnet file.

(import 'ksonnet-util/kausal.libsonnet') +
(import 'argo-cd.jsonnet') +
(import 'echo.jsonnet')

We can compare diffs between the cluster and our added Jsonnet code. We'll use Tanka's diff function with the target flag to only show diff for the deployment tk diff . -t deployment/echo. Tanka will display the diff and you can introspect it to ensure that you are content with the changes. However, we do not want to apply the changes, we want to push them to git and ArgoCD will take over from there. Push them and head over to localhost:8080 where the ArgoCD server is port forwarded. ArgoCD syncs every 5 minutes by default so it might take a minute or two for your changes to be synced. You can add webhooks or lower the sync time to remove the delay. When ArgoCD has synced your changes a DAG should be generated for the Application displaying the echo Service, Deployment, ReplicaSet and Pod.

ArgoCD Tanka

Now you've applied the GitOps workflow by using the continuous delivery tool ArgoCD which synced our changes from Git. On top of that we got to do it with Tanka and Jsonnet without any YAML template generation.

A full demo of the code can be found at adinhodovic/tanka-argocd-demo.

Next up is secret injection when using Tanka and ArgoCD with Hashicorp's Vault.


Similar Posts

1 month ago
jsonnet argo-cd tanka vault devops ci/cd

GitOps Secret Management with Vault, ArgoCD and Tanka

7 min read

Recently I wrote a blog post on how to use Grafana's Tanka with ArgoCD which is my prefered way to write Kubernetes configuration in Jsonnet. However, the post does not go into detail on the …


2 months ago
infrastructure argo-cd ci-cd devops

Migrating Kubernetes Resources between ArgoCD Applications

1 min read

I've been using ArgoCD for a while now, and as time went by I started to splitting my Kubernetes resources into smaller ArgoCD Applications. However, I could not figure out clear guidelines on how to …


1 year ago Popular post!
devops gitlab-ci kaniko automation ci/cd

Creating templates for Gitlab CI Jobs

4 min read

Writing Gitlab CI templates becomes repetitive when you have similar applications running the same jobs. If a change to a job is needed it will be most likely needed to do the same change in …