My most recent projects have consisted of using a microservice architecture and multiple third party services (analytics, events etc.). For better or worse, this seems to be getting more popular, even for smaller companies and startups. Local development becomes more difficult with a microservice centric approach, not only computationally but also in terms of configuration management and infrastructure. This blog post tackles the issue of secret management in local development environments where you want to share secrets with the team, fetch secrets from a remote storage and inject secrets into the developer environment. The secrets I refer to in this case are dev/test secrets from 3rd party services and secrets used for microservice authentication, among other things. The to-go approach I use is HashiCorp Vault with the vault cli for fetching secrets from a remote storage.

There are many approaches to this problem, in this blog post most of it is just manipulating a JSON file that's fetched from a remote storage (Vault). I'll showcase two examples:

1. Fetching Vault secrets as JSON and converting them to an .env file.
2. Using Tanka and Jsonnet to create Kubernetes secrets from Vault secrets. This is more useful if you develop locally with for example Tilt and Minikube.

First, we need to create the secret within Vault. I choose to use a JSON secret, as it's easy to parse JSON using jq or any programming language. The developers are also familiar with JSON. Below is a sample secret.

Developers need to have access to this secret and preferably be able to edit it. Then they append to the secret or change it as the services they are building evolve.

Fetch the Vault Secret and Convert it to an .env File

We'll need to use a command that fetches the secret using the vault cli and output it to a JSON file.

vault kv get --field data -format json kv/development/local > ./secrets.json

Note: Please, add secrets.json to .gitignore if you choose to use the above file.

Next, we'll need a command that transforms the JSON file to an .env file using Python. Here's a snippet:

import json
import re

pattern = re.compile(r"(?<!^)(?=[A-Z])")


def walk(node, parent=None):
    """Walks the node and prints the key and value"""
    for key, item in node.items():
        parsed_key = pattern.sub("_", key).lower()
        if isinstance(item, dict):
            if parent:
                walk(item, f"{parent}_{parsed_key.upper()}")
            else:
                walk(item, f"{parsed_key.upper()}")
        else:
            with open(".env", "a", encoding="utf-8") as env_file:
                if parent:
                    env_file.write(f"{parent}_{parsed_key.upper()}={item}\n")
                else:
                    env_file.write(f"{parsed_key.upper()}={item}\n")


def main():
    """Main entry point of the app"""
    with open("secrets.json", encoding="utf-8") as json_file:
        secrets = json.load(json_file)
        walk(secrets)


if __name__ == "__main__":
    main()

The above Python code walks through each JSON key in a nested way and writes key=value lines to the .env file. The snippet converts the following JSON file:

{
  "test": {
    "username": "george",
    "password": "george",
    "database": {
      "host": "localhost",
      "password": "test",
      "port": "5432"
    }
  }
}

To the following .env file:

TEST_USERNAME=george
TEST_PASSWORD=george
TEST_DATABASE_HOST=localhost
TEST_DATABASE_PASSWORD=test
TEST_DATABASE_PORT=5432

Developers would need to do the following commands to have the secrets locally:

1. Login into Vault: vault login -method=oidc role=rnd // Adjust method and role.
2. Fetch the secrets: vault kv get --field data -format json kv/development/local > ./secrets.json // Expects a secret to exist in Vault under development/local.
3. Parse and populate the .env file using the command python convert-to-env.py.

Fetch the Vault Secret and Convert it to a Kubernetes Secret

We use tanka and jsonnet for provisioning Kubernetes infrastructure in production and wanted to mimic the infrastructure locally using Tilt and minikube. Tilt has the possibility of extending the user interface and adding buttons. I add a button to synchronize secrets on click.

cmd_button(
    name="update-vault-secrets",
    argv=['/bin/sh', '-c', "vault kv get --field data -format json kv/development/local > ./environments/local/secrets.jsonnet && echo 'Fetched Vault secrets successfully!' && touch Tiltfile"],
    location=location.NAV,
        icon_name="lock",
        text="Update Vault secrets",
)

Preview:

We also use jsonnet to import the JSON file and create valid Kubernetes secrets. Note: Jsonnet syntax is quite specific and hard to grasp without in-depth knowledge, especially since it uses a Kubernetes library to create the Kubernetes secret.

local secrets = import 'secrets.jsonnet'; // The secret file
{
    ...
    secrets: {
      mongo: $.core.v1.secret.new(mongo, {
        baseUrl: std.base64(secrets.mongo.baseUrl), // Fetch the value from the secret and base 64 encode the value.
        password: std.base64(secrets.mongo.password),
      }),
      rabbitmq: $.core.v1.secret.new(rabbitmq, {
        host: std.base64(secrets.rabbitmq.host),
        password: std.base64(secrets.rabbitmq.password),
      }),

The Kubernetes deployment manifest consumes the secrets, and the application consumes environment variables that Kubernetes injects into the container.

Developers would need to do the following commands to get the secrets locally:

1. Login into Vault: vault login -method=oidc role=rnd.
2. Press the Tilt UI button to refresh secrets.

Summary

There are many ways to share secrets with the team. I really like the approach of using Vault as a remote secret storage, since it has a great and user-friendly CLI. Also, Vault's user interface is great for developers to change and add new secrets when new integrations are added to services. Manipulating the fetched data from Vault is quite easy, and there are many approaches to this. Hopefully, the .env and the Kubernetes/Tilt example serves as inspiration, and you can easily go from there and adjust them to your use case.