The kube-prometheus-stack Helm chart, which deploys the kubernetes-mixin, is designed for standard Kubernetes setups, often pre-configured for specific cloud environments. However, these configurations are not directly compatible with k3s, a lightweight Kubernetes distribution. Since k3s lacks many of the default cloud integrations, issues arise, such as missing metrics, broken graphs, and unavailable endpoints (example issue). This blog post will guide you through adapting the kube-prometheus-stack Helm chart and the kubernetes-mixin to work seamlessly in k3s environments, ensuring functional dashboards and alerts tailored to k3s.
Adjusting the kube-prometheus-stack Helm chart
Let's first take a look at how to disable the alerts and dashboards that do not work for k3s. The following configuration should be added to your values.yaml file for the kube-prometheus-stack Helm chart:
grafana:
# The default dashboards are not working for `k3s`, so we disable them.
defaultDashboardsEnabled: false
defaultRules:
# The default rules are not working for `k3s`, so we disable them.
create: false
# Source for issues/solutions: https://github.com/k3s-io/k3s/issues/3619#issuecomment-1425852034
# `k3s` exposes all metrics combined for each component, so we don't need to scrape them separately
# We'll only scrape kubelet, otherwise we'd get duplicate metrics.
kubelet:
enabled: true
# Kubernetes API server collects data from master nodes, while kubelet collects data from master and worker nodes
# To not duplicate metrics we'll only scrape Kubelet
kubeApiServer: {}
kubeControllerManager: {}
kubeProxy: {}
kubeScheduler: {}
# `k3s` runs SQLite by default and not etcd, so we don't need to scrape etcd.
kubeEtcd: {}
The above configuration disables the default dashboards and rules while ensuring only the kubelet metrics are scraped. This adjustment is necessary because k3s combines all metrics for its components into a single endpoint. Scraping these metrics separately would result in duplicates, leading to inconsistencies in your monitoring data.
Configuring the kubernetes-mixin
The kube-prometheus-stack Helm chart uses the kube-prometheus project as a baseline for the Helm chart. The kube-prometheus project is a collection of Kubernetes manifests, Grafana dashboards and Prometheus rules combined with Jsonnet libraries to generate them. The kube-prometheus project uses monitoring mixins to generate alerts and dashboards. Monitoring mixins are a collection of Jsonnet libraries that generate dashboards and alerts for Kubernetes. The kubernetes-mixin is a mixin that generates dashboards and alerts for Kubernetes. The node-exporter, coredns, grafana, prometheus and prometheus-operator mixins are also used to generate dashboards and alerts for the Kubernetes cluster.
Since we disable all the alerts and dashboard for the kube-prometheus-stack Helm chart we need to generate the dashboards ourselves by using jsonnet. Let's first install jsonnet and the jsonnet-bundler to initialize a new Jsonnet project:
go install github.com/google/go-jsonnet/cmd/jsonnet@latest
go install github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest
Next, we'll initialize a new Jsonnet project:
jb init
This will create a jsonnetfile.json file. Now we'll install the mixins:
jb install github.com/kubernetes-monitoring/kubernetes-mixin@master
jb install github.com/prometheus-operator/kube-prometheus/jsonnet/kube-prometheus@main
jb install github.com/povilasv/coredns-mixin@master # Install the core-dns mixin if you use CoreDNS
This will create a jsonnetfile.lock.json file and a vendor directory. The other mixins will be pulled in from the kubernetes-mixin mixin since they are dependencies. We install the kube-prometheus project since we use helper functions from the project to generate the dashboards and alerts.
Now we'll create a new file called main.jsonnet and add the following content:
# We use helper functions from kube-prometheus to generate dashboards and alerts for Kubernetes.
local addMixin = (import 'kube-prometheus/lib/mixin.libsonnet');
local kubernetesMixin = addMixin({
name: 'kubernetes',
dashboardFolder: 'Kubernetes',
mixin: (import 'kubernetes-mixin/mixin.libsonnet') + {
_config+:: {
cadvisorSelector: 'job="kubelet"',
kubeletSelector: 'job="kubelet"',
kubeSchedulerSelector: 'job="kubelet"',
kubeControllerManagerSelector: 'job="kubelet"',
kubeApiserverSelector: 'job="kubelet"',
kubeProxySelector: 'job="kubelet"',
},
},
});
local nodeExporterMixin = addMixin({
name: 'node-exporter',
dashboardFolder: 'General',
mixin: (import 'node-mixin/mixin.libsonnet') + {
_config+:: {},
},
});
local corednsMixin = addMixin({
name: 'coredns',
dashboardFolder: 'DNS',
mixin: (import 'coredns-mixin/mixin.libsonnet') + {
_config+:: {
corednsSelector: 'job="coredns"',
},
},
});
local grafanaMixin = addMixin({
name: 'grafana',
dashboardFolder: 'Grafana',
mixin: (import 'grafana-mixin/mixin.libsonnet') + {
_config+:: {},
},
});
local prometheusMixin = addMixin({
name: 'prometheus',
dashboardFolder: 'Prometheus',
mixin: (import 'prometheus/mixin.libsonnet') + {
_config+:: {},
},
});
local prometheusOperatorMixin = addMixin({
name: 'prometheus-operator',
dashboardFolder: 'Prometheus Operator',
mixin: (import 'prometheus-operator-mixin/mixin.libsonnet') + {
_config+:: {},
},
});
local stripJsonExtension(name) =
local extensionIndex = std.findSubstr('.json', name);
local n = if std.length(extensionIndex) < 1 then name else std.substr(name, 0, extensionIndex[0]);
n;
local grafanaDashboardConfigMap(folder, name, json) = {
apiVersion: 'v1',
kind: 'ConfigMap',
metadata: {
name: 'grafana-dashboard-%s' % stripJsonExtension(name),
namespace: 'monitoring',
labels: {
grafana_dashboard: '1',
},
},
data: {
[name]: std.manifestJsonEx(json, ' '),
},
};
local generateGrafanaDashboardConfigMaps(mixin) = if std.objectHas(mixin, 'grafanaDashboards') && mixin.grafanaDashboards != null then {
['grafana-dashboard-' + stripJsonExtension(name)]: grafanaDashboardConfigMap(folder, name, mixin.grafanaDashboards[folder][name])
for folder in std.objectFields(mixin.grafanaDashboards)
for name in std.objectFields(mixin.grafanaDashboards[folder])
} else {};
local nodeExporterMixinHelmGrafanaDashboards = generateGrafanaDashboardConfigMaps(nodeExporterMixin);
local kubernetesMixinHelmGrafanaDashboards = generateGrafanaDashboardConfigMaps(kubernetesMixin);
local corednsMixinHelmGrafanaDashboards = generateGrafanaDashboardConfigMaps(corednsMixin);
local grafanaMixinHelmGrafanaDashboards = generateGrafanaDashboardConfigMaps(grafanaMixin);
local prometheusMixinHelmGrafanaDashboards = generateGrafanaDashboardConfigMaps(prometheusMixin);
local prometheusOperatorMixinHelmGrafanaDashboards = generateGrafanaDashboardConfigMaps(prometheusOperatorMixin);
local grafanaDashboards =
kubernetesMixinHelmGrafanaDashboards +
nodeExporterMixinHelmGrafanaDashboards +
corednsMixinHelmGrafanaDashboards +
grafanaMixinHelmGrafanaDashboards +
prometheusMixinHelmGrafanaDashboards +
prometheusOperatorMixinHelmGrafanaDashboards;
local prometheusAlerts = {
'kubernetes-mixin-rules': kubernetesMixin.prometheusRules,
'node-exporter-mixin-rules': nodeExporterMixin.prometheusRules,
'coredns-mixin-rules': corednsMixin.prometheusRules,
'grafana-mixin-rules': grafanaMixin.prometheusRules,
'prometheus-mixin-rules': prometheusMixin.prometheusRules,
'prometheus-operator-mixin-rules': prometheusOperatorMixin.prometheusRules,
};
grafanaDashboards + prometheusAlerts
This file will generate the dashboards for Grafana and alerts for Prometheus. We use the addMixin helper function which is a helper function from the kube-prometheus project to generate the dashboards and alerts. The addMixin function accepts a mixin as an argument and generates a new object with dashboards and alerts included. The alerts are created as CustomResourceDefinition (CRD) PrometheusRules, making them compatible with Prometheus, while the dashboards are output as json files. We need to convert the dashboards to ConfigMap objects to be able to use them in Kubernetes. The grafanaDashboardConfigMap function takes a folder, name and json as arguments and returns a ConfigMap object.
For the kubernetes-mixin we use the job selector kubelet for all components. This is because the k3s kubelet exposes all metrics combined for each component, so we don't need to scrape them separately. This is necessary, otherwise dashboard and alerts will not work as expected. Now you can run the following command to generate the dashboards and alerts:
jsonnet main.jsonnet -J vendor > generated.json
This will generate a generated.json file with the dashboards and alerts. However, we want to output the generated json to individual files and also convert the files to yaml for better readability. Lastly, we'll need to escape brackets in the yaml files similar to how the kube-prometheus-stack Helm chart does it.
First, install gojsontoyaml:
go install github.com/brancz/gojsontoyaml@latest
Now, let's create a new file called generate.sh and add the following content:
#!/bin/bash
set -e # Exit on any error
set -u # Treat unset variables as an error
# Define paths
MIXINS_DIR="./templates"
# Function to escape YAML content
escape_yaml() {
local file_path="$1"
echo "Escaping $file_path..."
# Read the file content, process, and overwrite it
sed -i \
-e 's/{{/{{`{{/g' \
-e 's/}}/}}`}}/g' \
-e 's/{{`{{/{{`{{`}}/g' \
-e 's/}}`}}/{{`}}`}}/g' \
"$file_path"
echo "Escaped $file_path."
}
# Clean the templates directory
echo "Cleaning templates directory..."
rm -rf ${MIXINS_DIR}/*
echo "Templates directory cleaned."
# Convert Jsonnet to YAML
echo "Converting Jsonnet to YAML..."
jsonnet main.jsonnet -J vendor -m ${MIXINS_DIR} | xargs -I{} sh -c 'cat {} | gojsontoyaml > {}.yaml' -- {}
echo "Jsonnet conversion completed."
# Remove all non-YAML files
echo "Removing non-YAML files..."
find ${MIXINS_DIR} -type f ! -name "*.yaml" -exec rm {} +
echo "Non-YAML files removed."
# Escape YAML files
echo "Escaping YAML files..."
find ${MIXINS_DIR} -name '*.yaml' | while read -r file; do
escape_yaml "$file"
done
echo "YAML files escaped."
echo "Processing completed successfully!"
This script will generate the dashboards and alerts and convert them to yaml files. It will also escape the brackets in the yaml files. Now you can run the following command to generate the dashboards and alerts:
chmod +x generate.sh
./generate.sh
This will generate the dashboards and alerts in the templates directory. You can now apply the dashboards and alerts to your Kubernetes cluster by running the following command:
kubectl apply -f templates
Now, all alerts and dashboards will work as expected in k3s environments. You can now access the dashboards in Grafana and see the alerts in Prometheus.
Summary
In this blog post, we explored how to configure the kube-prometheus-stack Helm chart and the kubernetes-mixin to work out-of-the-box with k3s. We introduced tools like jsonnet, jsonnet-bundler, and monitoring mixins, highlighting how they simplify configuring dashboards and alerts for Kubernetes monitoring.
