The kube-prometheus-stack Helm chart, which deploys the kubernetes-mixin, targets standard Kubernetes setups, often pre-configured for specific cloud environments. However, these configurations aren’t directly compatible with k3s, a lightweight Kubernetes distribution. Since k3s lacks many of the default cloud integrations, issues arise, such as missing metrics, broken graphs, and unavailable endpoints (example issue). This blog post guides you through adapting the kube-prometheus-stack Helm chart and the kubernetes-mixin to work seamlessly in k3s environments, ensuring functional dashboards and alerts tailored to k3s.
Adjusting the kube-prometheus-stack Helm chart
First, take a look at how to turn off the alerts and dashboards that don’t work for k3s. Add the following configuration to your values.yaml file for the kube-prometheus-stack Helm chart:
grafana:
# The default dashboards are not working for `k3s`, so we disable them.
defaultDashboardsEnabled: false
defaultRules:
# The default rules are not working for `k3s`, so we disable them.
create: false
# Source for issues/solutions: https://github.com/k3s-io/k3s/issues/3619#issuecomment-1425852034
# `k3s` exposes all metrics combined for each component, so we don't need to scrape them separately
# We'll only scrape kubelet, otherwise we'd get duplicate metrics.
kubelet:
enabled: true
# Kubernetes API server collects data from master nodes, while kubelet collects data from master and worker nodes
# To not duplicate metrics we'll only scrape Kubelet
kubeApiServer: {}
kubeControllerManager: {}
kubeProxy: {}
kubeScheduler: {}
# `k3s` runs SQLite by default and not etcd, so we don't need to scrape etcd.
kubeEtcd: {}
The preceding configuration disables the default dashboards and rules while ensuring only the kubelet metrics gets scraped. This adjustment is necessary because k3s combines all metrics for its components into a single endpoint. Scraping these metrics separately would result in duplicates, leading to inconsistencies in your monitoring data.
Configuring the kubernetes-mixin
The kube-prometheus-stack Helm chart uses the kube-prometheus project as a baseline for the Helm chart. The kube-prometheus project is a collection of Kubernetes manifests, Grafana dashboards and Prometheus rules combined with Jsonnet libraries to generate them. The kube-prometheus project uses monitoring mixins to generate alerts and dashboards. Monitoring mixins are a collection of Jsonnet libraries that generate dashboards and alerts for Kubernetes. The kubernetes-mixin is a mixin that generates dashboards and alerts for Kubernetes. The node-exporter, coredns, grafana, prometheus and prometheus-operator mixins are also used to generate dashboards and alerts for the Kubernetes cluster.
Since the kube-prometheus-stack Helm chart with the new values turns off the alerts and dashboard, you need to generate the dashboards yourself by using jsonnet. Install jsonnet and the jsonnet-bundler to initialize a new Jsonnet project:
go install github.com/google/go-jsonnet/cmd/jsonnet@latest
go install github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest
Next, initialize a new Jsonnet project:
jb init
This creates a jsonnetfile.json file. Now install the mixins:
jb install github.com/kubernetes-monitoring/kubernetes-mixin@master
jb install github.com/prometheus-operator/kube-prometheus/jsonnet/kube-prometheus@main
jb install github.com/povilasv/coredns-mixin@master # Install the core-dns mixin if you use CoreDNS
This creates a jsonnetfile.lock.json file and a vendor directory. The other mixins pull in from the kubernetes-mixin mixin since their dependencies. The kube-prometheus project installs since helper functions from the project generate the dashboards and alerts.
Next, create a new file called main.jsonnet and add the following content:
# We use helper functions from kube-prometheus to generate dashboards and alerts for Kubernetes.
local addMixin = (import 'kube-prometheus/lib/mixin.libsonnet');
local kubernetesMixin = addMixin({
name: 'kubernetes',
dashboardFolder: 'Kubernetes',
mixin: (import 'kubernetes-mixin/mixin.libsonnet') + {
_config+:: {
cadvisorSelector: 'job="kubelet"',
kubeletSelector: 'job="kubelet"',
kubeSchedulerSelector: 'job="kubelet"',
kubeControllerManagerSelector: 'job="kubelet"',
kubeApiserverSelector: 'job="kubelet"',
kubeProxySelector: 'job="kubelet"',
},
},
});
local nodeExporterMixin = addMixin({
name: 'node-exporter',
dashboardFolder: 'General',
mixin: (import 'node-mixin/mixin.libsonnet') + {
_config+:: {},
},
});
local corednsMixin = addMixin({
name: 'coredns',
dashboardFolder: 'DNS',
mixin: (import 'coredns-mixin/mixin.libsonnet') + {
_config+:: {
corednsSelector: 'job="coredns"',
},
},
});
local grafanaMixin = addMixin({
name: 'grafana',
dashboardFolder: 'Grafana',
mixin: (import 'grafana-mixin/mixin.libsonnet') + {
_config+:: {},
},
});
local prometheusMixin = addMixin({
name: 'prometheus',
dashboardFolder: 'Prometheus',
mixin: (import 'prometheus/mixin.libsonnet') + {
_config+:: {},
},
});
local prometheusOperatorMixin = addMixin({
name: 'prometheus-operator',
dashboardFolder: 'Prometheus Operator',
mixin: (import 'prometheus-operator-mixin/mixin.libsonnet') + {
_config+:: {},
},
});
local stripJsonExtension(name) =
local extensionIndex = std.findSubstr('.json', name);
local n = if std.length(extensionIndex) < 1 then name else std.substr(name, 0, extensionIndex[0]);
n;
local grafanaDashboardConfigMap(folder, name, json) = {
apiVersion: 'v1',
kind: 'ConfigMap',
metadata: {
name: 'grafana-dashboard-%s' % stripJsonExtension(name),
namespace: 'monitoring',
labels: {
grafana_dashboard: '1',
},
},
data: {
[name]: std.manifestJsonEx(json, ' '),
},
};
local generateGrafanaDashboardConfigMaps(mixin) = if std.objectHas(mixin, 'grafanaDashboards') && mixin.grafanaDashboards != null then {
['grafana-dashboard-' + stripJsonExtension(name)]: grafanaDashboardConfigMap(folder, name, mixin.grafanaDashboards[folder][name])
for folder in std.objectFields(mixin.grafanaDashboards)
for name in std.objectFields(mixin.grafanaDashboards[folder])
} else {};
local nodeExporterMixinHelmGrafanaDashboards = generateGrafanaDashboardConfigMaps(nodeExporterMixin);
local kubernetesMixinHelmGrafanaDashboards = generateGrafanaDashboardConfigMaps(kubernetesMixin);
local corednsMixinHelmGrafanaDashboards = generateGrafanaDashboardConfigMaps(corednsMixin);
local grafanaMixinHelmGrafanaDashboards = generateGrafanaDashboardConfigMaps(grafanaMixin);
local prometheusMixinHelmGrafanaDashboards = generateGrafanaDashboardConfigMaps(prometheusMixin);
local prometheusOperatorMixinHelmGrafanaDashboards = generateGrafanaDashboardConfigMaps(prometheusOperatorMixin);
local grafanaDashboards =
kubernetesMixinHelmGrafanaDashboards +
nodeExporterMixinHelmGrafanaDashboards +
corednsMixinHelmGrafanaDashboards +
grafanaMixinHelmGrafanaDashboards +
prometheusMixinHelmGrafanaDashboards +
prometheusOperatorMixinHelmGrafanaDashboards;
local prometheusAlerts = {
'kubernetes-mixin-rules': kubernetesMixin.prometheusRules,
'node-exporter-mixin-rules': nodeExporterMixin.prometheusRules,
'coredns-mixin-rules': corednsMixin.prometheusRules,
'grafana-mixin-rules': grafanaMixin.prometheusRules,
'prometheus-mixin-rules': prometheusMixin.prometheusRules,
'prometheus-operator-mixin-rules': prometheusOperatorMixin.prometheusRules,
};
grafanaDashboards + prometheusAlerts
This file generates the dashboards for Grafana and alerts for Prometheus. The addMixin helper function from the kube-prometheus project generates the dashboards and alerts. The addMixin function accepts a mixin as an argument and generates a new object with dashboards and alerts included. The alerts appear as CustomResourceDefinition (CRD) PrometheusRules, making them compatible with Prometheus, while the dashboards output as json files. Convert the dashboards to ConfigMap objects to use them in Kubernetes. The grafanaDashboardConfigMap function takes a folder, name, and json as arguments and returns a ConfigMap object.
For the kubernetes-mixin, the job selector kubelet applies to all components. This is because the k3s kubelet exposes all metrics combined for each component, so there is no need to scrape them separately. This is necessary; otherwise, dashboard, and alerts won’t work as expected. Now, run the following command to generate the dashboards and alerts:
jsonnet main.jsonnet -J vendor > generated.json
This generates a generated.json file with the dashboards and alerts. However, the goal is to output the generated json to individual files and also convert the files to yaml for better readability. Lastly, it’s necessary to escape brackets in the yaml files similar to how the kube-prometheus-stack Helm chart does it.
First, install gojsontoyaml:
go install github.com/brancz/gojsontoyaml@latest
Now, create a new file called generate.sh and add the following content:
#!/bin/bash
set -e # Exit on any error
set -u # Treat unset variables as an error
# Define paths
MIXINS_DIR="./templates"
# Function to escape YAML content
escape_yaml() {
local file_path="$1"
echo "Escaping $file_path..."
# Read the file content, process, and overwrite it
sed -i \
-e 's/{{/{{`{{/g' \
-e 's/}}/}}`}}/g' \
-e 's/{{`{{/{{`{{`}}/g' \
-e 's/}}`}}/{{`}}`}}/g' \
"$file_path"
echo "Escaped $file_path."
}
# Clean the templates directory
echo "Cleaning templates directory..."
rm -rf ${MIXINS_DIR}/*
echo "Templates directory cleaned."
# Convert Jsonnet to YAML
echo "Converting Jsonnet to YAML..."
jsonnet main.jsonnet -J vendor -m ${MIXINS_DIR} | xargs -I{} sh -c 'cat {} | gojsontoyaml > {}.yaml' -- {}
echo "Jsonnet conversion completed."
# Remove all non-YAML files
echo "Removing non-YAML files..."
find ${MIXINS_DIR} -type f ! -name "*.yaml" -exec rm {} +
echo "Non-YAML files removed."
# Escape YAML files
echo "Escaping YAML files..."
find ${MIXINS_DIR} -name '*.yaml' | while read -r file; do
escape_yaml "$file"
done
echo "YAML files escaped."
echo "Processing completed successfully!"
This script generates the dashboards and alerts and convert them to yaml files. It also escapes the brackets in the yaml files. Now you can run the following command to generate the dashboards and alerts:
chmod +x generate.sh
./generate.sh
This generates the dashboards and alerts in the templates directory. You can now apply the dashboards and alerts to your Kubernetes cluster by running the following command:
kubectl apply -f templates
Now, all alerts and dashboards work as expected in k3s environments. You can now access the dashboards in Grafana and see the alerts in Prometheus.
Summary
In this blog post, this guide explores how to configure the kube-prometheus-stack Helm chart and the kubernetes-mixin to work out-of-the-box with k3s. Tools like jsonnet, jsonnet-bundler, and monitoring mixins are introduced, highlighting how they simplify configuring dashboards and alerts for Kubernetes monitoring.
