Blackbox-exporter is a Prometheus exporter that probes endpoints and exposes metrics of the probe result. There are multiple guides on how to use the Blackbox-exporter, and we won't go into that, but rather focus on newer things as Jsonnet as a templating language, the Kubernetes CustomResourceDefinition that comes with the Prometheus-operator and the open source monitoring mixin for the exporter. The goal of this post is to showcase the cool nits of Jsonnet and open source libraries to minimize boilerplate and utilize best practices and shared standards.

The blog post uses the following projects, which are dependencies for the showcase in the post:

• Prometheus-operator - The operator adds a CustomResourceDefinition to configure the Blackbox-exporter and manages and reloads the config.
• Kube-prometheus - Deploys the Prometheus-operator and the Blackbox-exporter, and has Jsonnet libraries. It's my to-go way of deploying and configuring a Prometheus/Grafana stack.
• Prometheus-operator-libsonnet - A Jsonnet library for the Prometheus-operator's CustomResourceDefinitions.
• Blackbox-exporter-mixin - A set of Grafana dashboards and Prometheus rules written using Jsonnet and packaged as an open source project.

Prometheus-operator's CustomResourceDefinition Probe

The Prometheus-operator has support for the Probe CustomResourceDefinition which will configure Blackbox-exporter's targets. Previously, you needed to manually deploy and change the config for the Blackbox-exporter for each probe. Now, the kube-prometheus project deploys the Blackbox-exporter by default and it is easily configurable using the CustomResourceDefinition Probe.

Let's take a look at a sample probe:

apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
  labels:
    blackbox-exporter/module: http_2xx
  name: hodovi-cc
  namespace: monitoring
spec:
  interval: 60s
  jobName: blackbox-exporter
  module: http_2xx
  prober:
    url: blackbox-exporter.monitoring.svc.cluster.local:19115
  targets:
    staticConfig:
      static:
        - https://hodovi.cc/health/

Creating the CustomResourceDefinition in your cluster will cause the Prometheus-operator to hot reload the Blackbox-exporter config.

We can simplify the config above by using the jsonnet library for the Prometheus-operator. Install it with jsonnet-bundler.

jb install github.com/jsonnet-libs/prometheus-operator-libsonnet@main

Now we can use the library to create probes with less boilerplate by using the prometheusOperator.monitoring.v1.probe variable. This will also ensure syntax correctness, since we are using functions from the library to configure the various fields.

{
    hodoviCcProbe:
        probe.new('hodovi-cc') +
        probe.metadata.withLabels({
          'blackbox-exporter/module': 'http_2xx',
        }) +
        probe.metadata.withNamespace('monitring') +
        probe.spec.withInterval('60s') +
        probe.spec.withModule('http_2xx') +
        probe.spec.withJobName('blackbox-exporter') +
        probe.spec.prober.withUrl('blackbox-exporter.monitoring.svc.cluster.local:19115') +
        probe.spec.targets.staticConfig.withStatic('https://hodovi.cc/health/'),
}

We'll also create our own function called createProbe to remove even more boilerplate.

local prometheusOperator = import 'github.com/jsonnet-libs/prometheus-operator-libsonnet/0.66/main.libsonnet';
{
  local probe = prometheusOperator.monitoring.v1.probe,

  local createProbe(name, module='http_2xx', namespace='monitoring', staticTargets=[], interval='60s', proberUrl='blackbox-exporter.%s.svc.cluster.local:19115' % namespace) =
    probe.new(name) +
    probe.metadata.withLabels({
      'blackbox-exporter/module': module,
    }) +
    probe.metadata.withNamespace(namespace) +
    probe.spec.withInterval(interval) +
    probe.spec.withModule(module) +
    probe.spec.withJobName('blackbox-exporter') +
    probe.spec.prober.withUrl(proberUrl) +
    probe.spec.targets.staticConfig.withStatic(staticTargets),

Now we can create probes using the createProbe function.

{
  ...
  probes: [
    createProbe('hodovi-cc', staticTargets=['https://hodovi.cc/health/']),
    createProbe('hodovi-cc-400', staticTargets=['https://hodovi.cc/400/']),
    createProbe('hodovi-cc-403', staticTargets=['https://hodovi.cc/403/']),
    createProbe('honeylogic-io', staticTargets=['https://honeylogic.io']),
    createProbe('findwork-dev', staticTargets=['https://findwork.dev/health/']),
    createProbe('findwork-dev-pricing', staticTargets=['https://findwork.dev/pricing']),
    ...
  ],
  ...

There it is 6 YAML files with 20 lines of Jsonnet code with the help of open source projects!

Blackbox-exporter Mixin

Monitoring-mixins are composed of open source Grafana dashboards and Prometheus rules. The maintainers of projects set guidelines on what to monitor and alert on, and you just extend them and adjust them to your needs easily using Jsonnet. There's a Blackbox-exporter mixin that has pre-configured Grafana dashboards and Prometheus alerts. Using a mixin is minimal work once you have your Jsonnet project setup. It also ensures that that Grafana dashboards and settings are configured using code rather than the UI, moving more towards a stateless (or completely stateless in my case) Grafana.

Add the mixin with jsonnet-bundler.

jb install github.com/adinhodovic/blackbox-exporter-mixin@main

Instantiate the mixin using addMixin from the kube-prometheus library and configure it according to your preferences.

local addMixin = (import 'kube-prometheus/lib/mixin.libsonnet');

local blackboxExporterMixin = addMixin({
  name: 'blackbox-exporter',
  dashboardFolder: 'Networking',
  mixin: (import 'blackbox-exporter-mixin/mixin.libsonnet') + {
    _config+:: {
      // Used for templating the alerts (dashboard URLs)
      grafanaUrl: 'https://grafana.<my_domain>',
      // The period in days to consider for the uptime evaluation
      uptimePeriodDays: 30,
      // Will alert if below the percentage for the configured uptime period
      uptimeThreshold: 99.9,
    },
  },
});

Add the dashboard to the kube-prometheus Grafana config.

local kp =
  (import '../config.jsonnet') +
  (import 'kube-prometheus/main.libsonnet') +
  (import 'kube-prometheus/addons/all-namespaces.libsonnet') +
  (import 'kube-prometheus/addons/networkpolicies-disabled.libsonnet') +
  (import 'kube-prometheus/addons/pyrra.libsonnet') +
  {
    values: {
      grafana+:: blackboxExporterMixin.grafanaDashboards {
      ...
      }
    }
  }

Instantiate the kube-prometheus components and add the PrometheusRules from the mixin.

{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['blackbox-exporter-' + name]: kp.blackboxExporter[name] for name in std.objectFields(kp.blackboxExporter) } +
// All other components
...

// Add the mixin
{ 'blackbox-exporter-prometheus-rules': blackboxExporterMixin.prometheusRules }

And the following alerts will be automatically added.

• Alert name: BlackboxProbeFailed

Alerts when a probe fails.

• Alert name: BlackboxLowUptime30d

Alerts when uptime of a probe is less than 99.9%, it's just an info alert, so not of high severity.

• Alert name: BlackboxSslCertificateWillExpireSoon

Alerts when an SSL certificate is 3 weeks from expiration.

They are also configurable with the config.libsonnet package in the repository, if you are familiar with Jsonnet then customizing the alerts should be fairly straight forward.

Adjust these and add any new ones that you require!

Summary

This post is very specific and not very beginner-friendly due to the dependency of many projects, however I hope it serves as a nice showcase of the advantages of Jsonnet with the combination of various open source projects that exist. Less code written in-house and more monitoring practices shared and used open source.