The kubernetes-mixin is a popular resource for providing excellent dashboards and alerts for monitoring Kubernetes clusters. However, it lacks comprehensive support for autoscaling components such as Pod Disruption Budgets (PDB), Horizontal and Vertical Pod Autoscalers (HPA, VPA), Karpenter, and the Cluster Autoscaler. These essential components are commonly deployed in Kubernetes environments but do not have standardized, open-source monitoring solutions. This blog post aims to solve that by introducing the kubernetes-autoscaling-mixin - a set of Prometheus rules and Grafana dashboards for Kubernetes autoscaling. This mixin includes coverage for Kubernetes core components such as HPA and PDB, as well as additional components like the VPA, Karpenter and the Cluster Autoscaler, ensuring you have visibility into the autoscaling mechanisms that keep your cluster running efficiently.

Note: given that the mixin includes additional components such as VPA, Karpenter, and the Cluster Autoscaler, I chose to create a separate repository rather than contribute to the Kubernetes-mixin. The Kubernetes-mixin is primarily focused on core Kubernetes components, and I was uncertain if non-core components like Karpenter, the Cluster Autoscaler, and VPA should be part of that mixin. However, this could change in the future, and the dashboards and alerts might eventually be merged into the Kubernetes-mixin.

The mixin can be found on GitHub. The dashboards are the following:

• Kubernetes Autoscaling Horizontal Pod Autoscaler - A dashboard that provides an overview of the Horizontal Pod Autoscaler (HPA) in your Kubernetes cluster.
• Kubernetes Autoscaling Pod Disruption Budget - A dashboard that provides an overview of the Pod Disruption Budget (PDB) in your Kubernetes cluster.
• Kubernetes Autoscaling Vertical Pod Autoscaler - A dashboard that provides an overview of the Vertical Pod Autoscaler (VPA) in your Kubernetes cluster.
• Kubernetes Autoscaling Cluster Autoscaler - A dashboard that provides an overview of the Cluster Autoscaler in your Kubernetes cluster.
• Kubernetes Autoscaling Karpenter Overview - A dashboard that provides an overview of Karpenter (node pools, instance types, node counts etc.) in your Kubernetes cluster.
• Kubernetes Autoscaling Karpenter Activity - A dashboard that provides an overview of Karpenter's activity (scaling up and down) in your Kubernetes cluster.
• Kubernetes Autoscaling Karpenter Performance - A dashboard that provides an overview of Karpenter's performance in your Kubernetes cluster. Shows an overview of the interruption queue, work queue and the controller.

The dashboards and alerts will be described in more detail in the upcoming sections. The prerequisites for non-core components in the mixin are also described.

Prerequisites

The core components as HPA and PDB have built-in metrics from kube-state-metrics, however, the additional components as VPA, Karpenter, and the Cluster Autoscaler require additional configuration. Here are the requirements for each component:

• Karpenter requires you to have a service monitor or a scrape configuration for the Karpenter metrics.
• For the Cluster Autoscaler, you need to have a service monitor or a scrape configuration for the Cluster Autoscaler metrics.
• The Vertical Pod Autoscaler requires you to configure CustomResourceStateMetrics according to the VPA requirements in the repository README. I've also written a blog post on how to set up VPA recommendation metrics here.

Grafana Dashboards

Kubernetes Core

Horizontal Pod Autoscaler

The Grafana dashboard provides an overview of the Horizontal Pod Autoscalers in your Kubernetes cluster. It includes the following panels:

• Filters - Allows us to filter by namespace, HPA and metric name (e.g. memory or CPU).
• Summary - Provides an overview of the HPA. Also, there are time series graphs of the utilization and threshold, as well as the current, available, minimum and maximum replicas. Time series graphs display utilization versus threshold, as well as replica counts (current, available, minimum, and maximum)."

Pod Disruption Budget

The Grafana dashboard provides an overview of the Pod Disruption Budgets in your Kubernetes cluster. It includes the following panels:

• Filters - Allows us to filter by namespace and PDB.
• Namespace summary - Provides an overview of Pod Disruption Budgets (PDBs) in the selected namespace, showing allowed disruptions along with the current, expected, and desired pod counts for each PDB.
• PDB summary - Displays an overview of the selected Pod Disruption Budget (PDB), including allowed disruptions and current, expected, and desired pod counts. Also provides a historical time series graph for disruptions allowed and pod counts (current, expected, and desired).

Karpenter

Karpenter Overview

The Grafana dashboard provides an overview of Karpenter in your Kubernetes cluster. It includes the following panels:

• Filters - Allows us to filter by namespace and Karpenter controller. The filters also allow us to break down the node pools by region, zone, architecture, OS, instance type and capacity type.
• Node pool summary - Provides an overview of the node pools. The node pool count and the usage and limits of the node pools. It also provides a summary of the node pools by region, zone, architecture, OS, instance type and capacity type.
• Pod summary - Provides an overview of pod usage and limits, along with a summary by node pool, instance type, and capacity type.
• Node pools - Displays a table of the node pools and their characteristics.
• Nodes - Displays a table of the nodes and their characteristics.

Karpenter Activity

The Grafana dashboard offers an overview of node pool status (disruptions and scaling) and pod activity (phases and startup times) in your Kubernetes cluster. It includes the following panels:

• Node pool activity - Provides the activity of the node pools - the amount of disruptions and scaling events and the reasoning behind them.
• Pod activity - Displays pod activity, including time series for pod phases and startup durations.

Karpenter Performance

The Grafana dashboard provides an overview of the Karpenter's performance in your Kubernetes cluster. It includes the following panels:

• Summary - Summarizes Karpenter performance, displaying cluster sync status, total node count, cloud provider errors, node termination duration, and pod startup duration.
• Interruption queue - Provides insights to the interruption queue. It shows the received messages, the deleted messages and the interruption duration.
• Work queue - Visualizes work queue depth along with queuing and processing durations.
• Controller - Summarizes the controller's reconciliation requests per second, categorized by request type.

Cluster Autoscaler

The Grafana dashboard provides an overview of the Cluster Autoscaler in your Kubernetes cluster. It includes the following panels:

• Summary - Summarizes Cluster Autoscaler metrics, including total and maximum node counts, node group count, and unschedulable pod count. Time series visualizations show unschedulable pods, node activity (ready, unready, unregistered), and autoscaling events (unneeded nodes, total nodes, and scaled-up nodes).

Vertical Pod Autoscaler

The Grafana dashboard provides an overview of the VPA recommendations for both memory and CPU. It includes the following panels:

• Namespace Summary - Provides an overview of the VPA recommendations per namespace. See the memory and CPU target and lower and upper bounds for each VPA in the selected namespace.
• VPA Summary - Provides a history of recommendations for the selected VPA. See the historical memory and CPU target and lower and upper bounds for each container in the selected VPA. Also, it provides a summary for what resource configuration would be required for guaranteed and burstable QoS classes.

Alerts

Alerts are trickier to get right for a generic use case, however they are still provided by the kubernetes-autoscaling-mixin. You can configure alerts using the config.libsonnet file in the repository. If you're familiar with Jsonnet, customizing these alerts is straightforward. The alerts can be found on GitHub, and I'll add a description for the alerts below.

Note: These alerts are a work in progress. Please open issues for any suggestions or improvements.

Karpenter Alerts

• Alert name: KarpenterCloudProviderErrors

Alerts when Karpenter has had cloud provider errors in the last 5 minutes.

• Alert name: KarpenterNodepoolNearCapacity

Alerts when a Karpenter node pool is near capacity in the last 15 minutes. The current threshold is 75% of the limit. This indicates that the node pool limits need to be scaled.

Cluster Autoscaler Alerts

• Alert name: ClusterAutoscalerNodeCountNearCapacity

Triggers an alert when the Cluster Autoscaler node count reaches 75% of maximum capacity over the last 15 minutes. This indicates that the node group may need scaling to increase the maximum node limit.

• Alert name: ClusterAutoscalerUnschedulablePods

Alerts when the Cluster Autoscaler detects unschedulable pods within the last 15 minutes. This typically indicates that node groups with their taints or label selectors do not match the pod requirements, or that there are insufficient resources available in the cluster.

Summary

The kubernetes-mixin is great but lacks support for autoscaling components such as HPA, PDB, VPA, Karpenter and the Cluster Autoscaler. The kubernetes-autoscaling-mixin aims to solve that by providing dashboards and alerts for these components. While this mixin provides a solid foundation, it's only the first iteration, and there is plenty of room for improvement. Therefore, I would love to get feedback and suggestions on how to improve the kubernetes-autoscaling-mixin. Please open issues in the repository if you have any suggestions or improvements. The goal is to eventually standardize Kubernetes autoscaling dashboards and alerts. Looking forward to collaborating!