5 min read
The Prometheus package for Django provides a great Prometheus integration, but the open source dashboards and alerts that exist are not that great. The to-go Grafana dashboard does not use a large portion of metrics provided by the Django-Prometheus package, alongside this there are no filters for views, methods, jobs and namespaces. This blog post will introduce the Django-mixin - a set of Prometheus rules and Grafana dashboards for Django. The dashboard and alerts will provide insights on applied/unapplied migrations, RED (requests per second, error percentage of the request, latency for each request) metrics, database ops and cache hit rate.
There are already three dashboards that are published in Grafana:
There are also Prometheus alerts stored in GitHub that you can import that cover RED metrics and database errors and missing migrations.
The dashboards and alerts are work in progress and feel free to share feedback in the Django-mixin repository of what you would like to see or any issues you experience.
If you want to go directly to the dashboards you can use the links above, the rest of the blog post will describe setting up Django-prometheus and the various alerts and dashboards.
First, install Django-Prometheus (pip/poetry):
poetry/pip add/install django-prometheus
All the following settings should be located in your settings.py
. Add django_prometheus
to installed apps:
INSTALLED_APPS = [
...
"django_prometheus",
...
]
Add the Django-prometheus request middleware to middleware, ensure that it is before and after all other middleware:
MIDDLEWARE = [
"django_prometheus.middleware.PrometheusBeforeMiddleware",
... My other middleware
"django_prometheus.middleware.PrometheusAfterMiddleware",
]
Change the database and cache backend to use the Django-prometheus backend:
DATABASES["default"]["ENGINE"] = "django_prometheus.db.backends.postgresql" # Adjust according to your database
CACHES = {
"default": {
"BACKEND": "django_prometheus.cache.backends.redis.RedisCache",
...
}
}
Ensure we enable migration metrics:
PROMETHEUS_EXPORT_MIGRATIONS = env.bool("PROMETHEUS_EXPORT_MIGRATIONS", True)
Lastly, add the Django-prometheus URLs:
urlpatterns = [
...
path("prometheus/", include("django_prometheus.urls")),
...
]
Now you should be able to go to <my-url>/prometheus/metrics
and see database, cache and request metrics! Add the target to Prometheus using your preferred approach and move on to the dashboard sections below.
As mentioned previously the Django-mixin has three dashboards. A Django overview, a Django request overview and a Django request breakdown by view. The dashboards are split as otherwise there would be many graphs in one dashboard, filters would be applicable for a portion of the panels as not all metrics contain the filtered labels making it unclear when they apply and some expensive metrics would put high pressure on your Prometheus backend.
The upcoming sections will describe each dashboard.
The Django overview dashboard focuses on providing an overview of your entire system with both the cache and database included. The following things are core for the dashboard:
The Django requests overview focuses on providing an overview of just requests to your system. The following things are core for the dashboard:
The Django requests by view focuses on providing a breakdown of specific views and visualizing the more expensive metrics such as request latency. The following things are core for the dashboard:
Alerts are trickier to get right for a generic use case, however they are still provided by the Django-mixin. They are also configurable with the config.libsonnet
package in the repository, if you are familiar with Jsonnet then customizing the alerts should be fairly straight forward. The alerts can be found on GitHub and I'll add a description for the alerts below.
DjangoMigrationsUnapplied
Alerts when Django has unapplied migrations for longer than 15 minutes, indicating that a new rollout was finished, but migrations have not been run.
DjangoDatabaseExceptions
Alerts when Django has hit database exceptions the last 10 minutes.
DjangoHighHttp4xxErrorRate
Alerts when more than 5% HTTP requests with status 4xx for a specific view the past 5 minutes.
DjangoHighHttp5xxErrorRate
Alerts when more than 5% HTTP requests with status 5xx for a specific view the past 5 minutes.
Adjust these and add any new ones that you require!
Django-prometheus is a great library and Grafana and Prometheus are amazing open source tools for monitoring purposes. The dashboard and alerts presented in this blog post should be easy to reuse and extend if needed. I think they set a good basis for monitoring, but they can be improved and adjusted, therefore it would be great if you have any suggestions that you open issues in the Django-mixin GitHub repository. Looking for any input to hopefully standardize dashboards and alerts for Django over time!
5 min read
Celery is a python project used for asynchronous job processing and task scheduling in web applications or distributed systems. It is very commonly used together with Django, Celery as the asynchronous job processor and Django …
4 min read
Apollo GraphQL and NestJS are gaining traction quickly, however the monitoring approaches are unclear. At the moment (late 2021 / early 2022) there are no default exporters or libraries for Prometheus metrics and the same …
4 min read
RabbitMQ has a native built-in Prometheus plugin and by default it has granular metrics disabled. Granular metrics means per-queue/vhost metrics - detailed metrics that provide message lag and consumer info on a `queue` and `vhost` …