a_b-optimize.png

Django A/B testing with Google Optimize

3 weeks ago New! Popular post!
1299 views

8 min read


A/B testing is a great way to decide what path your product should take. Create two variations, collect data points and see which variation is better. It will help you in understanding your users and their needs. Google Optimize is a tool that simplifies A/B testing by managing variant weights, targeting rules, provides analytics and integrates with other Google product as Google Adwords and Google Analytics.

There are two ways to run A/B tests: client-side and server-side. Client-side A/B tests with Google Optimize executes javascript in browser based on the changes you would like (style changes, positioning of elements etc.), it is fairly limited. In server-side A/B tests there are no limitations, you can write any code you'd like which will display different application versions based on the session's active experiment variant.

The Google Optimize basics

When a user visits your site and an experiment is active he will be given an experiment variant which is stored in a cookie called _gaexp. The cookie can have the following content GAX1.2.MD68BlrXQfKh-W8zrkEVPA.18361.1 and it contains multiple values where each value is divided by dots. There are two key values for server-side A/B testing. Firstly, the experiment ID which in the example is MD68BlrXQfKh-W8zrkEVPA. Secondly, the experiment variant for the session which is the last integer, which in this case is 1.

When multiple experiments are active the experiments will be separated by an exclamation marks as in the following _gaexp cookie GAX1.2.3x8_BbSCREyqtWm1H1OUrQ.18166.1!7IXTpXmLRzKwfU-Eilh_0Q.18166.0. You can extract two experiments from the cookie, 3x8_BbSCREyqtWm1H1OUrQ with the variant 1 and 7IXTpXmLRzKwfU-Eilh_0Q with the variant 0.

Usage in Django

We can parse a ga_exp cookie with this code snippet:

ga_exp = self.request.COOKIES.get("_gaexp")

parts = ga_exp.split(".")
experiments_part = ".".join(parts[2:])
experiments = experiments_part.split("!")
for experiment_str in experiments:
    experiment_parts = experiment_str.split(".")
    experiment_id = experiment_parts[0]
    variation_id = int(experiment_parts[2])
    experiment_variations[experiment_id] = variation_id

The snippet separates the cookie by Google Optimize's delimiters which are dots(.) for different values and exclamation marks(!) to separate experiments. We will then store the experiment variant for each experiment in a dict. Then we can render various templates based on which experiment variants are active for the current session:

def get_template_names(self):
    experiments.get("my_experiment_cookie_id", None)
        if variant == "1":
            return ["jobs/xyz_new.html"]
        return ["jobs/xyz_old.html"]

I've created a Django package that will simplify this and which also adds a middleware and experiment management in the Django-admin, which makes it possible to start, stop and pause experiments through the Django admin. It also simplifies local testing of experiment variants. I use the package at Findwork to A/B test UI/UX changes. Next up, I'll walk you through how to setup an experiment in Google Optimize and how to render various application versions based on the session's experiment variant using the package.

Using Django-google-optimize to run A/B tests

To get started with django-google-optimize install the package with pip:

pip install django-google-optimize

Add the application to installed Django applications:

DJANGO_APPS = [
    ...
    "django_google_optimize",
    ...
]

I've added a middleware which makes the experiment variant easily accessible both in templates and views so you'll need to add the middleware in you settings.py:

MIDDLEWARE = [
    ...
    "django_google_optimize.middleware.google_optimize",
    ...
]

Now run the migrations ./manage.py migrate and the setup is complete.

Creating a Google Optimize experiment

Head over to Google Optimize and create your first experiment. By default you'll have one variant which is the original variant and then you can add as many variants as you would like for the A/B test. Each variant will be given an index based on the order you added the variants, the original variant has always the index 0:

Experiment variants

Next we'll set goals for the A/B test but you need to link your Google Analytics property with Google Optimize to set goals and measure the success of your A/B test. If you do not have Google Analytics added to your site, django-analytical provides easy integration. When Google Analytics is linked you can add goals to the experiment which are either system default goals(e.g bounce rate or page views) or your own defined goals. There is a limit of 3 goals, where only one of them is a primary goal and it will determine the success of the experiment, the other two will be visually displayed in the Google Optimize experiment report. All other user defined goals and system goals will be visible in Google Analytics and they will be tied individually to each experiment, this again is only visually displayed and does not determine the success of the experiment.

Below is an experiment with two system goals and a goal I've defined for Findwork which measures how many users have subscribed to job emailing list:

Experiment goals

Variants and goals are required to start the experiment but Google Optimize provides more options as audience targeting (with Google Ads and Google Analytics integration), variant weights, traffic allocation and more. If you are already using Google's products as Google Analytics and Adwords, then Google Optimize is great!

Adding an experiment in the Django-admin

We can now head over to the Django-admin to add our experiment. When adding an experiment only the experiment ID is required and when it is added a middleware will add the experiments and their variants to a context variable called google_optimize. The experiments will be accessible in both views and templates(the request context processor needs to be added for the context to be accessible in templates) in the request.google_optimize.<experiment_id> object. However, referring to the experiment by ID in your Django templates and views will make it hard to grasp what experiment that specific ID refers to. Therefore, you can add aliases for the experiments and variants which will make your code much more readable. Below is an example of the differences:

Without an Google Experiment alias you will have to reference your experiment by ID:

{% if request.google_optimize.3x8_BbSCREyqtWm1H1OUrQ == 0 %}
{% include "jobs/xyz_new.html" %}
{% endif %}

Instead of by experiment alias:

{% if request.google_optimize.redesign_landing_page == 0 %}
{% include "jobs/xyz_new.html" %}
{% endif %}

Without an variant alias you will have to reference your variant by index as:

{% if request.google_optimize.redesign_landing_page == 0 %}
{% include "jobs/xyz_new.html" %}
{% endif %}

Instead of by variant alias:

{% if request.google_optimize.redesign_landing_page == "New Background Color" %}
{% include "jobs/xyz_new.html" %}
{% endif %}

For the examples above the following Google Experiment object has been added:

Google Experiment Object

Local Development

To test the various variants you can either add the _gaexp cookie or manage the active variant in the Django-admin. For each browser you can either via plugins or developer tools adjust the _gaexp cookie and test each variant by changing the variant index in the session's cookie. For example in the cookie GAX1.2.MD68BlrXQfKh-W8zrkEVPA.18361.1 you can adjust the last integer 1 to the variant index you would like to test. Alternatively, django-google-optimize provides for each experiment an experiment cookie object which can be used to set the active variant. In an experiment's Django admin panel you can add a cookie as in the image below:

Experiment cookie

Set it to active and it will override or add the variant chosen, otherwise the session's cookie will be used. The experiment cookie only works in DEBUG mode as Google will set the cookie in production and we would not want to override it.

Usage

As shown previously you can use django-google-optimize in templates as below:

{% if request.google_optimize.redesign_landing_page == "New Background Color" %}
{% include "jobs/xyz_new.html" %}
{% endif %}

To use the package in views and display two different templates based on the experiment variant:

def get_template_names(self):
    variant = request.google_optimize.get("redesign_landing_page", None)
    if variant == "New Background Color":
        return ["jobs/xyz_new.html"]
    return ["jobs/xyz_old.html"]

Adjusting the queryset in a view based on the experiment variant:

def get_queryset(self):
    variant = self.request.google_optimize.get("redesign_landing_page", None)

    if variant == "New Background Color":
        qs = qs.exclude(design__contains="old")

Results

Google Optimize will continuously report results on the experiment with both a probability for a variant to be better and the modelled improvement percentage. The experiment needs to run at least for two weeks for it to report a clear leader and to let you know that you can end the experiment. Although, there will be daily updates on the experiments progress, where you'll have all insights you'll need to end the experiment whenever you would like. At the top of the reporting page there will be a link to a Google Analytics dashboard for the experiment as seen below in the image:

Google Analytics

The final result of the experiment will be displayed as in the image below. The image displays an overview on how well each experiment variant performed in regards to the experiment goals set:

Final report

The above examples was ended early with few sessions which makes it difficult to pinpoint an exact modelled improvement of a variant. If you would like concrete results before concluding an experiment, it needs to run longer than 14 days as in the above examples.

Summary

Google Optimize integrates well with Google Analytics making it easy to analyze each A/B variant in depth. This makes it easy to understand your users and their needs and develop the application according to them. The package Django-google-optimize aims for simplifying running A/B tests with Google Optimize. Also, the Django admin is just amazing and can serve so many purposes, and in this case you can pause, start and remove the experiment directly from the admin panel.

The project can be found on Github.

The package is published on PyPI.

Documentation is available at read the docs.


Similar Blogs

9 months ago
mailgun statuscake terraform cloudflare devops s3 rds django

Kickstarting Infrastructure for Django Applications with Terraform

8 min read

When creating Django applications or using cookiecutters as Django Cookiecutter you will have by default a number of dependencies that will be needed to be created as a S3 bucket, a Postgres Database and a …


8 months ago
cms wagtail headless api django

Recipes when building a headless CMS with Wagtail's API

3 min read

Recently I built a headless CMS using Wagtail's API as a backend with NextJS/React/Redux as a frontend. Building the API I ran into some small issues with Image URL data, the API representation of snippets …