dancrew32 / ab

A/B Testing Framework for Python (with optional Multi-armed bandit implementation)

A/B Testing for Python

Without the need for a database, deterministically bucket users into A/B tests.

Build Status:

Install:

pip install ab

Thanks to Delvian for graciously donating the pip module name at https://pypi.org/project/ab/!

Example A/B Usage:

from ab import ab

# Define test & buckets
TEST_NAME = 'MY_TEST_V1'
TEST_BUCKET_TO_COLOR = {
    'control': 'green',
    'variant1': 'red',
    'variant2': 'blue',
}


# Implemention
def get_button_color(user_id):
    buckets = TEST_BUCKET_TO_COLOR.keys()
    bucket = ab.get_bucket(user_id, test=TEST_NAME, buckets=buckets)
    return TEST_BUCKET_TO_COLOR[bucket]

Thanks to Alexander Ejbekov for the allocation technique:

https://stackoverflow.com/a/23846715/61410

Example Multi-Armed Bandit Usage:

https://en.wikipedia.org/wiki/Multi-armed bandit

from ab import mab

# Define test & buckets
TEST_NAME = 'MY_TEST_V2'
TEST_BUCKET_TO_COLOR = {
    'control': 'green',
    'variant1': 'red',
    'variant2': 'blue',
}


# Implemention
def get_button_color():
    buckets = TEST_BUCKET_TO_COLOR.keys()
    bucket = mab.get_bucket(test=TEST_NAME, buckets=buckets)
    return TEST_BUCKET_TO_COLOR[bucket]


# Record success
def button_clicked(bucket):
    mab.success(test=TEST_NAME, bucket=bucket)

Demo MAB app:

You'll need Docker running and docker-compose.

• Install Docker: https://docs.docker.com/install/
• Install docker-compose: https://docs.docker.com/compose/install/

git clone https://github.com/dancrew32/ab.git ab
cd ab
make up

Then visit http://localhost:5000

Development

Ensure you have python3 and virtualenv installed:

sudo apt install python3.7 python3-venv python3.7-venv

Then make the virtualenv, install any dependencies (there aren't any at the moment), and run the unit tests.

make venv deps test

Publish new version to PyPI

make setup