Giters

Alephbet / gimel

Run your own A/B testing backend using AWS Lambda and Redis HyperLogLog

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

experiment dashboard

gingerlime opened this issue · comments

commented

I'd like to build a simple dashboard to display experiment results. Maybe based on http://keen.github.io/dashboards/ ?

open questions:

  • how would the end-user of gimel host it? (Lambda / S3)
  • would it be generated with a gimel command?
  • should this be hosted "globally" to all users and all they need to enter is their API Gateway endpoint + API key to view results?
  • should the dashboard allow you to also filter the experiments to show / delete experiments / hide them?
  • how are end-users authenticated to the dashboard? (using the AWS API key? something else?)
  • what kind of stats should the dashboard show? starting with something simple that just shows raw experiment data is good, but soon enough people would need to be able to "choose the winner"... this opens up a much bigger question of how to analyze results
commented

I've made myself following simple script, it just extracts the data for given experiment and prints conversion rates. For significance calculation I usually use https://vwo.com/downloads/ab_testing_significance_calculator.xls

import requests
from sys import argv

ENDPOINT = None
API_KEY = None

def request_results(experiment):
    res = requests.get(
        ENDPOINT,
        headers={'x-api-key': API_KEY}
    )
    data = res.json()

    for item in data:
        if item['experiment'] != experiment:
            continue
        goals = item['goals']
        for goal in goals:
            print 'Goal %s' % goal['goal']
            for result in goal['results']:
                success = float(result['successes'])
                trials = float(result['trials'])
                conversion = 100 * success / trials
                print 'Variant "%s" - %0.2f (%s, %s)' % (result['label'], conversion, success, trials)

if __name__ == "__main__":
    request_results(argv[1])
commented

Thanks for your input, @314l5926. I'm familiar with the vwo spreadsheet, but I think it's a bit outdated. This calculation approach is probably still valid, but even VWO is now using bayesian statistics they call SmartStats...

I should collect some resources for consideration, but I would really like to have a solid significance calculation built into some kind of dashboard. Maybe the user can even choose between different methodologies / algorithms. Unfortunately whilst I understand the concepts broadly, god is in the detail, so I'm a bit hesitant to implement anything without more solid statistics knoweldge / feedback / input.

It's a bit of a chicken&egg with the dashboard - having an awesome looking dashboard would be convincing to get contributions from someone with statistics knowledge (I hope), but the awesomeness of the dashboard very much lies in how accurate picture it gives you, to get the confidence you need...

commented

It would be quite easy to implement is https://en.wikipedia.org/wiki/Multi-armed_bandit if you want to optimize winner selection process

commented

Also, bayesian ab testing doesn't look much harder to do, if you look at http://www.evanmiller.org/bayesian-ab-testing.html
E.g. https://gist.github.com/arnov/60de0b1ad62d329bc222

commented

314l5926@2eedcc7

I think I would be tracking experiment conversions and winning probabilities every hour or so, and after reaching given probability, the winner would be chosen, manually.

commented

Thanks @314l5926. I'd very much like to explore multi armed bandit options for Alephbet / Gimel. Perhaps semi-manually though (if the js code needs to fetch the ratios of variants on the fly, before actually choosing a variant, it might slow things down considerably).

My personal priority is to get something solid for analyzing the results and making an informed decision based on solid data. Even Miller and others have done amazing work, but I'm still a little unsure about trying to implement this myself, without a more solid statistics background. It looks like you have better skills and I'd love to work with you on this and support you in any way I can.

Just a few links from my list:

I have a few more lying around :)

commented

Good reading for a holiday! Thanks :)

commented

A simple dashboard was implemented and can be accessed using gimel dashboard. See here