allenchng / exp-readings

A collection of resources for experimentation

Experimentation Resources

This is a list of resources that I found useful when studying for experimentation based data science roles. It is by no means an exhaustive list, please feel free to message me or submit a PR if you think the organization could be improved or I've omitted any outstanding examples.

Inspired by Dustin Tran's ML Video repo and PhD qual's lists.

General Overview

• Kohavi et al., 2014 - Seven Rules of Thumb for Web Site Experimenters

• Kohavi et al., 2020 - Online randomized controlled experiments at scale: lessons and extensions to medicine

• Kohavi & Longbotham, 2017 - Online Controlled Experiments and A/B Testing

• Kohavi et al., 2009 - Controlled experiments on the web: survey and practical guide

• Alex Deng - Trustworthy Analysis of Online A/B Tests: Pitfalls, Challenges and Solutions

• Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained

• Twitter - The what and why of product experimentation at Twitter

• AirBnB - 4 Principles for Making Experimentation Count

Measuring Treatment Effects

• Facebook - A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook

• Google - Inferring causal impact using Bayesian structural time-series models

• Twitch - A/B Testing using Google’s Staged Rollouts

• Twitch - Two-Stage Least Squares For A/B Tests

• Booking.com - Understanding Mechanisms of Change in Online Experiments at Booking.com

• Wayfair - Wayfair DS Explains It All: Jerry Chen on A/B Test Measurement Validation

• Wayfair - Modeling Uplift Directly: Uplift Decision Tree with KL Divergence and Euclidean Distance as Splitting Criteria

Statistical Testing

• VK Tech - Practitioner’s Guide to Statistical Tests

• Netflix - Streaming Video Experimentation at Netflix: Visualizing Practical and Statistical Significance

• Ellery Wulczyn - AB Testing and the Importance of Independent Observations

• Ellery Wulczyn - How to do AB Testing with Discrete Rewards?

• Simon Jackson - Recreating Netflix’s quantile bootstrapping in R

• Cross Validated - What is difference-in-differences?

• Wolfram - Quantile Regression—Theory, Implementations, and Applications

• Twitter - Detecting and avoiding bucket imbalance in A/B tests

• Twitter - Power, minimal detectable effect, and bucket size estimation in A/B tests

• Twitter - Implications of use of multiple controls in an A/B test

Power / Stopping Rules

• Booking - How Booking.com increases the power of online experiments with CUPED

• Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data

• Analytics Toolkit - What does "Sequential Testing" mean?

• Microsoft - Sequential Estimation of Quantiles with Applications to A/B-testing and Best-arm Identification

• Simon Jackson - The 4-step Refresher of Statistical Power

• Etsy - How Etsy Handles Peeking in A/B Testing

Bayesian A/B Testing

• John Krushke - Optional stopping in data collection: p values, Bayes factors, credible intervals, precision

• Analytics Toolkit - Bayesian A/B Testing Is Not Immune To Optional Stopping Issues

• Chris Stucchio - Easy Evaluation of Decision Rules in Bayesian A/B Testing

• Clauido Bellei - Bayesian A/B Testing: a step-by-step guide

• David Robinson - Is Bayesian A/B Testing Immune To Peeking? Not Exactly

• Stitch Fix - Large Scale Experimentation

Multivariate Testing

• Analytics Toolkit - Multivariate Testing – Best Practices & Tools for MVT (A/B/n) Tests

• Nielsen-Norman Group - Multivariate vs. A/B Testing: Incremental vs. Radical Changes

Bandits

• Stitch Fix - Your Client Engagement Program Isn't Doing What You Think It Is.

• Stitch Fix - Multi-Armed Bandits and the Stitch Fix Experimentation Platform

Case Studies

Quality of Experience

• Netflix: A/B Testing and Beyond: Improving the Netflix Streaming Experience with Experimentation and Data Science

Billing

• SecurionPay - Payment Gateway A/B Testing Guide – How to effectively test different payment providers and methods

• Recurly - Predicting Recurring Transaction Success

Marketing

• Wayfair - Uplift Modeling in Display Remarketing

Pricing

• Wayfair - Wayfair DS Explains It All: Lin Jia on Measuring Price Effects

Product Analytics

• StellarPeers - How would you find the cause of a 15% drop in Facebook Groups usage?

• StellarPeers - How do you decide between displaying Facebook’s ‘People You May Know’ feature or an Advertisement?

Experimentation Platforms

• Wayfair - Gemini: Wayfair’s advanced marketing test design and measurement platform

• LinkedIn - Building a Scalable Experimentation Platform by LinkedIn

• Uber - Building an Experimentation Platform at Uber

• Spotify - Building a scalable experimentation platform at Spotify by Karina Bunyik

• Pinterest - AB Testing At Pinterest: Building A Culture Of Experimentation

• Stitch Fix - Building our Centralized Experimental Platform

• Spotify - Spotify’s New Experimentation Platform (Part 1)

• Spotify - Spotify’s New Experimentation Platform (Part 2)

• Twitter - Twitter experimentation: technical overview

• DoorDash - Meet Dash-AB — The Statistics Engine of Experimentation at DoorDash

Software

• Wayfair - Pylift: A Fast Python Package for Uplift Modeling