This guide is intended to serve as a resource for individuals interested in the design and analysis of experiments. While its scope will encompass experimentation generally, particular attention will be paid to the considerations specific to online settings.
The goal of this guide is to help individuals understand causality in the context of randomized experimentation. It will cover the mechanics of experiments, focusing on how randomization solves the causal-inference problem, how to assess the validity of experimental designs, and how to deal with complications in experiments. In addition, it will address topics such as determining sample size, establishing experimental duration, and testing multiple conditions. To address the analysis component, this guide will cover sampling distributions, statistical inference, hypothesis testing, and using covariates with experimental data.
This will be presented as a set of Jupyter notebooks with text, formulae, and code examples, where possible.
This guide was created in order to formalize and share my understanding and enthusiasm for experimentation. It was first developed as part of an independent study course while I was a graduate student at the School of Information at the University of California, Berkeley.
These materials are based on multiple sources, including:
| Title | Authors |
|---|---|
| Field Experiments: Design, Analysis, and Interpretation | Gerber, Alan S. and Green, Donald P. |
| Introduction to Design and Analysis of Experiments | Cobb, George W. |
| Mostly Harmless Econometrics | Angrist, Joshua D. and Pischke, Jörn-Steffen |
| A First Course in Design and Analysis of Experiments | Oehlert, Gary W. |
This document will, no doubt, evolve with time. As a result, it might deviate from its original intent. Nevertheless, I hope you find it useful.