DAWNBench Analysis
Introduction
DAWNBench is an end-to-end deep learning training and inference benchmark. Computation time and cost are critical resources in building deep models, yet many existing benchmarks focus solely on model accuracy.
This repository contains the analysis of DAWNBench results. We will analyze two common deep learning workloads, CIFAR-10 and ImageNet time-to-accuracy and training cost across different optimization strategies, model architectures, software frameworks, clouds, and hardware.
For the start, we will analyze only CIFAR-10 workload across two model architectures:
- Custom Wide ResNet
- ResNet18 (TODO)
Rank | Time to 94% Accuracy | Model | Framework | Hardware |
---|---|---|---|---|
1 | 00:52:11 | Custom Wide ResNet + AdamW + modified 1 cycle policy Cedric Chee source |
fastai / PyTorch 0.4.0 | 1 K80 (AWS p2.xlarge) |
2 | 00:55:51 | Custom Wide ResNet + modified 1 cycle policy Cedric Chee source |
fastai / PyTorch 0.4.0 | 1 K80 (AWS p2.xlarge) |
3 | 0:59:38 | Custom DarkNet Cedric Chee source |
PyTorch 0.3.1 | 1 K80 (AWS p2.xlarge) |
4 | 1:18:22 | Custom Wide ResNet fast.ai students team source |
fastai / PyTorch 0.4.0 | 1 K80 (AWS p2.xlarge) |
Analysis
1. Custom Wide ResNet fast.ai students team DAWNBench submission (Baseline)
2. My Custom Wide ResNet + AdamW + modified 1 cycle policy hyper-parameters
3. My Custom Wide ResNet + modified 1 cycle policy hyper-parameters
Code to replicate all analyses
Instructions for building and running the container.
You will need to have Docker and nvidia-docker installed in order to run this.
- cd into cloned repo.
docker build -t cifar10 .
./run_container.sh cifar10
Once the Docker container started, all you have to do is access Jupyter Notebook through this url, https://localhost:8888 in your web browser. Then, enter jupyter
as password. Open the notebook and click 'run all cells'.