DS3Lab / MLBench

Benchmarking Machine Learning Clouds

We conduct a benchmark over several popular machine learning cloud platforms, including Google, Amazon AWS, Azure and Oracle. For reproducibility, we make all datasets and code public in this repo. Our benchmark contains two snapshots, mlbench and automlbanch.

mlbench

mlbench was conducted during 2016-2017 when Amazon and Azure provide ML training services.

• Dataset
• Configurations.
  • Amazon. Previous configurations have expired due to platform upgrades. Refer to the latest confugiration.
  • Azure. We share training workflows in this link. An Azure ML studio account is required.

automlbench

automlbench was conducted during 2020-2022 when many machine learning clouds provide automatic machine learning training. Our goal is to evaluate their AutoML functionalities.

• Dataset
• Code. We provide jupyter scripts for three platforms.
  • Google. Google does training and inference without any code.
  • AWS. We provide scripts for inference.
  • Azure. We provide scripts for inference.
  • Oracle. We provide scripts for training and inference.
• Configurations. We show screenshots of how we configure each platform.
  • Google
  • AWS
  • Azure
  • Oracle