Abaco performance study

[Joe Stubbs]

Let's do a performance study of the aggregate throughput for the Abaco system across a few different dimensions. We should write a program to perform the measurement on a fixed instance and then run the program a number of times on different instance sizes (compute cluster sizes) to determine how performance scales.

Here are some initial thoughts; we should move to a google doc to more easily collaborate on the design.

• We should consider CPU-bound workloads and I/O bound workloads. We should use standard benchmarks where possible; i.e., LINPACK (or a derivative thereof) for CPU-bound workloads, and possibly reading from and writing to a cloud/HTTP storage API for the I/O bound workloads.
• We should measure runs with pre-scheduled workers as well as runs that only leverage the autoscaler. The latter will be less performant, but if it is within a small percentage of the former that will be a compelling result.