✔️ Update: new benchmark available: Multi-client benchmark on Medalla testnet 2020/10/01

⚠️ Note, these numbers gathered below might contain a methodical uncertainty based on:

how the prysm binaries were compiled; apparently, there is an undocumented option to pass --config=release to bazel to enable optimized builds,

how the virtual machines competed for resources; apparently, creating all three VMs at the same time caused them to end up on the same physical host.

I'll do my best to publish a new report for all clients soon(tm) on the new Altona testnet. I'll double check the build and runtime flags with the client teams and consider a physical host. Stay tuned.

✔️ Update: new benchmark available: Multi-client benchmark on Altona testnet 2020/07/16

Multi-client benchmark on Witti testnet 2020/06/22

Preliminary, high-level ETH2-client benchmarks.

Lighthouse
Prysm
Teku

Report: ./res/2020-06-eth2-bench.pdf (PDF), raw data: ./dat (CSV), client logs: ./log (LOG).

Host systems (3x)

VPS: Scaleway DEV1-XL (*)
OS: Arch Linux, kernel 5.7.2-arch1-1
CPU: AMD EPYC 7281, 4 vCPUs
RAM: 12 GB
Disk: 120GB NVMe

*) Note, this is a preliminary, high-level benchmark that should be repeated in future on bare-metal machines after stable client-releases are available to eliminate potential performance invariances caused by virtualized systems.

Lighthouse

lighthouse/0.1.2
master @ 710409c2bafbd118ac8a9a4115ed7a8abfe9929a (June/20, 2020)
Rust 1.44.1, Cargo 1.44.1
Permalink: https://github.com/sigp/lighthouse/tree/710409c2bafbd118ac8a9a4115ed7a8abfe9929a

Prysm

Prysm/Git commit: 172435374d07982a8f15b0ceb042b53ce284fa7e. Built at: 2020-06-22T10:55:28+02:00
witti @ 172435374d07982a8f15b0ceb042b53ce284fa7e (June/07, 2020)
Go 1.14.4, Bazel 3.2.0
Permalink: https://github.com/prysmaticlabs/prysm/tree/172435374d07982a8f15b0ceb042b53ce284fa7e

Teku

teku/v0.11.5-dev-3bd72680/linux-x86_64/-na-openjdk64bitservervm-java-14
master @ 3bd726809afe32296bc0b480168e380f66702354 (June/22, 2020)
Java 14.0.1, Gradle 6.5
Permalink: https://github.com/PegaSysEng/teku/tree/3bd726809afe32296bc0b480168e380f66702354

Witti testnet

Spec version: v0.11.3
Circa 190_000 slots filled with circa 70% blocks at start time
3 different clients known to validate the network
5 different clients known to connect and sync the network
Permalink: https://github.com/goerli/witti/tree/255cf1ab944ebad5a756116e81cd9079b65fae50

Metrics collected

time: Unix time in seconds
- all: (Ruby) Time.now.to_i
db: Database size in bytes
- Lighthouse: (Shell) du -bs $HOME/.lighthouse/beacon/chain_db/
- Prysm: (Shell) du -bs $HOME/.eth2/beaconchaindata/
- Teku: (Shell) du -bs $HOME/.local/share/teku/data/db/
mem: Resident memory in bytes
- Lighthouse: (REST API) /node/health, .pid_mem_resident_set_size
- Prysm: (Metrics) process_resident_memory_bytes
- Teku: (Metrics) process_resident_memory_bytes
slot: Head slot number in 1
- Lighthouse: (REST API) /beacon/head, .slot
- Prysm: (gRPC API) /eth/v1alpha1/beacon/chainhead, .headSlot
- Teku: (REST API) /beacon/head, .slot
bps: Slot sync each second in 1/second (*)
- all: (Ruby) (h.to_f - prev_h.to_f) / dt
peers: Peer count in 1
- Lighthouse: (REST API) /network/peer_count
- Prysm: (gRPC API) /eth/v1alpha1/node/peers (counted)
- Teku: (REST API) /network/peer_count

*) Note, due to different ways of processing incoming badges of blocks, this metric should be recomputed based on an average taken over 60 seconds.

Metrics derived (TBD)

Slot sync speed over time (moving average)
Database size over sync time
Database size over slot count (future projection: 1Y, 5Y)
Resident memory usage over sync time
Resident memory usage over time augmented with chain finality (external data)

q9f / eth2-bench-2020-06