stillyslalom / benchmarks

Some benchmarks of different languages

Table of Content

• Overview
  • Measurements
• Test Cases
  • Brainfuck
    • bench.b
    • mandel.b
  • Base64
  • Json
  • Matmul
• Tests Execution
  • Environment
  • Using Docker
  • Manual Execution
    • Prerequisites
• Contribution
  • Makefile guide
    • Binary executables
    • Compiled artifacts
    • Scripting language

Overview

The benchmarks follow the criteria:

• They are written as the average software developer would write them, i.e.

  • The algorithms are implemented as cited in public sources;
  • The libraries are used as described in the tutorials, documentation and examples;
  • The used data structures are idiomatic.

• The used algorithms are similar between the languages (as the reference implementations), variants are acceptable if the reference implementation exists.

• All final binaries are releases (optimized for performance if possible) as debug performance may vary too much depending on the compiler.

My other benchmarks: jit-benchmarks, crystal-benchmarks-game

Measurements

The measured values are:

• time spent for the benchmark execution (loading required data and code self-testing are not measured);
• memory consumption of the benchmark process, reported as base + increase, where base is the RSS before the benchmark and increase is the peak increase of the RSS during the benchmark;
• energy consumption of the CPU (PP0 package) during the benchmark.

All values are presented as: median_{±median absolute deviation}.

UPDATE: 2021-02-27

Test Cases

Brainfuck

Testing brainfuck implementations using two code samples (bench.b and mandel.b). Supports two mode:

• Verbose (default). Prints the output immediately.
• Quiet (if QUIET environment variable is set). Accumulates the output using Fletcher-16 checksum, and prints it out after the benchmark.

Brainfuck

bench.b

Language	Time, s	Memory, MiB	Energy, J
C++/g++	0.892±0.022	1.48±00.01 + 0.00±00.00	18.30±00.92
Racket (Syntax Objects)	1.375±0.019	110.18±00.36 + 0.00±00.00	32.45±00.85
Kotlin	1.887±0.053	38.23±00.10 + 1.96±00.09	35.30±01.29
D/ldc2	1.888±0.048	3.03±00.03 + 0.00±00.00	37.00±02.52
Nim/gcc	1.900±0.091	1.83±00.03 + 0.00±00.00	40.81±02.49
C/gcc	1.939±0.048	0.54±00.01 + 0.00±00.00	37.72±03.49
D/gdc	1.940±0.124	6.26±00.04 + 0.00±00.00	40.74±03.16
Nim/clang	2.061±0.099	2.32±00.05 + 0.00±00.00	42.74±02.22
Rust	2.156±0.030	2.02±00.10 + 0.00±00.00	42.93±03.62
Java	2.184±0.076	37.52±00.13 + 1.03±00.07	40.80±02.68
Racket	2.194±0.194	116.20±00.06 + 1.55±00.26	55.26±02.86
C/clang	2.270±0.083	0.49±00.00 + 0.00±00.00	44.68±03.72
OCaml	2.270±0.056	2.56±00.02 + 2.51±00.03	46.23±05.72
Go	2.344±0.073	3.47±00.04 + 0.00±00.00	44.27±02.99
C#/.NET Core	2.362±0.124	34.19±00.04 + 0.01±00.00	46.54±02.62
F#/.NET Core	2.371±0.009	36.78±00.11 + 0.33±00.03	42.51±00.85
Vala/gcc	2.419±0.098	3.72±00.03 + 0.00±00.00	45.36±02.88
Vala/clang	2.492±0.076	3.67±00.05 + 0.00±00.00	51.39±04.14
V/gcc	2.536±0.121	0.55±00.01 + 0.00±00.00	54.42±03.05
Crystal	2.560±0.156	3.29±00.03 + 0.00±00.00	51.61±03.00
Go/gccgo	2.638±0.310	20.96±00.17 + 0.00±00.00	52.51±08.29
MLton	2.774±0.134	1.40±00.02 + 0.25±00.00	61.95±05.52
V/clang	2.916±0.086	0.85±00.00 + 0.00±00.00	62.00±03.80
Julia	3.033±0.098	168.18±00.20 + 0.00±00.00	54.70±04.97
Chez Scheme	3.059±0.083	24.83±00.02 + 4.21±00.04	59.99±04.24
Scala	3.520±0.024	80.43±00.92 + 60.43±06.97	69.69±01.65
D/dmd	3.590±0.105	3.60±00.02 + 0.00±00.00	70.59±04.10
Node.js	4.171±0.210	30.23±00.05 + 1.76±00.00	82.40±06.47
C#/Mono	4.475±0.155	20.05±00.10 + 0.00±00.00	81.10±04.83
Haskell (MArray)	4.648±0.172	3.59±00.05 + 1.16±00.00	99.26±07.84
Lua/luajit	6.416±0.322	2.89±00.05 + 0.00±00.00	134.68±08.07
Ruby/truffleruby	9.975±0.264	250.78±00.14 + 750.14±20.30	263.86±19.35
Ruby/truffleruby (--jvm)	10.610±0.952	565.07±05.30 + 546.80±66.01	352.97±15.23
Python/pypy	15.790±0.416	63.91±00.24 + 45.38±00.03	344.53±22.25
Haskell	16.417±0.706	3.79±00.04 + 0.88±00.00	386.41±14.14
Ruby (--jit)	58.145±1.318	14.03±00.02 + 0.23±00.00	1285.44±28.51
Lua	59.404±1.460	2.97±00.03 + 0.00±00.00	1116.62±41.84
Ruby	92.382±2.821	14.01±00.05 + 0.00±00.00	1720.82±80.86
Ruby/jruby	107.728±5.276	205.96±05.72 + 235.28±04.00	2175.45±74.60
Elixir	119.456±3.617	58.12±01.06 + 0.00±00.00	2606.79±74.01
Python	234.997±6.281	10.29±00.03 + 0.00±00.00	5337.77±84.07
Tcl (FP)	291.608±2.969	4.28±00.04 + 0.00±00.00	5530.18±199.13
Perl	362.950±6.122	6.48±00.06 + 0.00±00.00	7931.43±195.67
Tcl (OOP)	560.871±14.755	4.27±00.03 + 0.00±00.00	11993.69±613.22

mandel.b

Mandel in Brainfuck

Language	Time, s	Memory, MiB	Energy, J
C/gcc	13.477±0.263	0.52±00.03 + 1.09±00.03	234.35±09.07
C++/g++	14.001±0.347	3.14±00.15 + 0.50±00.03	244.94±11.59
D/ldc2	14.281±0.213	3.03±00.05 + 0.77±00.00	258.85±14.10
D/gdc	14.499±0.398	6.69±00.08 + 0.52±00.00	253.16±10.80
Nim/gcc	16.364±0.246	1.84±00.01 + 0.57±00.00	361.69±08.01
V/gcc	16.535±0.686	0.52±00.03 + 1.89±00.05	310.30±22.82
Kotlin	16.541±0.824	38.35±00.21 + 2.50±00.25	331.29±19.28
Go	16.760±0.574	3.43±00.09 + 1.29±00.00	339.53±29.47
Racket (Syntax Objects)	17.710±0.900	110.05±00.13 + 70.90±00.13	362.86±07.00
C/clang	18.400±0.625	0.49±00.00 + 1.11±00.01	363.79±24.65
C#/.NET Core	18.763±0.666	34.35±00.06 + 1.00±00.00	368.78±32.99
Rust	19.505±0.307	2.00±00.04 + 0.28±00.03	456.28±08.89
Crystal	19.997±0.534	3.32±00.04 + 0.45±00.02	393.08±28.54
Nim/clang	20.742±0.529	2.28±00.02 + 0.51±00.00	461.96±15.63
V/clang	20.757±0.507	0.88±00.04 + 1.97±00.12	371.94±14.40
Java	22.209±0.362	37.52±00.16 + 1.53±00.35	425.77±44.37
Vala/clang	22.473±0.543	3.56±00.05 + 2.04±00.04	428.17±22.08
Vala/gcc	22.723±0.469	3.55±00.01 + 2.05±00.01	412.52±16.09
Scala	24.482±0.359	80.25±00.58 + 37.60±04.43	465.50±11.89
Go/gccgo	26.182±0.458	21.37±00.17 + 1.28±00.04	450.94±10.14
F#/.NET Core	35.402±0.212	36.83±00.06 + 2.07±00.03	643.00±19.63
OCaml	38.179±1.251	3.85±00.02 + 6.95±00.26	748.19±52.73
Chez Scheme	40.251±0.542	25.43±00.04 + 3.67±00.02	926.00±36.88
Racket	42.992±4.642	115.71±00.18 + 2.06±00.26	832.01±71.41
D/dmd	45.225±1.074	3.56±00.05 + 0.77±00.00	869.87±28.71
C#/Mono	46.198±1.532	20.13±00.05 + 0.88±00.00	907.03±76.61
Node.js	47.642±0.449	30.20±00.13 + 5.83±00.11	866.33±21.85
MLton	50.890±0.252	1.40±00.03 + 4.11±00.00	1172.41±15.11
Julia	60.716±1.267	168.63±00.28 + 0.00±00.00	1067.66±10.06
Haskell (MArray)	61.456±0.617	3.62±00.04 + 2.68±00.00	1401.97±26.08
Python/pypy	67.820±0.658	63.93±00.19 + 45.90±00.04	1575.02±56.96
Ruby/truffleruby (--jvm)	131.730±2.236	565.14±10.92 + 466.56±18.77	2725.46±62.38
Ruby/truffleruby	172.357±4.092	251.03±00.30 + 758.66±17.72	3768.42±226.62
Haskell	216.273±4.050	3.83±00.05 + 26.16±00.00	4543.18±330.08
Lua/luajit	253.096±2.212	2.86±00.02 + 0.86±00.00	4740.45±61.20

Base64

Testing base64 encoding/decoding of the large blob into the newly allocated buffers.

Base64

Language	Time, s	Memory, MiB	Energy, J
C/gcc (aklomp)	0.159±0.003	1.89±00.04 + 0.00±00.00	3.60±00.20
C/gcc	1.240±0.029	1.84±00.02 + 0.00±00.00	24.80±01.95
Rust	1.328±0.049	2.46±00.09 + 0.01±00.00	25.90±02.42
Crystal	1.605±0.008	3.73±00.03 + 1.78±00.04	38.41±00.34
V/clang	1.659±0.032	1.96±00.02 + 0.49±00.02	34.24±03.03
Nim/clang	1.728±0.016	2.68±00.03 + 4.38±00.00	38.53±01.02
V/gcc	1.782±0.012	1.71±00.04 + 0.25±00.03	29.96±00.45
D/ldc2	2.000±0.008	3.41±00.04 + 3.66±00.00	36.64±00.17
Nim/gcc	2.013±0.030	2.21±00.06 + 4.44±00.00	44.68±01.82
D/gdc	2.023±0.077	7.08±00.05 + 3.46±00.00	43.81±02.27
Vala/gcc	2.148±0.051	4.94±00.05 + 0.57±00.02	42.36±01.27
Vala/clang	2.198±0.055	4.97±00.05 + 0.56±00.06	41.69±01.05
Ruby	2.312±0.028	14.38±00.04 + 55.53±01.54	52.98±01.37
Java	2.328±0.051	38.46±00.09 + 288.62±20.60	47.21±02.47
Ruby (--jit)	2.342±0.054	14.46±00.06 + 57.15±01.20	53.35±01.29
C++/g++ (libcrypto)	2.512±0.052	5.40±00.04 + 0.07±00.00	56.47±02.30
Kotlin	2.528±0.030	39.38±00.19 + 322.70±07.06	50.71±02.27
Go	2.619±0.006	4.55±00.03 + 5.38±00.14	49.05±00.38
Scala	2.625±0.015	80.13±00.22 + 75.91±04.39	49.38±00.74
Node.js	2.959±0.045	30.82±00.05 + 1029.46±00.16	63.51±01.44
Perl (MIME::Base64)	2.983±0.101	14.13±00.04 + 0.08±00.00	58.70±05.39
PHP	3.003±0.021	15.71±00.07 + 0.00±00.00	64.76±02.97
Go/gccgo	3.539±0.003	22.06±00.16 + 7.32±00.30	76.84±00.67
D/dmd	4.120±0.106	3.69±00.05 + 3.67±00.05	72.67±02.73
Python	4.341±0.115	10.03±00.03 + 0.18±00.00	77.43±02.75
Tcl	4.636±0.021	4.78±00.10 + 0.16±00.04	102.14±00.80
Python/pypy	5.054±0.128	63.94±00.25 + 45.73±00.05	91.32±05.98
F#/.NET Core	5.635±0.067	37.12±00.04 + 33.62±05.08	91.11±01.41
C#/.NET Core	5.836±0.042	34.61±00.02 + 33.39±06.29	109.33±00.58
Ruby/truffleruby (--jvm)	5.861±0.075	552.67±06.94 + 310.16±25.83	132.20±05.92
Julia	6.010±0.117	200.46±00.11 + 42.95±00.25	111.63±04.11
C#/Mono	7.334±0.082	20.58±00.03 + 18.49±00.02	170.73±08.58
Ruby/jruby	10.684±0.600	199.34±04.86 + 144.29±15.58	219.12±15.61
Perl (MIME::Base64::Perl)	16.395±0.314	15.38±00.05 + 0.27±00.03	368.17±11.67
Ruby/truffleruby	22.208±0.143	241.24±00.84 + 397.56±00.08	388.75±04.64

Json

Testing parsing and simple calculating of values from a big JSON file.

Json

Language	Time, s	Memory, MiB	Energy, J
C++/g++ (DAW JSON Link)	0.086±0.003	109.27±00.01 + 0.00±00.00	1.83±00.18
C++/g++ (simdjson On-Demand)	0.087±0.002	109.90±00.05 + 59.55±00.00	1.75±00.09
D/gdc (fast)	0.111±0.005	220.07±00.05 + 11.09±00.26	2.54±00.28
Rust (Serde Typed)	0.145±0.003	108.46±00.08 + 11.80±00.26	2.96±00.23
Rust (Serde Custom)	0.151±0.005	108.45±00.06 + 0.00±00.00	2.54±00.09
C++/g++ (gason)	0.156±0.004	109.25±00.01 + 97.11±00.03	3.47±00.26
C++/g++ (simdjson DOM)	0.160±0.002	109.84±00.00 + 176.60±00.00	3.09±00.07
C++/g++ (RapidJSON)	0.223±0.008	109.25±00.02 + 128.82±00.00	4.52±00.42
C++/g++ (Boost.JSON)	0.525±0.006	109.80±00.04 + 435.70±00.00	9.51±00.26
Java	0.536±0.018	252.55±00.08 + 74.23±00.79	13.49±00.61
C++/g++ (RapidJSON SAX)	0.574±0.018	109.46±00.01 + 0.00±00.00	10.10±00.65
Scala	0.611±0.023	328.42±02.85 + 74.73±00.60	15.73±01.00
Node.js	0.637±0.018	243.66±00.03 + 184.96±00.56	15.88±01.14
Go (jsoniter)	0.662±0.009	224.46±00.06 + 13.61±00.13	14.68±00.65
Crystal (Schema)	0.766±0.022	110.30±00.05 + 47.04±00.10	11.98±00.30
Crystal (Pull)	0.811±0.034	110.33±00.06 + 18.23±00.02	14.91±01.35
Python/pypy	0.843±0.019	277.49±00.19 + 127.96±00.00	15.94±00.94
Julia (JSON3)	0.848±0.009	361.87±00.43 + 360.75±00.26	19.82±00.29
Rust (Serde Untyped)	0.851±0.035	108.45±00.12 + 839.98±00.00	17.46±01.63
V/clang	0.859±0.016	108.36±00.05 + 484.15±00.06	15.38±00.63
V/gcc	0.885±0.013	107.44±00.06 + 484.48±00.12	20.51±00.87
Perl (Cpanel::JSON::XS)	0.961±0.023	121.30±00.05 + 402.72±00.00	22.33±00.43
Crystal	0.969±0.005	110.34±00.06 + 393.37±00.02	21.43±00.18
C#/.NET Core (System.Text.Json)	1.005±0.040	465.42±00.09 + 135.68±00.01	23.01±01.06
Go	1.093±0.011	113.98±00.09 + 83.29±00.02	24.16±00.38
PHP	1.281±0.042	121.76±00.14 + 682.01±00.00	23.12±01.01
Go/gccgo	1.437±0.035	132.60±00.13 + 95.90±00.10	31.82±00.91
Nim/clang (Packedjson)	1.548±0.064	109.17±00.10 + 290.55±00.00	31.93±03.05
Nim/gcc (Packedjson)	1.550±0.029	108.74±00.06 + 290.55±00.00	26.92±00.46
C++/g++ (json-c)	1.551±0.044	109.42±00.04 + 1216.08±00.00	31.40±03.00
Clojure	1.643±0.028	488.62±04.90 + 525.49±14.00	42.18±01.44
C#/.NET Core	1.734±0.053	474.26±00.03 + 288.66±00.06	34.47±04.08
Haskell	1.760±0.040	4.70±00.09 + 4.61±00.02	40.29±02.69
Python	1.763±0.056	116.74±00.03 + 377.21±00.00	37.76±02.63
CPython (UltraJSON)	1.783±0.029	118.39±00.03 + 543.47±00.39	37.20±01.46
Nim/clang	1.833±0.026	109.24±00.05 + 919.42±00.03	42.46±00.75
Nim/gcc	1.941±0.097	108.78±00.04 + 919.42±00.03	38.89±03.82
Ruby	2.206±0.028	120.55±00.03 + 410.67±00.01	50.84±00.77
C#/Mono	2.207±0.113	462.64±00.09 + 0.17±00.02	46.36±07.33
D/gdc	2.220±0.020	113.29±00.02 + 600.33±00.00	50.18±00.93
Ruby (YAJL)	2.272±0.030	120.47±00.05 + 281.58±00.01	52.01±01.72
Ruby (--jit)	2.300±0.076	120.61±00.02 + 410.78±00.01	50.17±03.75
F#/.NET Core (System.Text.Json)	2.555±0.005	471.58±00.05 + 444.65±01.22	52.43±00.74
D/ldc2	2.561±0.025	109.59±00.05 + 680.20±00.04	51.38±01.48
Rust (jq)	3.744±0.097	110.43±00.03 + 775.37±00.52	69.39±05.88
Ruby/jruby	3.887±0.102	475.26±04.27 + 1484.87±32.34	114.97±02.68
C++/g++ (Boost.PropertyTree)	4.359±0.073	109.61±00.03 + 1440.06±00.00	101.74±01.57
D/dmd	5.029±0.101	110.12±00.03 + 680.10±00.03	99.91±03.77
Vala/gcc	5.843±0.172	111.17±00.07 + 997.54±00.01	106.11±02.27
Vala/clang	6.087±0.149	111.21±00.05 + 932.12±00.12	108.56±03.08
Perl (JSON::Tiny)	11.675±0.170	122.00±00.04 + 525.16±00.03	266.13±04.55
Ruby/truffleruby (--jvm)	18.393±0.585	748.21±14.35 + 1639.80±63.41	495.50±12.87
Ruby/truffleruby	42.147±1.079	733.18±00.41 + 2342.65±22.01	885.48±36.81

Matmul

Testing allocating and multiplying matrices.

Matmul

Language	Time, s	Memory, MiB	Energy, J
D/ldc2 (lubeck)	0.063±0.001	6.69±00.08 + 55.58±00.12	3.61±00.06
Python (NumPy)	0.099±0.002	27.64±00.10 + 57.64±00.05	5.56±00.20
Nim/gcc (Arraymancer)	0.165±0.014	5.81±00.13 + 57.56±00.12	8.76±00.77
Nim/clang (Arraymancer)	0.185±0.049	6.56±00.08 + 57.41±00.05	8.78±01.76
Java (ND4J)	0.186±0.010	133.12±02.85 + 87.35±00.00	8.69±00.63
Julia (threads: 8)	0.197±0.003	221.88±00.31 + 52.84±00.11	10.82±00.17
Julia (threads: 1)	0.571±0.024	222.09±00.17 + 52.64±00.00	11.01±00.46
D/ldc2	1.998±0.008	3.58±00.05 + 70.11±00.00	45.53±00.46
D/gdc	2.130±0.033	6.71±00.05 + 70.71±00.01	49.98±03.33
D/dmd	2.190±0.029	3.51±00.06 + 70.11±00.00	47.95±01.36
C/gcc	3.361±0.032	1.96±00.03 + 68.06±00.00	76.13±00.66
Java	3.375±0.028	37.91±00.30 + 77.38±00.18	76.02±01.56
Scala	3.405±0.045	78.83±04.99 + 74.79±07.60	79.08±02.97
Rust	3.438±0.015	2.59±00.10 + 68.32±00.00	72.45±01.23
Nim/gcc	3.499±0.024	2.58±00.03 + 70.12±00.13	72.89±00.79
Nim/clang	3.515±0.020	3.10±00.03 + 79.66±05.67	74.79±01.28
Julia (no BLAS)	3.568±0.025	178.15±00.22 + 69.76±00.06	75.14±01.00
Vala/gcc	3.622±0.063	5.24±00.06 + 68.32±00.00	63.98±02.77
Go	3.638±0.029	3.86±00.09 + 73.33±00.09	79.54±02.48
Vala/clang	3.639±0.028	5.25±00.04 + 68.32±00.00	70.06±02.38
Go/gccgo	3.708±0.049	21.82±00.58 + 72.61±00.14	75.63±01.61
Swift	3.720±0.060	149.02±00.05 + 59.60±00.06	91.24±03.75
Crystal	3.766±0.092	4.10±00.04 + 59.69±00.04	77.91±05.35
Node.js	3.840±0.070	33.98±00.09 + 71.64±00.62	89.11±05.38
V/clang	3.924±0.056	2.40±00.05 + 68.84±00.00	79.38±05.46
Kotlin	4.024±0.118	37.85±00.05 + 77.94±00.17	76.73±06.10
V/gcc	4.941±0.036	1.99±00.03 + 68.84±00.00	89.41±03.53
Python/pypy	6.420±0.201	64.28±00.20 + 69.19±00.04	115.52±05.81
C#/.NET Core	6.819±0.095	34.00±00.08 + 69.11±00.00	163.77±05.84
C#/Mono	11.473±0.327	20.26±00.03 + 69.01±00.00	207.27±12.91
Ruby/truffleruby	48.858±0.386	522.26±01.54 + 739.45±02.45	1166.40±23.30
Ruby/truffleruby (--jvm)	71.053±1.215	613.42±10.65 + 420.01±15.34	1928.18±51.69
Ruby (--jit)	211.146±4.444	15.21±00.05 + 68.87±00.00	4762.60±75.89
Ruby	214.346±3.747	15.15±00.05 + 68.64±00.00	4777.97±185.82
Python	234.810±4.264	10.51±00.05 + 68.58±00.00	5089.89±88.84
Tcl	355.979±9.911	7.18±00.06 + 400.44±00.06	7466.13±555.27
Perl	396.637±4.175	8.99±00.07 + 599.63±00.03	9832.31±213.24
Ruby/jruby	500.411±9.064	279.30±03.56 + 663.99±11.11	11013.14±240.57

Tests Execution

Environment

CPU: Intel(R) Core(TM) i7-10710U

Base Docker image: Debian GNU/Linux bullseye/sid

Language	Version
.NET Core	5.0.103
C#/.NET Core	3.8.0-5.20604.10 (9ed4b774)
C#/Mono	6.12.0.107
C/clang	11.0.1
C/gcc	10.2.1
Chez Scheme	9.5.4
Clojure	"1.10.2"
Crystal	0.36.1
D/dmd	v2.095.1
D/gdc	10.2.1
D/ldc2	1.25.0
Elixir	1.10.3
F#/.NET Core	11.0.0.0 for F# 5.0
Go	go1.16
Go/gccgo	10.2.1
Haskell	8.10.4
Java	15.0.2
Julia	v"1.5.3"
Kotlin	1.4.30
Lua	Lua 5.4
Lua/luajit	LuaJIT 2.1.0-beta3
MLton	20210117
Nim	1.4.4
Node.js	v15.10.0
OCaml	4.11.1
PHP	7.4.15
Perl	v5.32.1
Python	3.9.1+
Python/pypy	7.3.3-beta0 for Python 3.7.9
Racket	"8.0"
Ruby	3.0.0p0
Ruby/jruby	9.2.14.0
Ruby/truffleruby	21.0.0.2
Rust	1.50.0
Scala	2.13.5
Swift	swift-5.3.3-RELEASE
Tcl	8.6
V	0.2.2
Vala	0.48.13

Using Docker

Build the image:

$ docker build docker/ -t benchmarks

Run the image:

$ docker run -it --rm -v $(pwd):/src benchmarks <cmd>

where is:

• versions (print installed language versions);
• shell (start the shell);
• brainfuck bench (build and run Brainfuck bench.b benchmarks);
• brainfuck mandel (build and run Brainfuck mandel.b benchmarks);
• base64 (build and run Base64 benchmarks);
• json (build and run Json benchmarks);
• matmul (build and run Matmul benchmarks);

Please note that the actual measurements provided in the project are taken semi-manually (via shell) as the full update takes days and could have occassional issues in Docker.

There is a Makefile that could be used to simlify Docker usage:

• make build (build the image);
• make versions (run the image with the versions command);
• make shell (run the image with the `shell' command);
• make toc (utility rule to update ToC in this README, requires git-markdown-toc available in PATH).

Please note that the make shell rule requires cpupower utility installed that is invoked with sudo to set cpufreq's performance governon (it runs the CPU at the maximum frequence to eliminate throttling issues).

Manual Execution

Makefiles contain recipes for building and executing tests with the proper dependencies. Please use make run (and make run2 where applicable). The measurements are taken using analyze.rb script:

$ cd <test suite>
$ ../analyze.rb make run
$ ../analyze.rb make run[<single test>]

Please note that the measurements could take hours. It uses 10 iterations by default, but it could be changed using ATTEMPTS environment variable:

$ ATTEMPTS=1 ../analyze.rb make run

Prerequisites

Please use Dockerfile as a reference regarding which packages and tools are required.

For all (optional):

• Powercap for reading energy counters in Linux (Debian package powercap-utils).

For Python:

• NumPy for matmul tests (Debian package python3-numpy).
• UltraJSON for JSON tests (Debian package python3-ujson).

For C++:

• Boost for JSON tests (Debian package libboost-dev).
• JSON-C for JSON tests (Debian package libjson-c-dev).

For Rust:

• libjq for jq test (Debian packages libjq-dev, libonig-dev and environment variable JQ_LIB_DIR=/usr/lib/x86_64-linux-gnu/).

For Java, Scala:

• Coursier for downloading Maven artifacts.

For Lua:

• LuaRocks for installing dependencies (Debian package luarocks).

For Haskell:

• network for TCP connectivity between the tests and the test runner.
• raw-strings-qq for raw string literals used in tests.

For Perl:

• cpanminus for installing modules from CPAN (Debian package cpanminus).

For Vala:

• JSON-GLib for JSON tests (Debian package libjson-glib-dev).

Contribution

Please follow the criteria specified in the overview. Besides that please ensure that the communication protocol between a test and the test runner is satisfied:

• The test runner listens on localhost:9001;
• All messages are sent using TCP sockets closed immediately after the message has been sent;
• There are two messages sent from a test (it establishes the measurement boundary):
  1. The beginning message having the format name of the test/tprocess ID (the process ID is used to measure the memory consumption). Please note that the name of the test couldn't use Tab character as it's a delimiter;
  2. The end message with any content (mostly it's "stop" for consistency).
• The test runner could be unavailable (if the test is launched as is) and the test should gracefully handle it.

Makefile guide

Binary executables

If the test is compiled into a single binary, then two sections of the Makefile require changes:

• append a new target (the final binary location) into executables variable;
• append the proper target rule.

Compiled artifacts

If the test is compiled, but can't be executed directly as a binary, then three sections of the Makefile require changes:

• append a new target (the final artifact location) into artifacts variable;
• append the proper target rule to compile the test;
• append run[<target_artifact>] rule to run the test.

Scripting language

If the test doesn't require compilation, then two sections of the Makefile requires changes:

• append run[<script_file>] into all_runners variable;
• append run[<script_file>] rule to run the test.