betseg / benchmarks

Some benchmarks of different languages

Table of Content

• Overview
  • Measurements
• Test Cases
  • Brainfuck
    • bench.b
    • mandel.b
  • Base64
  • Json
  • Matmul
  • Primes
• Tests Execution
  • Environment
  • Using Docker
  • Manual Execution
    • Prerequisites
• Contribution
  • Makefile guide
    • Binary executables
    • Compiled artifacts
    • Scripting language

Overview

The benchmarks follow the criteria:

• They are written as the average software developer would write them, i.e.

  • The algorithms are implemented as cited in public sources;
  • The libraries are used as described in the tutorials, documentation and examples;
  • The used data structures are idiomatic.

• The used algorithms are similar between the languages (as the reference implementations), variants are acceptable if the reference implementation exists.

• All final binaries are releases (optimized for performance if possible) as debug performance may vary too much depending on the compiler.

My other benchmarks: jit-benchmarks, crystal-benchmarks-game

Measurements

The measured values are:

• time spent for the benchmark execution (loading required data and code self-testing are not measured);
• memory consumption of the benchmark process, reported as base + increase, where base is the RSS before the benchmark and increase is the peak increase of the RSS during the benchmark;
• energy consumption of the CPU package during the benchmark: PP0 (cores) + PP1 (uncores like GPU) + DRAM.

All values are presented as: median_{±median absolute deviation}.

UPDATE: 2022-02-18

Test Cases

Brainfuck

Testing brainfuck implementations using two code samples (bench.b and mandel.b). Supports two mode:

• Verbose (default). Prints the output immediately.
• Quiet (if QUIET environment variable is set). Accumulates the output using Fletcher-16 checksum, and prints it out after the benchmark.

Brainfuck

bench.b

Language	Time, s	Memory, MiB	Energy, J
Racket (Syntax Objects)	0.848±0.013	112.71±00.74 + 0.00±00.00	18.86±01.01
C++/g++	1.143±0.063	1.68±00.00 + 0.00±00.00	20.79±02.04
C/gcc	1.319±0.006	0.73±00.00 + 0.00±00.00	28.21±00.90
Go	1.367±0.044	3.35±00.05 + 0.00±00.00	26.07±02.29
C++/clang++	1.407±0.054	1.53±00.03 + 0.00±00.00	28.82±02.69
D/ldc2	1.420±0.018	3.14±00.06 + 0.00±00.00	29.74±00.87
Rust	1.444±0.021	2.16±00.04 + 0.00±00.00	24.06±00.58
Vala/gcc	1.453±0.021	4.33±00.03 + 0.00±00.00	27.56±01.83
D/gdc	1.470±0.031	6.86±00.03 + 0.00±00.00	31.01±00.88
Nim/gcc	1.489±0.052	1.92±00.09 + 0.00±00.00	28.67±02.70
C/clang	1.526±0.018	0.68±00.00 + 0.00±00.00	26.22±00.52
Vala/clang	1.581±0.014	4.37±00.03 + 0.00±00.00	30.19±00.52
Java	1.648±0.017	37.17±00.23 + 1.56±00.36	36.73±00.64
Go/gccgo	1.678±0.094	22.37±00.12 + 0.00±00.00	33.14±01.98
Zig	1.682±0.048	0.72±00.00 + 0.00±00.00	33.60±03.41
Nim/clang	1.768±0.065	2.38±00.04 + 0.00±00.00	31.90±02.28
Kotlin/JVM	1.797±0.036	40.57±00.29 + 1.19±00.60	33.54±01.29
OCaml	1.956±0.033	2.85±00.05 + 2.27±00.03	42.49±02.50
MLton	2.039±0.044	1.47±00.07 + 0.25±00.00	44.69±02.30
Racket	2.131±0.137	95.76±00.26 + 0.00±00.00	42.11±03.49
C#/.NET Core	2.256±0.063	34.20±00.05 + 0.00±00.00	43.70±03.79
V/clang	2.288±0.096	0.74±00.00 + 0.00±00.00	49.66±02.87
Julia	2.392±0.097	207.12±00.32 + 0.70±00.03	35.68±01.69
Crystal	2.433±0.068	3.52±00.01 + 0.00±00.00	43.41±02.12
F#/.NET Core	2.551±0.068	37.80±00.08 + 0.61±00.00	45.74±02.08
C#/Mono	2.803±0.104	20.40±00.03 + 0.00±00.00	57.16±05.19
Scala	2.988±0.055	75.72±00.09 + 207.00±30.15	60.22±04.14
Chez Scheme	3.022±0.122	24.83±00.04 + 4.43±00.08	55.62±02.53
D/dmd	3.731±0.086	3.76±00.04 + 0.00±00.00	65.93±01.93
Node.js	4.569±0.094	35.26±00.04 + 1.10±00.09	78.53±01.73
Haskell (MArray)	4.822±0.141	3.85±00.07 + 1.11±00.00	87.24±02.20
Swift	5.678±0.081	13.17±00.06 + 0.00±00.00	128.69±02.71
Lua/luajit	7.852±0.387	2.42±00.03 + 0.00±00.00	142.46±12.07
Ruby/truffleruby (--jvm)	7.983±0.381	338.00±02.50 + 562.25±53.63	282.72±06.68
Ruby/truffleruby	9.986±0.467	278.40±01.31 + 707.98±60.03	264.21±20.52
Python/pypy	13.383±0.487	63.75±00.14 + 46.92±00.05	273.75±15.52
Haskell	14.413±0.224	3.95±00.12 + 0.82±00.17	343.68±08.44
Ruby (--jit)	42.420±1.098	271.26±00.04 + 0.02±00.00	759.89±21.19
Lua	53.184±1.056	2.28±00.03 + 0.00±00.00	929.47±10.98
Elixir	59.667±1.157	76.02±01.32 + 0.00±00.00	1077.70±71.88
Ruby	91.057±2.093	14.72±00.03 + 0.00±00.00	1799.94±149.19
Ruby/jruby	108.798±1.292	185.06±01.75 + 145.62±31.24	2173.96±119.87
Python	166.013±6.278	10.45±00.01 + 0.00±00.00	3395.94±151.28
Tcl (FP)	238.309±4.381	4.31±00.05 + 0.00±00.00	5134.52±121.99
Perl	319.914±2.781	6.59±00.04 + 0.00±00.00	5786.86±30.09
Tcl (OOP)	481.582±5.443	4.35±00.06 + 0.00±00.00	8798.96±234.20

mandel.b

Mandel in Brainfuck

Language	Time, s	Memory, MiB	Energy, J
C++/g++	10.950±0.251	3.48±00.06 + 0.52±00.00	218.96±08.02
C/gcc	12.523±0.183	0.71±00.03 + 1.01±00.02	257.16±05.69
V/gcc	13.707±0.426	1.79±00.04 + 1.03±00.00	285.62±08.07
Rust	13.731±0.171	2.12±00.07 + 0.28±00.03	235.48±05.28
C++/clang++	14.174±0.325	1.52±00.03 + 2.12±00.02	299.19±09.15
Vala/gcc	14.663±0.363	4.18±00.03 + 1.86±00.08	291.60±26.98
D/ldc2	14.976±0.486	3.25±00.04 + 0.77±00.00	283.84±27.12
C/clang	15.470±0.404	0.73±00.00 + 0.98±00.01	323.58±06.07
D/gdc	16.506±0.701	6.85±00.04 + 0.56±00.04	337.21±15.95
Racket (Syntax Objects)	16.950±0.862	112.16±00.33 + 68.29±00.23	361.35±27.86
Kotlin/JVM	17.402±0.433	40.71±00.16 + 1.10±00.29	337.89±11.57
Go	17.836±0.743	3.22±00.02 + 1.26±00.00	324.48±21.86
Nim/gcc	18.680±0.826	1.97±00.03 + 0.51±00.00	372.68±27.56
Zig	18.718±0.372	1.54±00.05 + 0.77±00.00	337.39±20.87
Vala/clang	18.810±0.517	4.16±00.07 + 1.86±00.09	401.28±40.20
Crystal	19.709±0.364	3.45±00.03 + 0.43±00.02	347.61±10.56
Scala	20.175±0.496	77.13±00.16 + 126.70±00.21	413.26±22.62
Java	20.628±0.997	37.38±00.53 + 1.93±00.45	424.03±32.83
Nim/clang	20.684±0.372	2.44±00.05 + 0.54±00.03	432.57±15.62
Go/gccgo	21.434±0.736	22.44±00.13 + 1.28±00.01	441.50±28.10
Swift	21.689±0.758	13.93±00.17 + 0.00±00.00	475.67±23.43
V/clang	22.521±0.469	1.76±00.03 + 0.77±00.00	466.48±25.32
C#/.NET Core	23.313±0.767	33.81±00.10 + 1.14±00.00	475.38±43.97
OCaml	34.780±1.291	4.10±00.04 + 8.86±01.05	683.02±52.59
Racket	39.128±1.913	95.71±00.25 + 0.26±00.00	839.39±90.78
Chez Scheme	42.220±1.691	25.69±00.06 + 3.68±00.02	813.73±36.13
D/dmd	43.995±1.064	3.81±00.05 + 0.77±00.00	902.73±36.53
Lua/luajit	44.276±1.444	2.43±00.04 + 0.44±00.00	936.01±28.17
F#/.NET Core	44.921±1.647	37.84±00.07 + 2.15±00.00	872.03±63.91
C#/Mono	45.631±0.283	20.30±00.07 + 0.89±00.00	795.13±10.84
MLton	47.547±1.556	1.59±00.09 + 4.11±00.00	1030.05±69.26
Node.js	50.279±1.020	35.20±00.05 + 5.20±00.00	942.47±48.01
Julia	51.288±0.979	208.13±00.12 + 0.65±00.01	1124.20±40.28
Python/pypy	63.640±1.713	63.78±00.14 + 47.77±00.01	1295.30±103.67
Haskell (MArray)	69.589±0.591	3.79±00.08 + 2.63±00.03	1285.05±07.56
Ruby/truffleruby (--jvm)	123.667±3.258	335.56±07.72 + 548.28±93.60	2637.61±76.37
Ruby/truffleruby	137.962±3.223	280.17±03.67 + 714.62±46.79	2642.24±170.32
Haskell	221.219±2.389	3.92±00.06 + 26.16±00.00	5035.28±152.83

Base64

Testing base64 encoding/decoding of the large blob into the newly allocated buffers.

Base64

Language	Time, s	Memory, MiB	Energy, J
C/clang (aklomp)	0.137±0.004	2.05±00.02 + 0.00±00.00	3.42±00.15
C/gcc (aklomp)	0.140±0.006	2.03±00.02 + 0.00±00.00	3.54±00.10
Rust	1.106±0.044	2.58±00.07 + 0.01±00.01	24.75±01.65
C/gcc	1.207±0.053	2.01±00.02 + 0.00±00.00	23.13±01.89
C/clang	1.226±0.045	2.00±00.02 + 0.00±00.00	24.68±02.34
V/clang	1.272±0.035	1.93±00.04 + 0.00±00.00	26.59±02.32
Nim/gcc	1.372±0.012	2.31±00.09 + 4.12±00.03	23.48±00.43
Nim/clang	1.507±0.063	2.82±00.07 + 4.38±00.03	33.97±02.72
D/ldc2	1.867±0.024	3.47±00.02 + 3.63±00.00	33.14±00.30
Ruby (--jit)	2.078±0.046	271.68±00.05 + 67.07±00.04	46.75±03.33
Ruby	2.099±0.033	15.13±00.05 + 58.73±00.00	46.06±01.90
Vala/clang	2.187±0.023	5.54±00.03 + 0.44±00.04	44.67±01.14
Java	2.205±0.062	38.96±00.16 + 262.42±04.83	41.97±01.95
Vala/gcc	2.361±0.103	5.58±00.05 + 0.46±00.03	54.23±04.60
C++/clang++ (libcrypto)	2.370±0.072	5.12±00.03 + 0.08±00.00	52.52±03.27
C++/g++ (libcrypto)	2.391±0.115	5.72±00.04 + 0.08±00.00	51.61±04.25
Crystal	2.419±0.010	3.92±00.03 + 1.17±00.01	48.52±00.35
Scala	2.435±0.023	74.49±00.26 + 240.30±16.40	46.46±01.14
Kotlin/JVM	2.451±0.048	41.52±00.68 + 241.74±00.39	57.94±02.46
Go	2.547±0.007	4.40±00.04 + 5.56±00.23	50.55±00.71
Perl (MIME::Base64)	2.825±0.094	14.33±00.04 + 0.07±00.06	55.10±04.96
PHP	2.958±0.047	16.74±00.09 + 0.00±00.00	63.91±04.43
Node.js	3.058±0.046	34.98±00.05 + 36.73±00.09	59.94±01.24
Go/gccgo	3.176±0.009	23.43±00.19 + 8.72±00.28	64.46±00.47
D/gdc	3.455±0.130	7.12±00.05 + 3.41±00.01	66.42±06.31
D/dmd	3.706±0.078	3.94±00.03 + 3.62±00.00	70.75±03.68
Python	4.077±0.137	10.16±00.04 + 0.18±00.00	79.47±08.79
Zig	4.106±0.040	1.50±00.02 + 0.37±00.00	93.20±04.91
Python/pypy	4.727±0.123	63.59±00.10 + 47.59±00.08	106.54±05.62
Julia	4.848±0.060	226.41±00.43 + 52.16±09.93	113.62±03.55
F#/.NET Core	5.150±0.021	38.55±00.03 + 39.05±06.54	88.05±00.95
C#/.NET Core	5.230±0.038	34.58±00.05 + 32.37±03.19	89.02±01.14
Tcl	5.931±0.092	4.88±00.02 + 0.18±00.03	104.96±02.94
Ruby/truffleruby (--jvm)	6.189±0.098	343.99±06.43 + 262.94±42.01	133.48±06.65
C#/Mono	7.444±0.281	20.87±00.06 + 18.47±00.05	168.84±12.40
Ruby/jruby	7.554±0.145	183.80±02.99 + 176.76±03.83	160.35±09.12
Perl (MIME::Base64::Perl)	14.333±0.442	15.62±00.05 + 0.27±00.04	303.59±08.81
Ruby/truffleruby	18.930±0.680	280.37±02.60 + 285.21±20.17	366.82±27.92

Json

Testing parsing and simple calculating of values from a big JSON file.

Few notes:

• gason mutates input strings;
• simdjson requires input strings with batch of trailing zeros: a special zero padding for SIMD instructions;
• DAW JSON Link "NoCheck" skips some JSON structure correctness checks;
• DAW JSON Link, gason, default (not "Precise") RapidJSON, and D implementations except Mir-based have some inaccuracies in number parsing:
  • DAW JSON Link's number parsing issue
  • gason's number parsing issue
  • D stdlib number parsing issue

Json

Language	Time, s	Memory, MiB	Energy, J
C++/g++ (simdjson On-Demand)	0.090±0.001	113.41±00.06 + 59.81±00.00	1.93±00.08
C++/clang++ (simdjson On-Demand)	0.095±0.002	113.07±00.06 + 59.81±00.00	2.11±00.18
C++/g++ (DAW JSON Link NoCheck)	0.096±0.003	113.18±00.05 + 0.00±00.00	2.02±00.11
C++/clang++ (DAW JSON Link NoCheck)	0.103±0.005	112.49±00.02 + 0.00±00.00	2.21±00.25
C++/g++ (DAW JSON Link)	0.104±0.003	113.25±00.03 + 0.00±00.00	2.65±00.18
C++/clang++ (DAW JSON Link)	0.123±0.002	112.52±00.10 + 0.00±00.00	2.56±00.27
C++/clang++ (simdjson DOM)	0.135±0.005	113.03±00.07 + 176.60±00.26	3.19±00.20
C++/g++ (simdjson DOM)	0.136±0.003	113.44±00.01 + 176.60±00.00	3.01±00.10
Rust (Serde Custom)	0.144±0.004	112.00±00.07 + 0.00±00.00	2.97±00.26
Rust (Serde Typed)	0.147±0.005	111.93±00.09 + 11.67±00.13	3.14±00.34
C++/g++ (gason)	0.165±0.006	113.15±00.06 + 96.80±00.03	3.27±00.20
C++/clang++ (gason)	0.191±0.002	112.49±00.05 + 96.97±00.00	3.69±00.09
C++/g++ (RapidJSON)	0.213±0.003	113.23±00.05 + 128.82±00.02	4.11±00.24
D/ldc2 (Mir Asdf DOM)	0.247±0.008	112.83±00.04 + 61.35±00.00	5.01±00.42
D/ldc2 (Mir Amazon's Ion DOM)	0.258±0.008	112.79±00.06 + 16.25±00.00	4.96±00.43
C++/clang++ (RapidJSON)	0.260±0.009	112.53±00.09 + 129.00±00.01	5.90±00.40
C++/g++ (RapidJSON Precise)	0.270±0.005	113.25±00.04 + 128.85±00.04	6.17±00.23
C++/clang++ (RapidJSON Precise)	0.417±0.009	112.58±00.06 + 129.00±00.00	7.63±00.24
C++/g++ (RapidJSON SAX)	0.473±0.021	112.93±00.01 + 0.00±00.00	10.24±00.69
C++/g++ (Boost.JSON)	0.500±0.016	113.21±00.04 + 435.85±00.03	12.05±00.83
Nim/clang (jsony)	0.533±0.014	112.14±00.02 + 42.31±00.03	9.99±00.44
C++/clang++ (RapidJSON SAX)	0.539±0.014	194.79±00.07 + 0.00±00.00	9.65±00.71
Nim/gcc (jsony)	0.556±0.029	111.61±00.05 + 42.34±00.06	12.71±00.92
C++/g++ (RapidJSON SAX Precise)	0.559±0.015	113.00±00.01 + 0.00±00.00	9.78±00.23
C++/clang++ (Boost.JSON)	0.565±0.019	112.55±00.04 + 436.25±00.00	11.17±00.75
Node.js	0.650±0.026	145.06±00.07 + 184.66±00.33	16.07±00.95
Go (jsoniter)	0.657±0.016	231.09±00.09 + 1.43±00.09	13.75±00.74
C++/clang++ (RapidJSON SAX Precise)	0.665±0.029	194.77±00.04 + 0.00±00.00	11.86±00.41
Java (DSL-JSON)	0.737±0.010	263.00±00.49 + 210.85±11.03	18.70±00.72
Python/pypy	0.780±0.032	283.42±00.15 + 123.29±00.00	16.10±02.00
Rust (Serde Untyped)	0.831±0.040	111.96±00.06 + 839.98±00.00	17.55±01.77
V/gcc	0.844±0.029	111.24±00.02 + 496.21±00.06	19.19±01.14
V/clang	0.884±0.020	111.23±00.03 + 496.21±00.06	17.92±01.90
Zig	0.924±0.038	110.80±00.02 + 12.18±00.00	19.69±01.71
Julia (JSON3)	0.924±0.032	462.42±00.29 + 257.55±00.49	17.61±00.72
C#/.NET Core (System.Text.Json)	0.985±0.030	478.82±00.04 + 138.78±00.00	18.62±01.95
Nim/clang (Packedjson)	0.999±0.045	112.74±00.06 + 293.91±00.00	22.41±01.07
Perl (Cpanel::JSON::XS)	1.004±0.038	124.90±00.03 + 402.77±00.00	22.90±01.36
Crystal (Pull)	1.035±0.029	113.89±00.05 + 18.18±00.02	20.58±00.30
Go	1.083±0.047	117.23±00.09 + 83.28±00.25	20.20±01.63
Crystal (Schema)	1.084±0.047	113.86±00.08 + 48.61±00.00	20.99±01.20
Nim/gcc (Packedjson)	1.137±0.035	112.18±00.08 + 293.91±00.00	25.53±01.02
PHP	1.192±0.046	125.95±00.06 + 682.09±00.00	25.15±01.84
Crystal	1.342±0.037	113.88±00.04 + 392.07±00.09	25.04±02.04
Nim/clang	1.456±0.039	112.70±00.04 + 924.80±00.03	31.77±02.54
Clojure	1.536±0.065	450.20±03.25 + 499.05±09.78	40.77±02.65
C++/g++ (json-c)	1.557±0.052	113.30±00.12 + 1215.96±00.00	35.02±02.95
C++/clang++ (Nlohmann)	1.602±0.055	112.66±00.05 + 360.15±00.03	36.01±03.19
C++/clang++ (json-c)	1.617±0.021	112.70±00.01 + 1216.09±00.02	30.62±01.36
Nim/gcc	1.642±0.030	112.18±00.09 + 919.39±00.03	30.55±01.62
C++/g++ (Nlohmann)	1.680±0.077	113.21±00.12 + 447.96±00.08	36.12±01.71
Python	1.703±0.054	120.14±00.02 + 377.33±00.00	39.86±01.82
Go/gccgo	1.705±0.020	137.43±00.26 + 83.45±00.04	31.40±00.86
CPython (UltraJSON)	1.724±0.021	121.89±00.03 + 547.29±01.16	37.53±01.69
C#/.NET Core	1.749±0.034	486.57±00.21 + 294.18±00.01	31.46±01.37
Ruby (--jit)	1.925±0.066	381.21±00.03 + 262.95±00.00	43.85±02.16
Ruby	1.970±0.076	124.65±00.05 + 262.92±00.01	41.77±04.36
F#/.NET Core (System.Text.Json)	2.348±0.099	486.03±00.07 + 452.20±01.80	45.45±04.39
C#/Mono	2.377±0.153	476.10±00.10 + 0.19±00.01	47.15±04.55
D/ldc2	2.442±0.051	113.13±00.09 + 680.13±00.03	49.10±02.48
Scala (uPickle)	2.569±0.085	303.83±00.12 + 714.35±50.07	53.86±02.62
Ruby (YAJL)	2.629±0.043	124.57±00.03 + 279.79±00.00	49.00±02.29
D/gdc	2.899±0.035	116.72±00.02 + 600.57±00.01	64.11±02.12
Haskell	3.490±0.107	115.97±00.03 + 715.06±00.12	77.20±05.47
Rust (jq)	3.699±0.127	113.83±00.01 + 778.38±00.26	80.25±02.99
C++/clang++ (Boost.PropertyTree)	3.867±0.111	194.94±00.05 + 1232.86±00.03	79.39±05.54
Ruby/jruby	3.867±0.072	459.34±07.57 + 1182.21±110.62	95.92±07.95
C++/g++ (Boost.PropertyTree)	4.057±0.156	113.12±00.02 + 1440.14±00.03	82.94±07.71
Vala/clang	5.012±0.164	114.73±00.03 + 932.45±00.02	97.95±08.40
Vala/gcc	5.109±0.192	114.72±00.02 + 996.94±00.04	101.13±09.80
D/dmd	5.449±0.092	113.78±00.02 + 680.12±00.03	99.31±06.83
Perl (JSON::Tiny)	12.389±0.059	125.54±00.06 + 528.63±00.01	219.19±06.26
Ruby/truffleruby (--jvm)	20.231±0.636	518.25±33.14 + 1531.78±51.59	658.38±20.97
Ruby/truffleruby	31.058±1.111	432.55±05.15 + 2239.55±42.82	741.89±33.73

Matmul

Testing allocating and multiplying matrices.

Matmul

Language	Time, s	Memory, MiB	Energy, J
D/ldc2 (lubeck)	0.082±0.003	7.09±00.02 + 55.94±00.12	2.93±00.74
C++/g++ (Eigen)	0.091±0.007	23.69±12.86 + 66.30±12.89	2.80±00.29
Python (NumPy)	0.108±0.006	28.38±00.04 + 57.59±00.04	4.37±00.58
C++/clang++ (Eigen)	0.118±0.016	15.25±10.81 + 75.25±10.75	3.49±00.41
Java (ND4J)	0.118±0.008	137.43±02.38 + 90.75±00.03	3.93±00.56
Nim/clang (Arraymancer)	0.125±0.021	6.58±00.08 + 55.68±00.31	4.04±00.40
Nim/gcc (Arraymancer)	0.140±0.007	5.70±00.11 + 55.84±00.47	4.42±00.80
Julia (threads: 8)	0.154±0.015	249.46±00.31 + 43.40±00.07	5.32±00.65
Julia (threads: 1)	0.206±0.004	249.49±00.27 + 41.87±00.03	4.01±00.11
Julia (no BLAS)	1.274±0.020	223.95±00.24 + 51.44±00.03	29.94±03.39
D/ldc2	2.058±0.047	3.57±00.03 + 70.26±00.13	43.34±01.51
D/dmd	2.201±0.030	3.85±00.14 + 70.27±00.13	46.81±01.61
D/gdc	2.245±0.047	7.14±00.02 + 70.04±00.02	45.11±00.81
Nim/gcc	3.420±0.045	2.70±00.04 + 65.87±00.13	77.88±02.61
C/gcc	3.456±0.085	1.71±00.32 + 68.51±00.23	74.79±02.64
C/clang	3.474±0.090	1.76±00.35 + 68.38±00.35	70.66±01.70
Nim/clang	3.483±0.032	3.09±00.14 + 81.73±04.25	72.13±01.80
Rust	3.515±0.042	2.71±00.17 + 68.32±00.25	73.23±01.49
Zig	3.536±0.038	1.73±00.06 + 68.58±00.00	74.47±02.16
Vala/gcc	3.594±0.063	5.80±00.09 + 68.32±00.00	61.46±01.34
Java	3.607±0.030	38.92±00.12 + 80.58±00.29	73.25±00.95
Swift	3.611±0.096	7.84±00.15 + 68.91±00.02	73.49±01.95
Go/gccgo	3.645±0.080	22.57±00.15 + 73.47±00.14	78.64±02.77
Go	3.661±0.090	3.54±00.06 + 73.31±00.26	83.72±02.83
Vala/clang	3.711±0.091	5.79±00.06 + 68.32±00.00	59.58±03.05
V/clang	3.726±0.071	2.31±00.02 + 68.84±00.00	84.50±02.34
Scala	3.790±0.045	75.11±00.21 + 144.23±00.43	79.63±01.18
Crystal	3.818±0.101	4.30±00.04 + 59.67±00.01	83.75±06.03
V/gcc	3.856±0.057	1.94±00.06 + 68.84±00.00	77.54±04.86
Kotlin/JVM	4.037±0.033	40.09±00.14 + 80.13±00.28	78.12±01.49
Node.js	4.059±0.055	40.39±00.07 + 70.68±00.13	77.87±05.83
Python/pypy	6.603±0.058	64.86±00.07 + 69.14±00.05	117.64±03.61
C#/.NET Core	7.545±0.085	33.17±00.05 + 69.46±00.00	128.54±03.88
C#/Mono	11.652±0.080	20.27±00.04 + 69.01±00.01	214.67±21.08
Ruby/truffleruby	25.218±0.727	315.64±07.64 + 877.79±215.04	548.71±51.22
Ruby/truffleruby (--jvm)	44.597±0.947	392.42±17.45 + 745.43±116.26	1007.27±37.75
Ruby (--jit)	202.034±6.199	272.42±00.02 + 68.84±00.00	3820.61±263.34
Ruby	222.467±6.667	15.83±00.06 + 68.58±00.00	4145.66±315.71
Python	238.462±7.000	10.52±00.02 + 68.58±00.00	4496.58±343.38
Tcl	321.774±7.723	7.21±00.01 + 400.44±00.00	6297.82±475.82
Perl	402.410±4.590	9.04±00.03 + 599.72±00.08	8259.27±234.35
Ruby/jruby	535.502±17.283	260.31±03.16 + 1172.21±31.18	10768.71±199.71

Primes

Testing:

• generating primes using the optimized sieve of Atkin;
• prefix search for their decimal numbers using Trie data structure.

Primes

Language	Time, s	Memory, MiB	Energy, J
Zig	0.074±0.001	1.42±00.02 + 55.56±02.12	1.59±00.07
Rust	0.114±0.003	2.35±00.09 + 77.02±00.00	2.40±00.14
Java	0.163±0.003	38.34±00.21 + 102.58±00.41	4.87±00.05
Crystal	0.170±0.002	3.53±00.08 + 88.55±00.03	3.73±00.04
C++/clang++	0.174±0.002	3.16±00.05 + 75.41±00.01	3.79±00.20
C++/g++	0.184±0.008	3.52±00.03 + 84.80±00.00	3.64±00.26
V/clang	0.187±0.003	1.72±00.04 + 270.91±03.22	3.60±00.11
V/gcc	0.189±0.009	1.75±00.05 + 263.56±04.06	3.82±00.40
Node.js	0.270±0.007	34.45±00.08 + 176.28±01.54	7.12±00.08
Lua/luajit	0.404±0.015	2.56±00.06 + 156.05±00.57	8.21±00.68
Scala	0.416±0.005	75.49±00.09 + 230.85±01.07	13.05±00.56
Python/pypy	0.836±0.018	63.07±00.16 + 251.34±00.04	18.52±00.48
Lua	1.656±0.021	2.27±00.04 + 284.17±00.48	34.45±00.71
Ruby (--jit)	1.851±0.018	271.23±00.02 + 147.76±00.00	31.36±00.73
Ruby/truffleruby	1.959±0.047	276.10±03.31 + 418.65±56.56	52.68±05.10
Ruby/truffleruby (--jvm)	2.089±0.069	334.89±04.28 + 472.94±77.31	76.26±01.68
Ruby	2.356±0.101	14.69±00.06 + 147.65±00.00	46.89±05.81
Ruby/jruby	2.852±0.054	182.88±01.82 + 431.31±33.87	69.85±06.73
Python	4.629±0.151	10.32±00.04 + 235.70±00.90	95.65±04.32

Tests Execution

Environment

CPU: Intel(R) Core(TM) i7-10710U

Base Docker image: Debian GNU/Linux bookworm/sid

Language	Version
.NET Core	6.0.102
C#/.NET Core	4.0.1-1.22053.6 (9942dc95)
C#/Mono	6.12.0.122
Chez Scheme	9.5.4
Clojure	"1.10.3"
Crystal	1.3.2
D/dmd	v2.098.1
D/gdc	11.2.0
D/ldc2	1.28.1
Elixir	1.12.2
F#/.NET Core	12.0.0.0 for F# 6.0
Go	go1.17.7
Go/gccgo	11.2.0
Haskell	9.0.1
Java	17.0.2
Julia	v"1.7.2"
Kotlin	1.6.10
Lua	5.4.4
Lua/luajit	2.1.0-beta3
MLton	20210117
Nim	1.6.4
Node.js	v17.5.0
OCaml	4.13.1
PHP	8.1.2
Perl	v5.34.0
Python	3.9.10
Python/pypy	7.3.7-final0 for Python 3.8.12
Racket	"8.4"
Ruby	3.1.0p0
Ruby/jruby	9.3.3.0
Ruby/truffleruby	22.0.0.2
Rust	1.58.1
Scala	3.1.1
Swift	5.5.3
Tcl	8.6
V	0.2.4 efdbe9d
Vala	0.54.6
Zig	0.9.0
clang/clang++	13.0.1
gcc/g++	11.2.0

Using Docker

Build the image:

$ docker build docker/ -t benchmarks

Run the image:

$ docker run -it --rm -v $(pwd):/src benchmarks <cmd>

where <cmd> is:

• versions (print installed language versions);
• shell (start the shell);
• brainfuck bench (build and run Brainfuck bench.b benchmarks);
• brainfuck mandel (build and run Brainfuck mandel.b benchmarks);
• base64 (build and run Base64 benchmarks);
• json (build and run Json benchmarks);
• matmul (build and run Matmul benchmarks);
• primes (build and run Primes benchmarks);

Please note that the actual measurements provided in the project are taken semi-manually (via shell) as the full update takes days and could have occassional issues in Docker.

There is a Makefile that could be used to simplify Docker usage:

• make build (build the image);
• make versions (run the image with the versions command);
• make shell (run the image with the `shell' command);
• make toc (utility rule to update ToC in this README, requires git-markdown-toc available in PATH).

Please note that the make shell rule requires cpupower utility installed that is invoked with sudo to set cpufreq's performance governon (it runs the CPU at the maximum frequence to eliminate throttling issues).

Manual Execution

Makefiles contain recipes for building and executing tests with the proper dependencies. Please use make run (and make run2 where applicable). The measurements are taken using analyze.rb script:

$ cd <test suite>
$ ../analyze.rb make run
$ ../analyze.rb make run[<single test>]

Please note that the measurements could take hours. It uses 10 iterations by default, but it could be changed using ATTEMPTS environment variable:

$ ATTEMPTS=1 ../analyze.rb make run

Prerequisites

Please use Dockerfile as a reference regarding which packages and tools are required.

For all (optional):

• Powercap for reading energy counters in Linux (Debian package powercap-utils).

For Python:

• NumPy for matmul tests (Debian package python3-numpy).
• UltraJSON for JSON tests (Debian package python3-ujson).

For C++:

• Boost for JSON tests (Debian package libboost-dev).
• JSON-C for JSON tests (Debian package libjson-c-dev).

For Rust:

• libjq for jq test (Debian packages libjq-dev, libonig-dev and environment variable JQ_LIB_DIR=/usr/lib/x86_64-linux-gnu/).

For Java, Scala:

• Coursier for downloading Maven artifacts.

For Haskell:

• network for TCP connectivity between the tests and the test runner.
• raw-strings-qq for raw string literals used in tests.

For Perl:

• cpanminus for installing modules from CPAN (Debian package cpanminus).

For Vala:

• JSON-GLib for JSON tests (Debian package libjson-glib-dev).

Contribution

Please follow the criteria specified in the overview. Besides that please ensure that the communication protocol between a test and the test runner is satisfied:

• The test runner listens on localhost:9001;
• All messages are sent using TCP sockets closed immediately after the message has been sent;
• There are two messages sent from a test (it establishes the measurement boundary):
  1. The beginning message having the format name of the test/tprocess ID (the process ID is used to measure the memory consumption). Please note that the name of the test couldn't use Tab character as it's a delimiter;
  2. The end message with any content (mostly it's "stop" for consistency).
• The test runner could be unavailable (if the test is launched as is) and the test should gracefully handle it.

Makefile guide

Binary executables

If the test is compiled into a single binary, then two sections of the Makefile require changes:

• append a new target (the final binary location) into executables variable;
• append the proper target rule.

Compiled artifacts

If the test is compiled, but can't be executed directly as a binary, then three sections of the Makefile require changes:

• append a new target (the final artifact location) into artifacts variable;
• append the proper target rule to compile the test;
• append run[<target_artifact>] rule to run the test.

Scripting language

If the test doesn't require compilation, then two sections of the Makefile requires changes:

• append run[<script_file>] into all_runners variable;
• append run[<script_file>] rule to run the test.