WillAbides / benchmarks

Some benchmarks of different languages

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Table of Content

Overview

The benchmarks follow the criteria:

  • They are written as the average software developer would write them, i.e.

    • The algorithms are implemented as cited in public sources;
    • The libraries are used as described in the tutorials, documentation and examples;
    • The used data structures are idiomatic.
  • The used algorithms are similar between the languages (as the reference implementations), variants are acceptable if the reference implementation exists.

  • All final binaries are releases (optimized for performance if possible) as debug performance may vary too much depending on the compiler.

My other benchmarks: jit-benchmarks, crystal-benchmarks-game

Measurements

The measured values are:

  • time spent for the benchmark execution (loading required data and code self-testing are not measured);
  • memory consumption of the benchmark process, reported as base + increase, where base is the RSS before the benchmark and increase is the peak increase of the RSS during the benchmark;
  • energy consumption of the CPU package during the benchmark: PP0 (cores) + PP1 (uncores like GPU) + DRAM. Currently, only Intel CPU are supported via the powercap interface.

All values are presented as: median±median absolute deviation.

UPDATE: 2023-10-16

Test Cases

Brainfuck

Testing brainfuck implementations using two code samples (bench.b and mandel.b). Supports two mode:

  • Verbose (default). Prints the output immediately.
  • Quiet (if QUIET environment variable is set). Accumulates the output using Fletcher-16 checksum, and prints it out after the benchmark.

Brainfuck

bench.b

Language Time, s Memory, MiB Energy, J
Scala (Staged) 0.500±0.011 214.99±04.56 + 21.33±01.86 28.38±02.06
Racket (Staged) 1.300±0.000 100.79±00.40 + 0.00±00.00 47.18±00.05
C++/g++ 1.323±0.000 1.84±00.03 + 0.00±00.00 52.88±00.06
Java 1.582±0.000 39.81±00.08 + 1.19±00.04 61.57±00.09
Nim/gcc 1.612±0.000 0.92±00.01 + 0.00±00.00 64.40±00.05
C++/clang++ 1.642±0.000 1.62±00.02 + 0.00±00.00 63.90±00.13
V/gcc 1.649±0.002 1.84±00.03 + 0.00±00.00 63.31±00.62
Rust 1.658±0.000 0.92±00.03 + 0.00±00.00 64.18±00.04
D/gdc 1.658±0.000 6.34±00.02 + 0.00±00.00 68.09±00.23
D/ldc2 1.664±0.000 1.39±00.01 + 0.00±00.00 66.39±00.05
C/gcc 1.667±0.000 0.85±00.03 + 0.00±00.00 64.65±00.11
C/clang 1.674±0.000 0.89±00.02 + 0.00±00.00 64.93±00.16
Kotlin/JVM 1.689±0.000 43.17±00.06 + 0.79±00.27 65.90±00.17
Zig 1.790±0.001 0.92±00.02 + 0.00±00.00 70.28±00.46
Go 1.900±0.000 3.08±00.03 + 0.00±00.00 73.52±00.12
Chez Scheme 1.932±0.001 24.74±00.03 + 4.47±00.11 79.80±00.06
F#/.NET Core 1.959±0.003 36.38±00.11 + 0.58±00.00 80.39±00.23
C#/.NET Core 1.979±0.000 31.68±00.15 + 0.20±00.00 81.21±00.02
OCaml 1.997±0.001 3.24±00.04 + 2.40±00.03 91.15±00.11
Racket 2.027±0.022 93.25±00.28 + 20.64±00.63 81.62±00.91
Nim/clang 2.050±0.001 1.14±00.03 + 0.00±00.00 80.91±00.74
Vala/gcc 2.083±0.000 4.56±00.06 + 0.00±00.00 78.05±00.05
Vala/clang 2.259±0.000 4.58±00.03 + 0.00±00.00 84.70±00.11
Go/gccgo 2.331±0.000 23.56±00.03 + 0.00±00.00 94.42±00.99
Crystal 2.345±0.001 2.95±00.01 + 0.00±00.00 93.56±00.08
V/clang 2.511±0.038 1.84±00.01 + 0.00±00.00 106.19±00.94
MLton 2.608±0.000 1.61±00.05 + 0.25±00.00 104.24±01.38
C#/Mono 2.685±0.000 25.41±00.08 + 0.00±00.00 108.49±01.15
Julia 2.858±0.002 249.20±00.09 + 0.35±00.03 110.50±00.26
D/dmd 3.128±0.001 3.40±00.04 + 0.00±00.00 117.91±00.51
Scala 3.277±0.004 72.46±00.13 + 186.43±00.26 138.10±00.61
Haskell (MArray) 3.861±0.001 5.81±00.03 + 4.84±00.00 157.16±01.03
Node.js 4.160±0.004 38.00±00.02 + 3.94±00.03 166.25±01.10
Haskell (FP) 4.183±0.002 5.99±00.02 + 4.99±00.03 177.11±00.22
Swift 5.479±0.000 16.14±00.09 + 0.00±00.00 205.37±00.21
Ruby/truffleruby 5.733±0.233 224.91±04.25 + 647.13±70.73 278.60±11.01
Lua/luajit 6.489±0.005 2.44±00.01 + 0.00±00.00 259.08±00.13
Ruby/truffleruby (JVM) 6.637±0.177 398.30±18.36 + 602.85±86.95 317.74±07.84
Python/pypy 12.214±0.008 59.50±00.13 + 29.66±00.18 510.88±01.26
Idris 15.676±0.024 20.65±00.06 + 8.82±00.05 686.28±01.00
Elixir 23.946±0.021 70.45±00.70 + 0.00±00.00 942.72±02.80
Lua 51.056±0.422 2.23±00.04 + 0.00±00.00 1973.41±15.02
Ruby (--jit) 54.186±0.425 16.16±00.02 + 1.79±00.01 2302.73±18.18
PHP 55.973±0.024 17.81±00.08 + 0.00±00.00 2181.36±02.17
Ruby 78.429±0.569 14.95±00.02 + 0.00±00.00 3302.47±21.59
Ruby/jruby 85.643±1.154 197.30±04.27 + 212.10±11.71 3804.72±62.43
Python 119.026±0.454 10.14±00.03 + 0.00±00.00 4666.45±30.38
Tcl (FP) 270.164±2.350 3.93±00.04 + 0.00±00.00 11304.53±116.31
Perl 316.357±2.409 7.05±00.06 + 0.00±00.00 12793.56±88.59
Tcl (OOP) 530.477±4.818 3.93±00.03 + 0.00±00.00 22275.06±348.47

mandel.b

Mandel in Brainfuck

Language Time, s Memory, MiB Energy, J
C++/g++ 10.158±0.026 1.80±00.02 + 2.29±00.04 414.88±04.38
Java 14.106±0.016 39.81±00.09 + 2.27±00.07 561.52±04.28
C/gcc 14.185±0.004 0.86±00.01 + 0.83±00.05 563.74±02.93
Racket (Staged) 14.379±0.428 100.46±00.16 + 76.88±02.04 572.08±14.26
Kotlin/JVM 14.396±0.014 42.93±00.21 + 2.18±00.22 592.87±01.55
Rust 15.057±0.014 0.90±00.01 + 1.10±00.02 594.38±02.79
Scala (Staged) 15.065±0.646 216.61±02.00 + 103.20±07.08 742.04±18.57
D/gdc 15.070±0.006 6.27±00.02 + 1.44±00.02 636.33±01.52
Zig 15.107±0.007 0.90±00.02 + 1.41±00.01 622.38±01.04
C/clang 15.113±0.005 0.86±00.01 + 0.88±00.03 648.78±04.04
C++/clang++ 15.135±0.011 1.59±00.02 + 1.96±00.01 621.55±02.98
D/ldc2 15.319±0.005 3.01±00.03 + 0.79±00.02 613.95±00.68
Go 15.413±0.141 3.04±00.07 + 1.26±00.00 616.56±06.53
C#/.NET Core 16.331±0.009 31.53±00.09 + 0.88±00.00 691.29±02.05
Crystal 16.589±0.353 2.87±00.04 + 0.72±00.04 689.62±13.84
Vala/gcc 18.058±0.007 4.40±00.04 + 1.19±00.03 698.11±00.81
V/gcc 18.122±0.068 1.80±00.02 + 1.16±00.01 702.84±01.94
Swift 18.141±0.044 15.91±00.04 + 0.00±00.00 745.20±01.74
Nim/gcc 18.170±0.031 2.00±00.01 + 1.29±00.00 768.06±03.91
Vala/clang 19.705±0.003 4.39±00.04 + 1.21±00.02 763.62±00.28
Nim/clang 19.914±0.585 2.26±00.04 + 1.29±00.00 819.31±20.88
Go/gccgo 20.479±0.021 23.90±00.09 + 1.28±00.01 867.17±00.59
Scala 20.991±0.024 72.57±00.12 + 137.46±00.42 885.00±01.03
V/clang 22.603±0.245 1.83±00.01 + 1.14±00.02 985.37±15.29
OCaml 26.617±0.004 3.97±00.01 + 3.47±00.06 1284.03±06.98
Chez Scheme 27.772±0.070 25.50±00.05 + 3.68±00.01 1221.71±03.21
Node.js 28.483±0.392 38.93±00.06 + 6.70±00.12 1209.63±15.82
Julia 30.379±0.066 250.43±00.03 + 0.41±00.01 1160.03±01.94
C#/Mono 31.446±0.014 25.27±00.06 + 0.82±00.00 1337.71±01.79
F#/.NET Core 34.935±0.027 36.02±00.12 + 2.14±00.02 1480.47±04.19
Lua/luajit 35.086±0.061 2.42±00.02 + 0.43±00.00 1406.88±05.61
Racket 37.036±0.513 92.86±00.08 + 22.57±00.15 1640.19±13.64
Haskell (MArray) 37.162±0.014 5.61±00.01 + 5.82±00.00 1523.81±05.62
D/dmd 38.423±0.006 3.25±00.04 + 0.86±00.03 1412.70±09.18
MLton 44.897±0.052 1.65±00.03 + 4.11±00.00 1924.64±09.32
Python/pypy 48.246±0.092 59.57±00.07 + 30.36±00.08 2077.33±19.41
Ruby/truffleruby 48.288±0.912 222.51±09.39 + 557.98±63.76 2344.82±60.98
Ruby/truffleruby (JVM) 54.733±0.355 388.41±10.42 + 460.00±43.85 2492.14±26.26
Idris 67.241±0.176 21.86±00.06 + 9.54±00.00 2893.54±10.95
Haskell (FP) 80.301±0.045 5.75±00.03 + 75.70±00.00 3338.80±16.07

Base64

Testing base64 encoding/decoding of the large blob into the newly allocated buffers.

Base64

Language Time, s Memory, MiB Energy, J
C/clang (aklomp) 0.096±0.000 2.07±00.04 + 0.00±00.00 4.59±00.07
C/gcc (aklomp) 0.099±0.000 2.08±00.02 + 0.00±00.00 4.69±00.05
PHP 0.106±0.000 18.51±00.04 + 0.00±00.00 4.98±00.02
Go 0.306±0.006 6.29±00.05 + 4.62±00.35 14.18±00.25
Rust 0.956±0.000 2.31±00.03 + 0.00±00.00 39.05±00.21
C/clang 0.997±0.000 1.98±00.07 + 0.00±00.00 36.78±00.05
D/ldc2 1.095±0.009 3.58±00.04 + 3.41±00.00 46.71±00.59
C/gcc 1.098±0.001 2.00±00.04 + 0.00±00.00 40.28±00.05
Crystal 1.102±0.001 3.57±00.02 + 1.27±00.03 45.33±00.27
Nim/clang 1.471±0.001 1.92±00.03 + 5.83±00.06 58.38±00.37
Java 1.508±0.002 40.71±00.09 + 209.37±12.91 60.15±00.52
V/clang 1.563±0.000 2.36±00.04 + 2388.26±00.77 59.06±00.12
Nim/gcc 1.575±0.001 1.66±00.02 + 5.25±00.03 64.12±00.20
Scala 1.578±0.001 68.33±00.19 + 318.32±05.49 64.35±00.22
V/gcc 1.584±0.001 2.32±00.04 + 2386.10±00.91 58.12±00.34
Kotlin/JVM 1.635±0.009 43.84±00.04 + 250.75±01.85 66.17±00.82
Vala/gcc 1.644±0.002 5.69±00.01 + 0.01±00.00 62.83±00.12
Vala/clang 1.645±0.001 5.61±00.04 + 0.00±00.00 63.55±00.56
Ruby (--jit) 1.670±0.002 16.79±00.02 + 41.32±02.06 65.08±00.30
Ruby 1.678±0.001 15.57±00.03 + 40.68±00.33 65.41±00.13
C++/clang++ (libcrypto) 1.717±0.002 5.03±00.05 + 0.68±00.04 69.64±00.50
C++/g++ (libcrypto) 1.717±0.003 5.54±00.08 + 0.70±00.03 69.18±00.27
Node.js 1.729±0.005 38.73±00.03 + 37.05±00.10 71.17±00.74
Go 1.784±0.002 4.17±00.04 + 4.48±00.39 75.34±00.29
Perl (MIME::Base64) 1.880±0.004 14.81±00.06 + 0.12±00.05 74.45±00.50
F#/.NET Core 2.339±0.035 36.42±00.08 + 43.54±01.89 87.52±01.18
D/gdc 2.383±0.001 7.29±00.04 + 3.35±00.00 105.29±00.61
C#/.NET Core 2.605±0.016 31.59±00.09 + 50.83±09.00 95.85±00.78
D/dmd 2.733±0.001 3.53±00.04 + 3.35±00.00 119.71±00.60
Python 3.186±0.013 10.23±00.02 + 0.09±00.00 125.50±00.65
Zig 3.196±0.003 1.52±00.04 + 0.00±00.00 125.26±00.49
Python/pypy 3.580±0.001 59.43±00.07 + 31.58±00.10 157.31±00.19
Tcl 3.624±0.003 5.05±00.04 + 0.00±00.00 148.02±00.31
Go/gccgo 3.674±0.001 24.81±00.12 + 8.08±00.25 166.56±00.23
Racket 3.903±0.036 91.21±00.25 + 19.37±00.50 155.49±01.84
Ruby/truffleruby (JVM) 3.967±0.036 391.81±10.87 + 298.87±50.79 201.52±02.09
C#/Mono 4.775±0.001 26.15±00.06 + 18.66±00.07 196.42±00.82
Julia 4.827±0.001 266.49±00.12 + 43.51±00.12 181.97±00.63
Ruby/jruby 10.655±0.215 190.12±01.70 + 91.63±06.55 428.25±09.08
Ruby/truffleruby 12.337±0.012 226.13±03.32 + 492.67±49.16 546.38±03.47
Perl (MIME::Base64::Perl) 13.588±0.098 16.08±00.04 + 0.28±00.09 574.28±06.82

Json

Testing parsing and simple calculating of values from a big JSON file.

Few notes:

Json

Language Time, s Memory, MiB Energy, J
C++/clang++ (simdjson On-Demand) 0.059±0.000 112.62±00.29 + 59.80±00.25 2.48±00.01
C++/g++ (simdjson On-Demand) 0.060±0.000 113.51±00.06 + 59.81±00.00 2.56±00.02
C++/clang++ (DAW JSON Link NoCheck) 0.083±0.000 112.36±00.07 + 0.00±00.00 3.38±00.02
C++/g++ (DAW JSON Link NoCheck) 0.083±0.000 113.16±00.06 + 0.00±00.00 3.41±00.03
C++/clang++ (DAW JSON Link) 0.084±0.000 112.42±00.01 + 0.00±00.00 3.53±00.01
C++/g++ (DAW JSON Link) 0.087±0.000 113.11±00.07 + 0.00±00.00 3.66±00.02
Rust (Serde Custom) 0.096±0.000 111.31±00.03 + 0.00±00.00 4.11±00.04
Rust (Serde Typed) 0.104±0.000 111.67±00.03 + 11.77±00.00 4.43±00.01
C++/clang++ (simdjson DOM) 0.123±0.000 112.83±00.13 + 176.09±00.13 5.33±00.03
C++/g++ (simdjson DOM) 0.128±0.000 113.51±00.04 + 176.60±00.00 5.60±00.07
C++/clang++ (gason) 0.138±0.000 112.37±00.04 + 96.97±00.06 5.62±00.03
D/ldc2 (Mir Asdf DOM) 0.138±0.000 112.85±00.03 + 61.10±00.00 5.78±00.06
C++/g++ (gason) 0.138±0.000 113.13±00.05 + 96.93±00.05 5.50±00.02
C++/g++ (RapidJSON) 0.157±0.000 113.11±00.02 + 128.90±00.02 6.70±00.06
Scala (jsoniter-scala) 0.158±0.002 291.61±00.13 + 16.65±00.51 8.50±00.11
Go (rjson custom) 0.196±0.000 115.18±00.04 + 0.00±00.00 7.63±00.02
Zig 0.221±0.000 110.91±00.01 + 39.02±00.03 9.87±00.04
Go (rjson) 0.221±0.000 115.23±00.07 + 68.03±00.00 8.68±00.02
Go (Sonic) 0.225±0.002 123.66±00.07 + 111.83±00.23 9.79±00.15
C++/clang++ (RapidJSON) 0.226±0.000 112.42±00.03 + 128.97±00.03 9.46±00.06
D/ldc2 (Mir Amazon's Ion DOM) 0.231±0.000 112.87±00.02 + 80.70±00.00 9.62±00.05
C++/g++ (RapidJSON Precise) 0.233±0.000 113.17±00.05 + 128.33±00.59 10.02±00.06
Go (goccy/go-json) 0.269±0.000 115.61±00.15 + 112.20±00.14 10.66±00.04
C++/clang++ (RapidJSON Precise) 0.302±0.000 112.46±00.04 + 128.97±00.03 13.00±00.06
C/gcc (yajl) 0.374±0.001 110.83±00.03 + 0.00±00.00 16.15±00.07
C/clang (yajl) 0.374±0.000 110.89±00.04 + 0.00±00.00 16.29±00.09
C++/g++ (Boost.JSON) 0.380±0.000 113.27±00.01 + 308.09±00.05 15.91±00.04
C++/g++ (RapidJSON SAX) 0.388±0.000 112.91±00.03 + 0.00±00.00 16.51±00.02
Nim/clang (jsony) 0.391±0.000 111.38±00.02 + 146.18±00.03 16.73±00.07
C++/clang++ (Boost.JSON) 0.395±0.001 112.52±00.03 + 308.18±00.00 16.62±00.04
Nim/gcc (jsony) 0.407±0.000 111.10±00.01 + 156.78±01.00 17.43±00.16
C++/g++ (RapidJSON SAX Precise) 0.445±0.001 112.97±00.01 + 0.00±00.00 19.60±00.06
Go (jsoniter) 0.520±0.001 227.75±00.08 + 1.78±00.21 22.21±00.23
Rust (Serde Untyped) 0.547±0.001 111.65±00.01 + 839.98±00.00 23.09±00.15
C#/.NET Core (System.Text.Json) 0.573±0.001 488.19±00.29 + 139.73±00.02 25.40±00.12
C++/clang++ (RapidJSON SAX) 0.585±0.000 194.66±00.03 + 0.00±00.00 23.44±00.03
Julia (JSON3) 0.615±0.003 462.74±00.18 + 224.58±01.29 26.04±00.29
Java (DSL-JSON) 0.633±0.022 262.57±00.10 + 198.23±00.93 32.50±00.48
V/clang 0.634±0.000 111.43±00.01 + 496.21±00.06 26.61±00.19
V/gcc 0.638±0.000 111.40±00.03 + 496.21±00.00 26.68±00.26
Node.js 0.646±0.004 150.07±00.04 + 196.56±01.49 31.69±00.18
Crystal (Pull) 0.648±0.002 113.25±00.01 + 18.39±00.03 29.04±00.18
Python/pypy 0.668±0.001 279.67±00.01 + 125.79±00.07 28.63±00.11
Crystal (Schema) 0.669±0.002 113.24±00.04 + 48.84±00.06 30.10±00.21
C++/clang++ (RapidJSON SAX Precise) 0.695±0.001 194.65±00.05 + 0.00±00.00 28.76±00.07
Nim/gcc (Packedjson) 0.791±0.001 111.83±00.02 + 294.16±00.00 32.60±00.13
Perl (Cpanel::JSON::XS) 0.796±0.008 125.50±00.07 + 402.87±00.02 33.22±00.36
PHP 0.844±0.000 127.86±00.21 + 517.86±00.00 36.27±00.19
Go 0.887±0.002 115.35±00.10 + 104.95±00.22 36.93±00.10
Nim/clang (Packedjson) 0.899±0.002 112.14±00.01 + 294.16±00.00 36.99±00.21
Crystal 0.950±0.011 113.24±00.01 + 392.50±00.00 41.49±00.82
C#/.NET Core 1.084±0.012 496.41±00.12 + 271.70±00.01 48.39±00.47
Nim/gcc 1.116±0.002 111.86±00.03 + 1001.34±00.00 46.65±00.27
C++/clang++ (json-c) 1.163±0.004 112.67±00.02 + 1216.08±00.00 48.79±00.78
C++/g++ (json-c) 1.164±0.005 113.30±00.06 + 1216.02±00.06 49.22±00.81
Clojure 1.246±0.014 460.37±07.51 + 546.75±16.98 66.52±01.33
Nim/clang 1.277±0.001 112.14±00.01 + 999.02±00.00 52.22±00.36
Go/gccgo 1.286±0.001 138.88±00.10 + 83.51±00.03 55.05±00.31
CPython (UltraJSON) 1.337±0.001 122.63±00.05 + 495.71±02.58 49.85±00.19
C++/clang++ (Nlohmann) 1.343±0.006 112.63±00.05 + 360.14±00.00 57.01±00.32
Python 1.391±0.002 120.04±00.02 + 326.36±00.04 55.39±00.55
Ruby 1.452±0.002 125.12±00.04 + 261.61±00.07 61.45±00.35
Ruby (--jit) 1.539±0.008 126.19±00.04 + 263.91±00.01 65.79±00.58
F#/.NET Core (System.Text.Json) 1.576±0.004 498.64±00.26 + 229.56±04.48 69.97±00.45
C++/g++ (Nlohmann) 1.611±0.005 113.26±00.07 + 448.03±00.03 66.24±00.72
C#/Mono 1.802±0.028 253.15±00.17 + 31.50±00.00 78.74±01.13
Ruby (YAJL) 1.808±0.004 125.06±00.04 + 276.09±00.04 75.84±00.58
D/ldc2 2.044±0.006 112.68±00.01 + 680.39±00.04 85.05±00.13
Haskell 2.257±0.006 117.34±00.01 + 725.38±00.23 94.48±00.60
Rust (jq) 2.596±0.001 113.53±00.04 + 903.76±01.16 108.96±00.28
Ruby/jruby 2.986±0.024 456.73±06.29 + 913.88±75.59 154.96±02.12
C++/g++ (Boost.PropertyTree) 3.065±0.003 113.12±00.04 + 1440.12±00.00 128.54±01.53
C++/clang++ (Boost.PropertyTree) 3.092±0.005 194.90±00.04 + 1232.84±00.00 128.35±00.98
Vala/clang 3.326±0.012 114.98±00.01 + 980.04±00.01 144.32±01.14
Vala/gcc 3.327±0.003 114.98±00.08 + 980.04±00.01 143.32±01.85
D/gdc 3.612±0.016 116.49±00.05 + 681.07±00.10 155.27±00.75
Racket 3.779±0.036 220.86±00.19 + 263.51±25.06 156.87±01.63
D/dmd 4.453±0.006 113.07±00.03 + 680.43±00.05 179.34±00.49
Perl (JSON::Tiny) 9.657±0.113 126.14±00.09 + 528.61±00.05 423.18±04.31
Ruby/truffleruby 10.672±0.121 466.47±23.50 + 1900.14±89.47 619.64±07.46
Ruby/truffleruby (JVM) 10.982±0.124 498.64±10.98 + 2558.62±197.73 687.46±08.96

Matmul

Testing allocating and multiplying matrices.

Matmul

Language Time, s Memory, MiB Energy, J
D/ldc2 (lubeck) 0.042±0.000 6.08±00.05 + 57.76±00.05 4.44±00.02
V/gcc (VSL + CBLAS) 0.046±0.000 6.62±00.02 + 58.28±00.00 4.65±00.03
V/clang (VSL + CBLAS) 0.046±0.000 6.66±00.01 + 58.29±00.00 4.65±00.03
Nim/gcc (Arraymancer) 0.063±0.004 5.46±00.06 + 57.49±00.12 5.32±00.22
Python (NumPy) 0.064±0.000 31.77±00.13 + 58.54±00.10 6.19±00.02
C++/g++ (Eigen) 0.065±0.003 28.29±10.15 + 61.41±10.18 4.90±00.13
C++/clang++ (Eigen) 0.065±0.003 30.17±09.25 + 60.15±09.15 5.48±00.18
Nim/clang (Arraymancer) 0.068±0.001 6.04±00.07 + 57.60±00.18 5.90±00.06
Java (ND4J) 0.076±0.001 112.18±02.20 + 92.14±00.02 5.98±00.05
Julia (threads: 2) 0.083±0.000 283.67±00.03 + 57.16±00.06 5.25±00.02
Rust (ndarray) 0.085±0.000 2.37±00.05 + 68.47±00.00 5.99±00.04
Julia (threads: 1) 0.133±0.000 283.77±00.18 + 56.76±00.06 6.62±00.04
V/clang (VSL) 0.272±0.004 7.33±00.07 + 51.56±00.00 19.00±00.23
V/gcc (VSL) 0.505±0.002 7.05±00.07 + 51.82±00.00 36.74±00.18
Julia (no BLAS) 1.018±0.004 267.67±00.06 + 51.50±00.00 45.38±00.67
D/ldc2 1.717±0.001 3.25±00.01 + 70.47±00.02 63.31±00.18
D/gdc 1.869±0.001 7.30±00.03 + 70.16±00.01 73.12±00.03
D/dmd 1.882±0.002 3.17±00.02 + 70.49±00.02 70.93±00.15
C/gcc 3.026±0.001 1.50±00.05 + 68.66±00.02 111.93±00.58
V/gcc 3.038±0.001 2.49±00.06 + 68.58±00.00 112.89±00.08
Vala/clang 3.056±0.000 5.48±00.04 + 68.32±00.00 104.92±00.17
V/clang 3.058±0.000 2.79±00.05 + 68.58±00.00 104.95±00.09
C/clang 3.060±0.000 1.49±00.01 + 68.70±00.00 104.77±00.04
Rust 3.062±0.000 2.10±00.02 + 68.57±00.00 105.83±00.47
Zig 3.070±0.001 1.79±00.05 + 68.58±00.00 108.93±00.04
Swift 3.090±0.000 7.91±00.03 + 68.75±00.01 110.70±00.57
Nim/gcc 3.093±0.001 2.54±00.04 + 57.75±00.00 114.70±00.43
Nim/clang 3.117±0.001 2.79±00.01 + 64.97±03.74 107.58±00.51
Vala/gcc 3.121±0.000 5.32±00.20 + 68.32±00.00 114.54±00.07
Go 3.150±0.001 3.62±00.19 + 72.15±00.13 115.35±00.13
Go/gccgo 3.157±0.001 24.18±00.07 + 73.45±00.04 112.37±00.18
Crystal 3.159±0.000 3.52±00.04 + 60.05±00.06 115.85±00.21
Java 3.175±0.001 40.65±00.08 + 69.13±00.15 124.54±00.09
Kotlin/JVM 3.207±0.005 41.86±00.02 + 68.71±00.41 130.79±00.32
Node.js 3.215±0.004 44.73±00.03 + 72.69±00.11 129.83±00.12
Python/pypy 3.275±0.001 60.40±00.21 + 68.76±00.17 136.34±00.07
Scala 3.313±0.003 68.84±00.22 + 160.83±00.27 121.22±00.21
C#/.NET Core 4.382±0.001 33.31±00.06 + 69.05±00.00 176.43±00.07
C#/Mono 7.411±0.000 26.02±00.06 + 69.47±00.00 304.12±02.00
Ruby/truffleruby 19.501±1.622 424.02±38.01 + 560.75±63.85 702.98±52.89
Ruby/truffleruby (JVM) 24.995±0.199 442.91±19.85 + 454.12±55.17 894.29±06.62
Ruby (--jit) 152.114±0.134 17.91±00.07 + 69.69±00.04 6574.42±25.26
Ruby 178.566±0.786 15.79±00.04 + 69.15±00.05 7779.16±40.92
Perl 223.173±1.482 9.54±00.08 + 599.64±00.09 9122.71±66.22
Python 238.744±0.471 10.51±00.02 + 68.84±00.00 9445.43±16.40
Tcl 332.625±1.772 7.31±00.05 + 400.44±00.00 13948.04±55.34
Ruby/jruby 445.418±12.396 276.55±04.29 + 1023.72±133.69 18883.42±466.58

Primes

Testing:

  • generating primes using the optimized sieve of Atkin;
  • prefix search for their decimal numbers using Trie data structure.

Notes:

  • All languages but V and Python use unordered hashmaps (V and Python don't provide those out of box, and their hashmaps use keys in the insertion order);
  • The results are always sorted (could be unstable or stable though).

Primes

Language Time, s Memory, MiB Energy, J
Zig 0.058±0.000 0.90±00.03 + 50.20±00.26 2.44±00.04
C++/g++ 0.068±0.000 3.61±00.01 + 79.45±00.38 2.63±00.02
C++/clang++ 0.070±0.000 3.05±00.00 + 59.19±00.13 2.62±00.04
V/clang 0.106±0.000 1.86±00.01 + 211.02±00.55 4.27±00.05
V/gcc 0.108±0.000 1.83±00.02 + 207.57±00.39 4.40±00.03
Rust 0.125±0.000 2.00±00.05 + 72.94±00.00 4.85±00.01
Crystal 0.146±0.000 3.71±00.02 + 88.43±00.00 5.94±00.07
Java 0.157±0.003 39.69±00.06 + 150.56±01.21 8.72±00.13
Scala 0.226±0.005 73.14±00.21 + 210.71±00.50 12.73±00.18
Node.js 0.256±0.002 38.33±00.02 + 151.44±00.14 12.60±00.07
Nim/clang 0.296±0.000 2.01±00.02 + 601.22±01.55 11.45±00.02
Nim/gcc 0.298±0.001 1.74±00.05 + 615.91±00.00 11.25±00.07
Lua/luajit 0.334±0.000 2.58±00.06 + 157.98±00.68 13.11±00.13
Julia 0.675±0.002 267.69±00.07 + 342.19±00.25 26.19±00.26
Racket 0.730±0.003 102.20±00.12 + 237.07±00.09 29.01±00.31
Python/pypy 0.793±0.003 59.10±00.02 + 249.23±00.08 31.40±00.13
Ruby/truffleruby 0.912±0.014 225.50±01.85 + 779.85±19.22 61.61±01.20
Ruby/truffleruby (JVM) 1.359±0.037 388.78±12.63 + 567.15±29.85 88.28±02.40
Lua 1.487±0.003 2.25±00.02 + 284.27±00.80 58.31±00.46
Ruby (--jit) 1.775±0.020 15.95±00.02 + 172.01±01.28 80.44±00.60
Ruby 1.887±0.003 14.91±00.02 + 142.89±00.03 77.16±00.57
Ruby/jruby 2.484±0.033 192.08±00.89 + 464.73±41.25 134.20±02.35
Python 3.101±0.013 10.14±00.04 + 181.56±00.77 125.18±00.55

Tests Execution

Environment

CPU: Intel(R) Xeon(R) E-2324G

Base Docker image: Debian GNU/Linux bookworm/sid

Language Version
.NET Core 7.0.402
C#/.NET Core 4.7.0-3.23416.9 (43b0b05c)
C#/Mono 6.12.0.200
Chez Scheme 9.5.8
Clojure "1.11.1"
Crystal 1.10.1
D/dmd v2.105.2
D/gdc 13.2.0
D/ldc2 1.34.0
Elixir 1.14.0
F#/.NET Core 12.7.0.0 for F# 7.0
Go go1.21.3
Go/gccgo 13.2.0
Haskell 9.4.7
Idris 2 0.6.0
Java 21
Julia v"1.9.3"
Kotlin 1.9.10
Lua 5.4.4
Lua/luajit 2.1.1696562864
MLton 20210117
Nim 2.0.0
Node.js v20.8.1
OCaml 5.1.0
PHP 8.2.10
Perl v5.36.0
Python 3.11.6
Python/pypy 7.3.13-final0 for Python 3.10.13
Racket "8.10"
Ruby 3.2.2p53
Ruby/jruby 9.4.3.0
Ruby/truffleruby 23.1.0
Rust 1.73.0
Scala 3.3.1
Swift 5.9
Tcl 8.6
V 0.4.2 2332c17
Vala 0.56.13
Zig 0.11.0
clang/clang++ 16.0.6 (15)
gcc/g++ 13.2.0

Using Docker

Build the image:

$ docker build docker/ -t benchmarks

Run the image:

$ docker run -it --rm -v $(pwd):/src benchmarks <cmd>

where <cmd> is:

  • versions (print installed language versions);
  • shell (start the shell);
  • brainfuck bench (build and run Brainfuck bench.b benchmarks);
  • brainfuck mandel (build and run Brainfuck mandel.b benchmarks);
  • base64 (build and run Base64 benchmarks);
  • json (build and run Json benchmarks);
  • matmul (build and run Matmul benchmarks);
  • primes (build and run Primes benchmarks);

Please note that the actual measurements provided in the project are taken semi-manually (via shell) as the full update takes days and could have occassional issues in Docker.

There is a ./run.sh that could be used to simplify Docker usage:

  • ./run.sh build (build the image);
  • ./run.sh make versions (run the image with the versions command);
  • sudo ./run.sh shell (run the image with the `shell' command, sudo is required to read energy levels).

Manual Execution

Makefiles contain recipes for building and executing tests with the proper dependencies. Please use make run (and make run2 where applicable). The measurements are taken using analyze.rb script:

$ cd <test suite>
$ ../analyze.rb make run
$ ../analyze.rb make run[<single test>]

Please note that the measurements could take hours. It uses 10 iterations by default, but it could be changed using ATTEMPTS environment variable:

$ ATTEMPTS=1 ../analyze.rb make run

Prerequisites

Please use Dockerfile as a reference regarding which packages and tools are required.

For all (optional):

  • Powercap for reading energy counters in Linux (Debian package powercap-utils).

For Python:

  • NumPy for matmul tests (Debian package python3-numpy).
  • UltraJSON for JSON tests (Debian package python3-ujson).

For C++:

  • Boost for JSON tests (Debian package libboost-dev).
  • JSON-C for JSON tests (Debian package libjson-c-dev).

For Rust:

  • libjq for jq test (Debian packages libjq-dev, libonig-dev and environment variable JQ_LIB_DIR=/usr/lib/x86_64-linux-gnu/).

For Java, Scala:

  • Coursier for downloading Maven artifacts.

For Haskell:

  • network for TCP connectivity between the tests and the test runner.
  • raw-strings-qq for raw string literals used in tests.

For Perl:

  • cpanminus for installing modules from CPAN (Debian package cpanminus).

For Vala:

  • JSON-GLib for JSON tests (Debian package libjson-glib-dev).

Contribution

Please follow the criteria specified in the overview. Besides that please ensure that the communication protocol between a test and the test runner is satisfied:

  • The test runner listens on localhost:9001;
  • All messages are sent using TCP sockets closed immediately after the message has been sent;
  • There are two messages sent from a test (it establishes the measurement boundary):
    1. The beginning message having the format name of the test/tprocess ID (the process ID is used to measure the memory consumption). Please note that the name of the test couldn't use Tab character as it's a delimiter;
    2. The end message with any content (mostly it's "stop" for consistency).
  • The test runner could be unavailable (if the test is launched as is) and the test should gracefully handle it.

Makefile guide

Binary executables

If the test is compiled into a single binary, then two sections of the Makefile require changes:

  • append a new target (the final binary location) into executables variable;
  • append the proper target rule.

Compiled artifacts

If the test is compiled, but can't be executed directly as a binary, then three sections of the Makefile require changes:

  • append a new target (the final artifact location) into artifacts variable;
  • append the proper target rule to compile the test;
  • append run[<target_artifact>] rule to run the test.

Scripting language

If the test doesn't require compilation, then two sections of the Makefile requires changes:

  • append run[<script_file>] into all_runners variable;
  • append run[<script_file>] rule to run the test.

README update

TOC is regenerated using git-markdown-toc:

./run.sh toc

About

Some benchmarks of different languages

License:MIT License


Languages

Language:Makefile 9.0%Language:C++ 8.6%Language:C 5.2%Language:Ruby 5.0%Language:Java 4.3%Language:Zig 4.2%Language:Rust 4.0%Language:C# 3.8%Language:Lua 3.5%Language:Racket 3.4%Language:D 3.1%Language:Go 3.0%Language:Scala 2.9%Language:Nim 2.9%Language:Vala 2.6%Language:JavaScript 2.6%Language:Python 2.6%Language:Perl 2.6%Language:Brainfuck 2.5%Language:V 2.4%Language:Haskell 2.3%Language:Julia 2.3%Language:Crystal 2.2%Language:Dockerfile 2.2%Language:F# 2.0%Language:Tcl 2.0%Language:Kotlin 1.5%Language:PHP 1.4%Language:Swift 1.2%Language:Standard ML 1.0%Language:Scheme 0.9%Language:OCaml 0.7%Language:Idris 0.7%Language:Elixir 0.7%Language:Clojure 0.3%Language:Shell 0.3%