🚀 Servers Benchmark

1 - Java 20 - Tomcat 10.1
2 - Java 20 - Netty 4
3 - Java 20 - Netty 4 Reactive
4 - Java 21 - Netty 4 - GraalVM 21 - Boot 3x
5 - Java 20 - Undertow
6 - Java 20 - Micronaut
7 - Java 20 - Quarkus
8 - Java 21 - Smart Http
9 - Java 21 - nio-iouring
10 - Kotlin 1.9 - Ktor
11 - NodeJS 20 -Express
12 - Deno 1.37 - Fresh
13 - Bun 1.0 - Elysia
14 - Bun 1.0 - Hono
15 - Python 3.11 - Twisted
16 - Python 3.11 - Tornado
17 - V 0.4 - pico
18 - C++ - Drogon
19 - Go 1.21 - net/http
20 - Rust 1.71 - Actix
21 - Rust 1.71 - Axum
22 - Rust 1.71 - may_minihttp
23 - Zig 0.11 - Zap
24 - Julia 1.8.5 - Genie
25 - Nim 2 - httpbeast
26 - Ocaml 5.1 - http_async
27 - PHP 8.1 - embedded server
28 - PHP 8.1 - Nginx
29 - Scala 3.0 - JDK 21 - Zio Http
30 - Scala 3.0 - JDK 21 - Play 2.9
31 - Scala 3.0 - JDK 21 - Akka-Http

🚀 Servers Benchmark - Contributions

1 - Rust 1.71 - Hyper (@andreixmartins)
2 - Lua 5.4 - Pegasus (@andreixmartins)
3 - Ruby 3.2 - TCP (@andreixmartins)
4 - Java 21 - default HTTP Server (@alex-carvalho)
5 - Bun http - (@mrarticuno)
6 - Bun http workers - (@mrarticuno)
7 - Nodejs 20 http - (@mrarticuno)
8 - Nodejs 20 http workers - (@mrarticuno)
9 - Python 3.11 Flask - (@rRickson)
10 - Python 3.11 fast-api-uvicorn - (@rRickson)
11 - Python 3.11 fast-api-hypercorn - (@rRickson)
12 - Java 21 - Quarkus Reactive - (@dadpig)
13 - Dart 3.1 - Shelf - (@brscherer)
14 - Java 17 - Quarkus Native - Mandrel Podman (@andremayer)
15 - Clojure 1.11 - Ring/Jetty - (@deividfsantos)
16 - Haskell Scotty - (@deividfsantos)
17 - Elixir - Phoenix/Cowboy - (@deividfsantos)
18 - C - (@Thorugoh)
19 - Racket - Spin (@kilpp)
20 - Crystal 1.9.2 - Spider - (@LucasKonrath)
21 - Java 8 - Blade - (@LucasKonrath)
22 - Gleam - Mist - (@LucasKonrath)
23 - Rust 1.7.1 - Gotham - (@andreixmartins)
24 - Rust 1.72.1 - Salvo (@andremayer)
25 - Rust 1.74.0-nightly - xitca (@andremayer)
26 - Haskell - Warp (@brunohaetinger)
27 - Kotlin -Http4k (@codegik)

Gatling reports 📈

Gatling reports - Contributions 📈

Throwing a little bit of Salt 🧂

Things that this should be doing to be better.

Wram up
Run on AWS
Have 2 different machines for server and gatling
Make more rounds with more users, 10k, 100k, 1M
Have other uses cases like, read json from DB, read static file, persist data, etc...

Results Summary (☕ Java)

1k users, during 1 minute non-stop

Boot-Netty : 60000 total/OK, p99: 2 ms
Boot-Tomcat : 60000 total/OK, p99: 6 ms
Boot-Undertow : 60000 total/OK, p99: 15 ms
Micronaut : 60000 total/OK, p99: 129 ms
Quarkus : 60000 total/OK, p99: 4 ms

Build All 💻

Make sure you have installed and configured on the $PATH

Java sdk 20
Kotlin 1.9.10
Zig version 0.11
Rust 1.71.0
Go 1.21.0
g++ 11.4.0
bun 1.0.0
Deno 1.37
Nodejs 20
Python 3.11
V 0.4
Julia 1.8.5
Nim 2.0
Lua 5.4
Ruby 3.2.2
Dart 3.2.1
Podman 3.4.2
Clojure 1.11 / Leiningen 2.9
Haskell 9.2 / Cabal 3.6
gcc 11.4
Racket 8.2
Ocaml 5.1 / opam 2.1 / dune 3.7
PHP 8.1
Scala 3.3.1 / sbt 1.9.6
Crystal 1.9.2
Gleam 0.31.0 / mist 0.13.2

./build-all.sh

build Java ☕

for Java - install sdkman - https://sdkman.io/

curl -s "https://get.sdkman.io" | bash

source "$HOME/.sdkman/bin/sdkman-init.sh"
sdk install java 20.0.2-amzn
sdk use java 20.0.2-amzn
java -version

❯ java -version
openjdk version "20.0.1" 2023-04-18
OpenJDK Runtime Environment Homebrew (build 20.0.1)
OpenJDK 64-Bit Server VM Homebrew (build 20.0.1, mixed mode, sharing)

./mvnw clean install

Important OS tuning

default file descriptors for linux is 1024, mac is 256 - need to be twecked.
also open files
switch to ip v4 here are steps on how to do it: https://gist.github.com/diegopacheco/ad1e63691380ad1a6b3be6b62910c3fb

for V and pico

gatling is having issues, but apache ab is working fine.

ab -n 60000 -c 1000 http://127.0.0.1:8080/

This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 6000 requests
Completed 12000 requests
Completed 18000 requests
Completed 24000 requests
Completed 30000 requests
Completed 36000 requests
Completed 42000 requests
Completed 48000 requests
Completed 54000 requests
Completed 60000 requests
Finished 60000 requests


Server Software:        
Server Hostname:        127.0.0.1
Server Port:            8080

Document Path:          /
Document Length:        36 bytes

Concurrency Level:      1000
Time taken for tests:   487.475 seconds
Complete requests:      60000
Failed requests:        0
Total transferred:      6000000 bytes
HTML transferred:       2160000 bytes
Requests per second:    123.08 [#/sec] (mean)
Time per request:       8124.588 [ms] (mean)
Time per request:       8.125 [ms] (mean, across all concurrent requests)
Transfer rate:          12.02 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   13   9.0     13      56
Processing:  7043 8026 271.9   8059   15159
Waiting:        0    7   6.6      6      69
Total:       7056 8039 272.1   8069   15179

Percentage of the requests served within a certain time (ms)
  50%   8069
  66%   8072
  75%   8075
  80%   8077
  90%   8090
  95%   8105
  98%   8126
  99%   8129
 100%  15179 (longest request)

Fix gatling for V and pic by sharing connection.

HttpProtocolBuilder httpProtocol = http
        .baseUrl("http://localhost:8080")
        .acceptHeader("text/html")
        .doNotTrackHeader("1")
        .userAgentHeader("Gat")
        .shareConnections();

Open a PR to add new language / server?

Create a gatling simulation
Create a gatling script
Create a run.sh script - add the tuning for linux os pls
Add the new project with new language/framework (1 pr per lang/framework)
Dont publish gatling results
I will re-run on my machine, you can run all in our machine in our fork to compare different results with different hardware
Add a different script to install dependencies

Note on Java 21

Java 21 there is some performance degradetion comparing spring boot 3x and netty.

Correto JDK 21

================================================================================
---- Global Information --------------------------------------------------------
> request count                                      60000 (OK=60000  KO=0     )
> min response time                                      0 (OK=0      KO=-     )
> max response time                                    276 (OK=276    KO=-     )
> mean response time                                     2 (OK=2      KO=-     )
> std deviation                                         14 (OK=14     KO=-     )
> response time 50th percentile                          1 (OK=1      KO=-     )
> response time 75th percentile                          1 (OK=1      KO=-     )
> response time 95th percentile                          1 (OK=1      KO=-     )
> response time 99th percentile                          8 (OK=8      KO=-     )
> mean requests/sec                                923.077 (OK=923.077 KO=-     )
---- Response Time Distribution ------------------------------------------------
> t < 800 ms                                         60000 (100%)
> 800 ms <= t < 1200 ms                                  0 (  0%)
> t >= 1200 ms                                           0 (  0%)
> failed                                                 0 (  0%)
================================================================================

Correto JDK 20

================================================================================
---- Global Information --------------------------------------------------------
> request count                                      60000 (OK=60000  KO=0     )
> min response time                                      0 (OK=0      KO=-     )
> max response time                                    267 (OK=267    KO=-     )
> mean response time                                     2 (OK=2      KO=-     )
> std deviation                                         13 (OK=13     KO=-     )
> response time 50th percentile                          1 (OK=1      KO=-     )
> response time 75th percentile                          1 (OK=1      KO=-     )
> response time 95th percentile                          1 (OK=1      KO=-     )
> response time 99th percentile                          7 (OK=7      KO=-     )
> mean requests/sec                                923.077 (OK=923.077 KO=-     )
---- Response Time Distribution ------------------------------------------------
> t < 800 ms                                         60000 (100%)
> 800 ms <= t < 1200 ms                                  0 (  0%)
> t >= 1200 ms                                           0 (  0%)
> failed                                                 0 (  0%)
================================================================================

Application boot time on Java 21, GraalVM 21 and Netty

0.049 seconds


  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v3.1.3)

2023-09-22T16:06:05.325-07:00  INFO 450420 --- [           main] c.g.d.sandboxspring.Application          : Starting AOT-processed Application using Java 21 with PID 450420 (/mnt/e35d88d4-42b9-49ea-bf29-c4c3b018d429/diego/git/diegopacheco/servers-benchmark/server-boot-netty/target/server-boot-netty started by diego in /mnt/e35d88d4-42b9-49ea-bf29-c4c3b018d429/diego/git/diegopacheco/servers-benchmark/server-boot-netty)
2023-09-22T16:06:05.325-07:00  INFO 450420 --- [           main] c.g.d.sandboxspring.Application          : No active profile set, falling back to 1 default profile: "default"
2023-09-22T16:06:05.361-07:00  INFO 450420 --- [           main] o.s.b.web.embedded.netty.NettyWebServer  : Netty started on port 8080
2023-09-22T16:06:05.362-07:00  INFO 450420 --- [           main] c.g.d.sandboxspring.Application          : Started Application in 0.049 seconds (process running for 0.055)
Spring Boot 3.1.x working!

Server Spec

Linux Kernel: 6.2.0-1009-lowlatency 
OS          : Ubuntu 22.04.3 LTS

inxi -Fxz

System:
  Kernel: 6.2.0-1009-lowlatency x86_64 bits: 64 compiler: N/A
    Desktop: GNOME 42.9 Distro: Ubuntu 22.04.3 LTS (Jammy Jellyfish)
Machine:
  Type: Laptop System: Avell High Performance product: Avell G1555 MUV / A62
    v: Standard serial: <superuser required>
  Mobo: Avell High Performance model: Avell G1555 MUV / A62 v: Standard
    serial: <superuser required> UEFI: American Megatrends v: N.1.04
    date: 08/13/2019
Battery:
  ID-1: BAT0 charge: 44.9 Wh (96.1%) condition: 46.7/46.7 Wh (100.0%)
    volts: 12.5 min: 11.4 model: standard status: Not charging
CPU:
  Info: 6-core model: Intel Core i7-9750H bits: 64 type: MT MCP
    arch: Coffee Lake rev: A cache: L1: 384 KiB L2: 1.5 MiB L3: 12 MiB
  Speed (MHz): avg: 3498 high: 4202 min/max: 800/4500 cores: 1: 4154
    2: 4200 3: 2600 4: 4196 5: 2600 6: 4092 7: 2600 8: 4127 9: 4202 10: 4014
    11: 2600 12: 2600 bogomips: 62399
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx
Graphics:
  Device-1: Intel CoffeeLake-H GT2 [UHD Graphics 630]
    vendor: Tongfang Hongkong driver: i915 v: kernel bus-ID: 00:02.0
  Device-2: NVIDIA TU116M [GeForce GTX 1660 Ti Mobile]
    vendor: Tongfang Hongkong driver: nouveau v: kernel bus-ID: 01:00.0
  Device-3: Logitech BRIO Ultra HD Webcam type: USB
    driver: snd-usb-audio,uvcvideo bus-ID: 1-12.4:10
  Device-4: Acer HD Webcam type: USB driver: uvcvideo bus-ID: 1-13:5
  Display: wayland server: X.Org v: 1.22.1.1 with: Xwayland v: 22.1.1
    compositor: gnome-shell driver: X: loaded: modesetting,nvidia
    unloaded: fbdev,nouveau,vesa gpu: i915,nouveau resolution:
    1: 3840x2160~60Hz 2: 1920x1080~60Hz
  OpenGL: renderer: Mesa Intel UHD Graphics 630 (CFL GT2)
    v: 4.6 Mesa 23.0.4-0ubuntu1~22.04.1 direct render: Yes
Network:
  Device-1: Intel Cannon Lake PCH CNVi WiFi driver: iwlwifi v: kernel
    bus-ID: 00:14.3
  IF: wlo1 state: up mac: <filter>
  Device-2: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet
    driver: r8169 v: kernel port: N/A bus-ID: 03:00.0
Drives:
  Local Storage: total: 1.36 TiB used: 742.63 GiB (53.1%)
  ID-1: /dev/nvme0n1 vendor: Samsung model: SSD 970 EVO Plus 500GB
    size: 465.76 GiB temp: 50.9 C
  ID-2: /dev/sda vendor: Crucial model: CT1000MX500SSD1 size: 931.51 GiB
Partition:
  ID-1: / size: 456.89 GiB used: 372.27 GiB (81.5%) fs: ext4
    dev: /dev/nvme0n1p2
  ID-2: /boot/efi size: 511 MiB used: 6.1 MiB (1.2%) fs: vfat
    dev: /dev/nvme0n1p1
Swap:
  ID-1: swap-1 type: file size: 2 GiB used: 0 KiB (0.0%) file: /swapfile
Sensors:
  System Temperatures: cpu: 84.0 C pch: 75.0 C mobo: 84.0 C gpu: nouveau
    temp: 69.0 C
  Fan Speeds (RPM): N/A
Info:
  Processes: 449 Uptime: 1d 45m Memory: 62.65 GiB used: 10.58 GiB (16.9%)
  Init: systemd runlevel: 5 Compilers: gcc: 11.4.0 clang: 16.0.4
  Packages: 4360 Shell: Zsh v: 5.8.1 inxi: 3.3.13

codegik / servers-benchmark