kilpp / servers-benchmark

🚀 Servers Benchmark: Simple POC to benchmark servers and languages.

🚀 Servers Benchmark

• 1 - Java 20 - Tomcat 10.1
• 2 - Java 20 - Netty 4
• 3 - Java 20 - Netty 4 Reactive
• 4 - Java 21 - Netty 4 - GraalVM 21 - Boot 3x
• 5 - Java 20 - Undertow
• 6 - Java 20 - Micronaut
• 7 - Java 20 - Quarkus
• 8 - Java 21 - Smart Http
• 9 - Java 21 - nio-iouring
• 10 - Kotlin 1.9 - Ktor
• 11 - NodeJS 20 -Express
• 12 - Deno 1.37 - Fresh
• 13 - Bun 1.0 - Elysia
• 14 - Bun 1.0 - Hono
• 15 - Python 3.11 - Twisted
• 16 - Python 3.11 - Tornado
• 17 - V 0.4 - pico
• 18 - C++ - Drogon
• 19 - Go 1.21 - net/http
• 20 - Rust 1.71 - Actix
• 21 - Rust 1.71 - Axum
• 22 - Rust 1.71 - may_minihttp
• 23 - Zig 0.11 - Zap
• 24 - Julia 1.8.5 - Genie
• 25 - Nim 2 - httpbeast

🚀 Servers Benchmark - Contributions

• 1 - Rust 1.71 - Hyper (@andreixmartins)
• 2 - Lua 5.4 - Pegasus (@andreixmartins)
• 3 - Ruby 3.2 - TCP (@andreixmartins)
• 4 - Java 21 - default HTTP Server (@alex-carvalho)
• 5 - Bun http - (@mrarticuno)
• 6 - Bun http workers - (@mrarticuno)
• 7 - Nodejs 20 http - (@mrarticuno)
• 8 - Nodejs 20 http workers - (@mrarticuno)
• 9 - Python 3.11 Flask - (@rRickson)
• 10 - Python 3.11 fast-api-uvicorn - (@rRickson)
• 11 - Python 3.11 fast-api-hypercorn - (@rRickson)
• 12 - Java 21 - Quarkus Reactive - (@dadpig)
• 13 - Dart 3.1 - Shelf - (@brscherer)
• 14 - Java 17 - Quarkus Native - Mandrel Podman (@andremayer)
• 15 - Clojure 1.11 - Ring/Jetty - (@deividfsantos)
• 16 - Haskell Scotty - (@deividfsantos)
• 17 - Elixir - Phoenix/Cowboy - (@deividfsantos)
• 18 - C - (@Thorugoh)

Gatling reports 📈

• Boot Netty 4
• Boot Mono Netty 4
• Boot Mono Netty 4 V2
• Java 21 - Boot 3x Netty 4 - GraalVM 21
• Boot Tomcat 10.1
• Boot Undertow
• Micronaut
• Quarkus
• Java 21 - Smart Http
• Java 21 - nio-iouring
• Kotlin 1.9 Ktor (Netty)
• NodeJS 20 Express
• Deno Fresh
• Bun 1 Elysia
• Bun 1 Hono
• V 0.4 pico
• Python 3.11 Twisted
• Python 3.11 Tornado
• Go 1.21
• C++ Drogon 3x
• Rust 1.71 Actix
• Rust 1.71 Axum
• Rust 1.71 may_minihttp
• Zig 0.11 Zap
• Julia 1.8.5 Genie
• Nim 2 - httpbeast

Gatling reports - Contributions 📈

• Rust 1.71 Hyper
• Lua 5.4 Pegasus
• Ruby TCP
• Java 21 - Default HTTP Server
• Bun http
• Bun http workers
• Nodejs 20 http
• Nodejs 20 http workers
• Python 3.11 Flask
• Python 3.11 fast-api-uvicorn
• Python 3.11 fast-api-hypercorn
• Java 21 - Quarkus Reactive
• Dart 3 - Shelf
• Java 17 - Quarkus Native - Mandrel Podman
• Clojure 1.11 - Ring/Jetty
• Haskell Scotty
• Elixir - Phoenix/Cowboy
• C

Throwing a little bit of Salt 🧂

Things that this should be doing to be better.

• Wram up
• Run on AWS
• Have 2 different machines for server and gatling
• Make more rounds with more users, 10k, 100k, 1M
• Have other uses cases like, read json from DB, read static file, persist data, etc...

Results Summary (☕ Java)

1k users, during 1 minute non-stop

• Boot-Netty : 60000 total/OK, p99: 2 ms
• Boot-Tomcat : 60000 total/OK, p99: 6 ms
• Boot-Undertow : 60000 total/OK, p99: 15 ms
• Micronaut : 60000 total/OK, p99: 129 ms
• Quarkus : 60000 total/OK, p99: 4 ms

Build All 💻

Make sure you have installed and configured on the $PATH

1. Java sdk 20
2. Kotlin 1.9.10
3. Zig version 0.11
4. Rust 1.71.0
5. Go 1.21.0
6. g++ 11.4.0
7. bun 1.0.0
8. Deno 1.37
9. Nodejs 20
10. Python 3.11
11. V 0.4
12. Julia 1.8.5
13. Nim 2.0
14. Lua 5.4
15. Ruby 3.2.2
16. Dart 3.2.1
17. Podman 3.4.2
18. Clojure 1.11 / Leiningen 2.9
19. Haskell 9.2 / Cabal 3.6
20. gcc 11.4

./build-all.sh

build Java ☕

1. for Java - install sdkman - https://sdkman.io/

curl -s "https://get.sdkman.io" | bash

source "$HOME/.sdkman/bin/sdkman-init.sh"
sdk install java 20.0.2-amzn
sdk use java 20.0.2-amzn
java -version

❯ java -version
openjdk version "20.0.1" 2023-04-18
OpenJDK Runtime Environment Homebrew (build 20.0.1)
OpenJDK 64-Bit Server VM Homebrew (build 20.0.1, mixed mode, sharing)

./mvnw clean install

Important OS tuning

1. default file descriptors for linux is 1024, mac is 256 - need to be twecked.
2. also open files
3. switch to ip v4 here are steps on how to do it: https://gist.github.com/diegopacheco/ad1e63691380ad1a6b3be6b62910c3fb

for V and pico

gatling is having issues, but apache ab is working fine.

ab -n 60000 -c 1000 http://127.0.0.1:8080/

This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 6000 requests
Completed 12000 requests
Completed 18000 requests
Completed 24000 requests
Completed 30000 requests
Completed 36000 requests
Completed 42000 requests
Completed 48000 requests
Completed 54000 requests
Completed 60000 requests
Finished 60000 requests


Server Software:        
Server Hostname:        127.0.0.1
Server Port:            8080

Document Path:          /
Document Length:        36 bytes

Concurrency Level:      1000
Time taken for tests:   487.475 seconds
Complete requests:      60000
Failed requests:        0
Total transferred:      6000000 bytes
HTML transferred:       2160000 bytes
Requests per second:    123.08 [#/sec] (mean)
Time per request:       8124.588 [ms] (mean)
Time per request:       8.125 [ms] (mean, across all concurrent requests)
Transfer rate:          12.02 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   13   9.0     13      56
Processing:  7043 8026 271.9   8059   15159
Waiting:        0    7   6.6      6      69
Total:       7056 8039 272.1   8069   15179

Percentage of the requests served within a certain time (ms)
  50%   8069
  66%   8072
  75%   8075
  80%   8077
  90%   8090
  95%   8105
  98%   8126
  99%   8129
 100%  15179 (longest request)

Fix gatling for V and pic by sharing connection.

HttpProtocolBuilder httpProtocol = http
        .baseUrl("http://localhost:8080")
        .acceptHeader("text/html")
        .doNotTrackHeader("1")
        .userAgentHeader("Gat")
        .shareConnections();

Open a PR to add new language / server?

1. Create a gatling simulation
2. Create a gatling script
3. Create a run.sh script - add the tuning for linux os pls
4. Add the new project with new language/framework (1 pr per lang/framework)
5. Dont publish gatling results
6. I will re-run on my machine, you can run all in our machine in our fork to compare different results with different hardware
7. Add a different script to install dependencies

Note on Java 21

Java 21 there is some performance degradetion comparing spring boot 3x and netty.

Correto JDK 21

================================================================================
---- Global Information --------------------------------------------------------
> request count                                      60000 (OK=60000  KO=0     )
> min response time                                      0 (OK=0      KO=-     )
> max response time                                    276 (OK=276    KO=-     )
> mean response time                                     2 (OK=2      KO=-     )
> std deviation                                         14 (OK=14     KO=-     )
> response time 50th percentile                          1 (OK=1      KO=-     )
> response time 75th percentile                          1 (OK=1      KO=-     )
> response time 95th percentile                          1 (OK=1      KO=-     )
> response time 99th percentile                          8 (OK=8      KO=-     )
> mean requests/sec                                923.077 (OK=923.077 KO=-     )
---- Response Time Distribution ------------------------------------------------
> t < 800 ms                                         60000 (100%)
> 800 ms <= t < 1200 ms                                  0 (  0%)
> t >= 1200 ms                                           0 (  0%)
> failed                                                 0 (  0%)
================================================================================

Correto JDK 20

================================================================================
---- Global Information --------------------------------------------------------
> request count                                      60000 (OK=60000  KO=0     )
> min response time                                      0 (OK=0      KO=-     )
> max response time                                    267 (OK=267    KO=-     )
> mean response time                                     2 (OK=2      KO=-     )
> std deviation                                         13 (OK=13     KO=-     )
> response time 50th percentile                          1 (OK=1      KO=-     )
> response time 75th percentile                          1 (OK=1      KO=-     )
> response time 95th percentile                          1 (OK=1      KO=-     )
> response time 99th percentile                          7 (OK=7      KO=-     )
> mean requests/sec                                923.077 (OK=923.077 KO=-     )
---- Response Time Distribution ------------------------------------------------
> t < 800 ms                                         60000 (100%)
> 800 ms <= t < 1200 ms                                  0 (  0%)
> t >= 1200 ms                                           0 (  0%)
> failed                                                 0 (  0%)
================================================================================

Application boot time on Java 21, GraalVM 21 and Netty

0.049 seconds

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v3.1.3)

2023-09-22T16:06:05.325-07:00  INFO 450420 --- [           main] c.g.d.sandboxspring.Application          : Starting AOT-processed Application using Java 21 with PID 450420 (/mnt/e35d88d4-42b9-49ea-bf29-c4c3b018d429/diego/git/diegopacheco/servers-benchmark/server-boot-netty/target/server-boot-netty started by diego in /mnt/e35d88d4-42b9-49ea-bf29-c4c3b018d429/diego/git/diegopacheco/servers-benchmark/server-boot-netty)
2023-09-22T16:06:05.325-07:00  INFO 450420 --- [           main] c.g.d.sandboxspring.Application          : No active profile set, falling back to 1 default profile: "default"
2023-09-22T16:06:05.361-07:00  INFO 450420 --- [           main] o.s.b.web.embedded.netty.NettyWebServer  : Netty started on port 8080
2023-09-22T16:06:05.362-07:00  INFO 450420 --- [           main] c.g.d.sandboxspring.Application          : Started Application in 0.049 seconds (process running for 0.055)
Spring Boot 3.1.x working!