Giters

47degrees / metrifier

Comparing HTTP endpoints against mu-rpc services.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

THIS PROJECT IS DISCONTINUED — USE AT YOUR OWN RISK

It has been a fun and great project but it's time for us to move on. Check out our recent work that we are doing with Scala and follow us on Github and Twitter for new and exciting open source projects. Thanks for your continuing support. If you wish to take on maintenance of this library please contact us through the issue tracker.

metrifier

Comparing HTTP against frees-rpc services.

Table of Contents generated with DocToc

Running Demo

http

  • Run Server:
sbt "http/runMain metrifier.http.server.HttpServer"
  • Run Client:
sbt "demo/runMain metrifier.demo.HttpDemoApp"

By default, host and port will be localhost and 8080, respectively. You can override this configuration through either configuration key or environment variable:

  • Host: http.host VS HTTP_HOST.
  • Port: http.port VS HTTP_PORT.

frees-rpc

In this case, we will try out with two different of binary serializations: Protobuf and Avro.

By default, for both cases, the host and the port will be localhost and 8080, respectively. You can override this configuration through either configuration key or environment variable:

  • Host: rpc.host VS RPC_HOST.
  • Port: rpc.port VS RPC_PORT.

frees-rpc Protobuf

  • Run Protobuf based Server:
sbt "frees-rpc/runMain metrifier.rpc.server.RPCProtoServer"
  • Run Protobuf based Client:
sbt "demo/runMain metrifier.demo.RPCProtoDemoApp"

frees-rpc Avro

  • Run Avro based Server:
sbt "frees-rpc/runMain metrifier.rpc.server.RPCAvroServer"
  • Run Avro based Client:
sbt "demo/runMain metrifier.demo.RPCAvroDemoApp"

Running Benchmarks Locally

We are using the Java Microbenchmark Harness (JMH) tool, which is helping us to get an experimental answer to a basic question about which implementation executes fastest among:

  • HTTP stack based on:
    • http4s, version 0.18.15.
    • circe, version 0.9.3.
  • RPC services stack based on:
    • frees-rpc, version 0.14.1 (atop of gRPC, version 1.11.0).

HTTP Benchmarks

  • Run Server:
sbt "http/runMain metrifier.http.server.HttpServer"
  • Run Benchmarks:
sbt "bench/jmh:run -o http-benchmark-results.txt -i 20 -wi 20 -f 2 -t 4 -r 1 -w 1 metrifier.benchmark.HttpBenchmark"

Which means "20 iterations", "20 warmup iterations", "2 forks", "4 threads". r and w are specifying the minimum time (seconds) to spend at each measurement warmup iteration/iteration.

frees-rpc Protobuf Benchmarks

  • Run Protobuf based Server:
sbt "frees-rpc/runMain metrifier.rpc.server.RPCProtoServer"
  • Run Protobuf based Benchmarks:
sbt "bench/jmh:run -o rpc-proto-benchmark-results.txt -i 20 -wi 20 -f 2 -t 4 -r 1 -w 1 metrifier.benchmark.RPCProtoBenchmark"

Which means "20 iterations", "20 warmup iterations", "2 forks", "4 threads". r and w are specifying the minimum time (seconds) to spend at each measurement warmup iteration/iteration.

frees-rpc Avro Benchmarks

  • Run Avro based Server:
sbt "frees-rpc/runMain metrifier.rpc.server.RPCAvroServer"
  • Run Avro based Benchmarks:
sbt "bench/jmh:run -o rpc-avro-benchmark-results.txt -i 20 -wi 20 -f 2 -t 4 -r 1 -w 1 metrifier.benchmark.RPCAvroBenchmark"

Which means "20 iterations", "20 warmup iterations", "2 forks", "4 threads". r and w are specifying the minimum time (seconds) to spend at each measurement warmup iteration/iteration.

Running Benchmarks on Google Cloud Platform

Before starting detailing how to deploy metrifier to GCP, let's see how to assemble it.

Assembling metrifier

To make a JAR file containing only the external dependencies, type:

sbt assemblyPackageDependency

Output (assuming we are in the project path):

bench/target/scala-2.12/metrifier-bench-assembly-[project-version]-deps.jar
demo/target/scala-2.12/metrifier-demo-assembly-[project-version]-deps.jar
frees-rpc/target/scala-2.12/metrifier-frees-rpc-assembly-[project-version]-deps.jar
http/target/scala-2.12/metrifier-http-assembly-[project-version]-deps.jar
shared/target/scala-2.12/metrifier-shared-assembly-[project-version]-deps.jar

This is intended to be used with a JAR that only contains your project, so now, you can write:

sbt assembly

And we'll get the following artifacts as the result:

bench/target/scala-2.12/metrifier-bench-assembly-[project-version].jar
demo/target/scala-2.12/metrifier-demo-assembly-[project-version].jar
frees-rpc/target/scala-2.12/metrifier-frees-rpc-assembly-[project-version].jar
http/target/scala-2.12/metrifier-http-assembly-[project-version].jar
shared/target/scala-2.12/metrifier-shared-assembly-[project-version].jar

Uploading jars to Google Cloud Storage

In this case, we've created a bucket named as metrifier within our GCP project. Assuming this name, these would be the set of commands to run (we're skipping the bench artifacts since we are not going to use them):

export METRIFIER_VERSION=0.1.0
gsutil cp demo/target/scala-2.12/metrifier-demo-assembly-${METRIFIER_VERSION}-deps.jar gs://metrifier/jars
gsutil cp frees-rpc/target/scala-2.12/metrifier-frees-rpc-assembly-${METRIFIER_VERSION}-deps.jar gs://metrifier/jars
gsutil cp http/target/scala-2.12/metrifier-http-assembly-${METRIFIER_VERSION}-deps.jar gs://metrifier/jars
gsutil cp shared/target/scala-2.12/metrifier-shared-assembly-${METRIFIER_VERSION}-deps.jar gs://metrifier/jars
gsutil cp demo/target/scala-2.12/metrifier-demo-assembly-${METRIFIER_VERSION}.jar gs://metrifier/jars
gsutil cp frees-rpc/target/scala-2.12/metrifier-frees-rpc-assembly-${METRIFIER_VERSION}.jar gs://metrifier/jars
gsutil cp http/target/scala-2.12/metrifier-http-assembly-${METRIFIER_VERSION}.jar gs://metrifier/jars
gsutil cp shared/target/scala-2.12/metrifier-shared-assembly-${METRIFIER_VERSION}.jar gs://metrifier/jars

If the project dependencies have not changed, you could just upload the project JARs:

export METRIFIER_VERSION=0.1.0
gsutil cp demo/target/scala-2.12/metrifier-demo-assembly-${METRIFIER_VERSION}.jar gs://metrifier/jars
gsutil cp frees-rpc/target/scala-2.12/metrifier-frees-rpc-assembly-${METRIFIER_VERSION}.jar gs://metrifier/jars
gsutil cp http/target/scala-2.12/metrifier-http-assembly-${METRIFIER_VERSION}.jar gs://metrifier/jars
gsutil cp shared/target/scala-2.12/metrifier-shared-assembly-${METRIFIER_VERSION}.jar gs://metrifier/jars

Provisioning the Infrastructure

See this guide to get information about how to deploy and to provision the different services in Google Compute Engine.

Once everything is up, follow the next sections to run the benchmarks atop GCP.

HTTP Benchmarks

Running the Server

  1. SSH into http-server-vm instance.
  2. Run the HTTP Server:
export METRIFIER_VERSION=0.1.0
env \
    HTTP_HOST=http-server-vm \
    HTTP_PORT=8080 \
    java -cp \
    /metrifier/jars/metrifier-shared-assembly-${METRIFIER_VERSION}-deps.jar:/metrifier/jars/metrifier-shared-assembly-${METRIFIER_VERSION}.jar:/metrifier/jars/metrifier-http-assembly-${METRIFIER_VERSION}-deps.jar:/metrifier/jars/metrifier-http-assembly-${METRIFIER_VERSION}.jar \
    metrifier.http.server.HttpServer

Running the Benchmarks

  1. SSH into http-jmh-vm instance.
  2. Run the following GET to fetch all the persons (checking connectivity):
curl "http://http-server-vm:8080/person"
  1. If step was successful, run the benchmarks:
export METRIFIER_VERSION=0.1.0
cd /metrifier/repo
env \
    HTTP_HOST=http-server-vm \
    HTTP_PORT=8080 \
    sbt "bench/jmh:run -o /metrifier/bench_results/http-benchmark-results-${METRIFIER_VERSION}.txt -i 20 -wi 20 -f 2 -t 4 -r 1 -w 1 metrifier.benchmark.HttpBenchmark"

Given the port 8080 was opened to the exterior when deploying the cluster with Google Cloud Manager, you could even run the benchmarks from your local machine, using the external IP address (changing to HTTP_HOST=[HTTP_SERVER_INSTANCE_EXTERNAL_IP]).

frees-rpc Proto Benchmarks

Running the Server

  1. SSH into rpc-proto-server-vm instance.
  2. Run the RPC Protobuf based Server:
export METRIFIER_VERSION=0.1.0
env \
    RPC_HOST=rpc-proto-server-vm \
    RPC_PORT=8080 \
    java -cp \
    /metrifier/jars/metrifier-shared-assembly-${METRIFIER_VERSION}-deps.jar:/metrifier/jars/metrifier-shared-assembly-${METRIFIER_VERSION}.jar:/metrifier/jars/metrifier-frees-rpc-assembly-${METRIFIER_VERSION}-deps.jar:/metrifier/jars/metrifier-frees-rpc-assembly-${METRIFIER_VERSION}.jar \
    metrifier.rpc.server.RPCProtoServer

Running the Benchmarks

  1. SSH into rpc-proto-jmh-vm instance.
  2. Run the benchmarks:
export METRIFIER_VERSION=0.1.0
cd /metrifier/repo
env \
    RPC_HOST=rpc-proto-server-vm \
    RPC_PORT=8080 \
    sbt "bench/jmh:run -o /metrifier/bench_results/rpc-proto-benchmark-results-${METRIFIER_VERSION}.txt -i 20 -wi 20 -f 2 -t 4 -r 1 -w 1 metrifier.benchmark.RPCProtoBenchmark"

As we mentioned for the Http benchmarks, in this case we could also run the benchmarks from our local machine, using the external IP address (changing to RPC_HOST=[RPC_SERVER_INSTANCE_EXTERNAL_IP]).

frees-rpc Avro Benchmarks

Running the Server

  1. SSH into rpc-avro-server-vm instance.
  2. Run the RPC Avro based Server:
export METRIFIER_VERSION=0.1.0
env \
    RPC_HOST=rpc-avro-server-vm \
    RPC_PORT=8080 \
    java -cp \
    /metrifier/jars/metrifier-shared-assembly-${METRIFIER_VERSION}-deps.jar:/metrifier/jars/metrifier-shared-assembly-${METRIFIER_VERSION}.jar:/metrifier/jars/metrifier-frees-rpc-assembly-${METRIFIER_VERSION}-deps.jar:/metrifier/jars/metrifier-frees-rpc-assembly-${METRIFIER_VERSION}.jar \
    metrifier.rpc.server.RPCAvroServer

Running the Benchmarks

  1. SSH into rpc-avro-jmh-vm instance.
  2. Run the benchmarks:
export METRIFIER_VERSION=0.1.0
cd /metrifier/repo
env \
    RPC_HOST=rpc-avro-server-vm \
    RPC_PORT=8080 \
    sbt "bench/jmh:run -o /metrifier/bench_results/rpc-avro-benchmark-results-${METRIFIER_VERSION}.txt -i 20 -wi 20 -f 2 -t 4 -r 1 -w 1 metrifier.benchmark.RPCAvroBenchmark"

As above, we could also run the benchmarks from our local machine, using the external IP address (changing to RPC_HOST=[RPC_SERVER_INSTANCE_EXTERNAL_IP]).

Benchmark Results

We've experimented with two different environments, local (development laptop) and the cloud (GCP). Expanded version of these results are in:

Running Benchmarks Locally

Machine Details

  • Model Name: MacBook Pro
  • Model Identifier: MacBookPro12,1
  • Intel(R) Core(TM) i5-5257U CPU @ 2.70GHz
  • Number of Processors: 1
  • Total Number of Cores: 2
  • L2 Cache (per Core): 256 KB
  • L3 Cache: 3 MB
  • Memory: 16 GB

HTTP vs frees-rpc Protobuf vs frees-rpc Avro

  • HttpBenchmark Raw output:
# Run complete. Total time: 00:07:08

Benchmark                          Mode  Cnt     Score     Error  Units
HttpBenchmark.createPerson        thrpt   40  2567.182 ± 490.607  ops/s
HttpBenchmark.getPerson           thrpt   40  4352.536 ± 101.832  ops/s
HttpBenchmark.getPersonLinks      thrpt   40  3535.380 ±  69.672  ops/s
HttpBenchmark.listPersons         thrpt   40  3766.406 ±  62.676  ops/s
HttpBenchmark.programComposition  thrpt   40   422.560 ±  30.245  ops/s
  • RPCProtoBenchmark Raw output:
# Run complete. Total time: 00:07:02

Benchmark                              Mode  Cnt     Score     Error  Units
RPCProtoBenchmark.createPerson        thrpt   40  6417.909 ± 333.242  ops/s
RPCProtoBenchmark.getPerson           thrpt   40  7345.886 ± 148.966  ops/s
RPCProtoBenchmark.getPersonLinks      thrpt   40  5604.365 ± 108.714  ops/s
RPCProtoBenchmark.listPersons         thrpt   40  6106.618 ± 114.832  ops/s
RPCProtoBenchmark.programComposition  thrpt   40   818.099 ±  17.076  ops/s
  • RPCAvroBenchmark Raw output:
# Run complete. Total time: 00:07:06

Benchmark                             Mode  Cnt     Score     Error  Units
RPCAvroBenchmark.createPerson        thrpt   40  3716.135 ± 208.849  ops/s
RPCAvroBenchmark.getPerson           thrpt   40  4778.613 ± 242.627  ops/s
RPCAvroBenchmark.getPersonLinks      thrpt   40  3812.347 ± 250.107  ops/s
RPCAvroBenchmark.listPersons         thrpt   40  4424.835 ± 162.293  ops/s
RPCAvroBenchmark.programComposition  thrpt   40   501.256 ±  43.743  ops/s

Comparing benchmarks one by one

createPerson
Source Mode Cnt Score Error Units
HttpBenchmark.createPerson thrpt 40 2567.182 490.607 ops/s
RPCProtoBenchmark.createPerson thrpt 40 6417.909 333.242 ops/s
RPCAvroBenchmark.createPerson thrpt 40 3716.135 208.849 ops/s
getPerson
Source Mode Cnt Score Error Units
HttpBenchmark.getPerson thrpt 40 4352.536 101.832 ops/s
RPCProtoBenchmark.getPerson thrpt 40 7345.886 148.966 ops/s
RPCAvroBenchmark.getPerson thrpt 40 4778.613 242.627 ops/s
getPersonLinks
Source Mode Cnt Score Error Units
HttpBenchmark.getPersonLinks thrpt 40 3535.380 69.672 ops/s
RPCProtoBenchmark.getPersonLinks thrpt 40 5604.365 108.714 ops/s
RPCAvroBenchmark.getPersonLinks thrpt 40 3812.347 250.107 ops/s
listPersons
Source Mode Cnt Score Error Units
HttpBenchmark.listPersons thrpt 40 3766.406 62.676 ops/s
RPCProtoBenchmark.listPersons thrpt 40 6106.618 114.832 ops/s
RPCAvroBenchmark.listPersons thrpt 40 4424.835 162.293 ops/s
programComposition
Source Mode Cnt Score Error Units
HttpBenchmark.programComposition thrpt 40 422.560 30.245 ops/s
RPCProtoBenchmark.programComposition thrpt 40 818.099 17.076 ops/s
RPCAvroBenchmark.programComposition thrpt 40 501.256 43.743 ops/s

Comparing benchmarks in Charts

You can find the following charts in this jsfiddle.

  • Bar Chart bar-chart-bench

  • Radar Chart radar-chart-bench

Running Benchmarks on GCP

We are implementing two Google Compute Engine instances, one for the server (n1-standard-2), another one for the benchmarks (n1-standard-1). See Google Docs - Machine Types for deeper information.

Server GCE instance Details

  • n1-standard-2.
  • 2 virtual CPUs.
  • 7.5 GB of memory.

Benchmarks GCE instance Details

  • n1-standard-1.
  • 1 virtual CPU.
  • 3.75 GB of memory.

HTTP vs frees-rpc Protobuf vs frees-rpc Avro

  • HttpBenchmark Raw output:
# Run complete. Total time: 00:07:35

Benchmark                          Mode  Cnt    Score    Error  Units
HttpBenchmark.createPerson        thrpt   40  599.976 ± 43.344  ops/s
HttpBenchmark.getPerson           thrpt   40  680.914 ± 52.230  ops/s
HttpBenchmark.getPersonLinks      thrpt   40  713.296 ± 43.999  ops/s
HttpBenchmark.listPersons         thrpt   40  690.897 ± 76.741  ops/s
HttpBenchmark.programComposition  thrpt   40   73.471 ± 10.356  ops/s
  • RPCProtoBenchmark Raw output:
# Run complete. Total time: 00:07:06

Benchmark                              Mode  Cnt     Score     Error  Units
RPCProtoBenchmark.createPerson        thrpt   40  7366.238 ± 825.910  ops/s
RPCProtoBenchmark.getPerson           thrpt   40  7924.244 ± 980.973  ops/s
RPCProtoBenchmark.getPersonLinks      thrpt   40  3708.267 ± 444.061  ops/s
RPCProtoBenchmark.listPersons         thrpt   40  5486.742 ± 465.748  ops/s
RPCProtoBenchmark.programComposition  thrpt   40   608.205 ±  63.402  ops/s
  • RPCAvroBenchmark Raw output:
# Run complete. Total time: 00:07:11

Benchmark                             Mode  Cnt     Score     Error  Units
RPCAvroBenchmark.createPerson        thrpt   40  2090.695 ± 147.251  ops/s
RPCAvroBenchmark.getPerson           thrpt   40  2771.051 ± 253.325  ops/s
RPCAvroBenchmark.getPersonLinks      thrpt   40  1821.527 ± 123.062  ops/s
RPCAvroBenchmark.listPersons         thrpt   40  2002.270 ± 178.043  ops/s
RPCAvroBenchmark.programComposition  thrpt   40   231.998 ±  14.091  ops/s

Comparing benchmarks one by one

createPerson
Source Mode Cnt Score Error Units
HttpBenchmark.createPerson thrpt 40 599.976 43.344 ops/s
RPCProtoBenchmark.createPerson thrpt 40 7366.238 825.910 ops/s
RPCAvroBenchmark.createPerson thrpt 40 2090.695 147.251 ops/s
getPerson
Source Mode Cnt Score Error Units
HttpBenchmark.getPerson thrpt 40 680.914 52.230 ops/s
RPCProtoBenchmark.getPerson thrpt 40 7924.244 980.973 ops/s
RPCAvroBenchmark.getPerson thrpt 40 2771.051 253.325 ops/s
getPersonLinks
Source Mode Cnt Score Error Units
HttpBenchmark.getPersonLinks thrpt 40 713.296 43.999 ops/s
RPCProtoBenchmark.getPersonLinks thrpt 40 3708.267 444.061 ops/s
RPCAvroBenchmark.getPersonLinks thrpt 40 1821.527 123.062 ops/s
listPersons
Source Mode Cnt Score Error Units
HttpBenchmark.listPersons thrpt 40 690.897 76.741 ops/s
RPCBenchmark.listPersons thrpt 40 5486.742 465.748 ops/s
RPCAvroBenchmark.listPersons thrpt 40 2002.270 ± 178.043 ops/s
programComposition
Source Mode Cnt Score Error Units
HttpBenchmark.programComposition thrpt 40 73.471 10.356 ops/s
RPCProtoBenchmark.programComposition thrpt 40 608.205 63.402 ops/s
RPCAvroBenchmark.programComposition thrpt 40 231.998 14.091 ops/s

Comparing benchmarks in Charts

You can find the following charts in this jsfiddle.

  • Bar Chart bar-chart-bench

  • Radar Chart radar-chart-bench

Conclusion

Using JMH, we have checked out quickly the performance characteristics for both service architectures trying out on two different ecosystems:

  • Local Environment
  • GCP Environment

As we have seen, the RPC solution is noticeably faster, using both metered kind of serialization methods: Avro and Protocol Buffers. Moreover, when network traffic enters the scene, the differences between HTTP and RPC are bigger.

However, digging into numbers related to RPC, the results turn out different for Avro and Protobuf, running the benchmarks locally and atop GCP: TBD.

About

Comparing HTTP endpoints against mu-rpc services.

License:Apache License 2.0

Languages

Language:Scala 89.6%Language:Jinja 10.4%