This repository contains different Java Virtual Machine (JVM) benchmarks to measure the JVM energy consumption using various off-the-shelf applications implemented with multiple technology stacks.
The objective of this project is to evaluate energy consumption among various JVM distributions by executing custom-made Java programs employing different coding paradigms, as well as off-the-shelf applications such as Spring Boot and Quarkus web-based applications.
While comparing energy consumption across different frameworks is not the primary focus (due to variations in code, making direct comparisons challenging), the goal is to maintain the same application (or code samples) and assess the energy consumption when changing only the runtime or the JVM.
For a comprehensive analysis, we need to record both real-time energy consumption and additionally the internal temperature (reported by available PC hardware sensors) while running the JVM application.
Therefore, both physical and software measurements are needed:
- Physical measurements rely on physical devices (e.g., wall power meters) to capture end-to-end energy consumption.
- Software measurements rely on existing OS/CPU architecture-specific interfaces to report the energy consumption for different parts of the system and the available sensor temperatures.
On system under test machine runs the target JVM application.
When measuring JVM energy consumption, it is crucial to simulate a realistic application workload, ensuring the usage of the application and triggering as many endpoints as possible within a reasonable time interval. Merely starting and stopping the application is insufficient, as it may skip critical factors such as the Garbage Collector footprint and Just-In-Time compiler optimizations, thereby rendering the measurements less relevant. In this context, load test scenarios should be conducted for certain applications, such as Spring Boot and Quarkus web-based applications.
The load testing tool should run on a different host than the target JVM application, otherwise, the energy measurements will be negatively impacted.
On test client machine runs the load testing tool (e.g., wrk) as well as any additional resource needed for the application (e.g., PostgreSQL database).
The network latency between the system under test machine and the test client machine (i.e., round trip time) must be constant and neglectable, that's why a wired connection is preferred.
Energy consumption reporting methods vary depending on the operating system and CPU architecture.
On GNU/Linux, the Running Average Power Limit (RAPL) interface is utilized. RAPL offers power-limiting capabilities and precise energy readings for multiple power domains. Each supported power domain exposes a Machine Specific Register (MSR) containing a 32-bit integer, which is updated at approximately 1-millisecond intervals.
The RAPL power domains that are available on Intel CPUs (and potentially AMD Ryzen CPUs) include:
- Package (PKG) domain: Measures the energy consumption of the entire socket, including all cores, integrated graphics, and uncore components like last-level caches and memory controller.
- Power Plane 0 (PP0) domain: Measures the energy consumption of all processor cores on the socket.
- Power Plane 1 (PP1) domain: Measures the energy consumption of the processor graphics (GPU) on the socket (desktop models only).
- DRAM domain: Measures the energy consumption of the random access memory (RAM) attached to the integrated memory controller.
- PSys domain: Monitors and controls the thermal and power specifications of the entire SoC (System on a Chip). It is especially useful when the power consumption source is neither the CPU nor the GPU. PSys includes power consumption from the package domain, System Agent, PCH, eDRAM, and other domains within a single-socket SoC.
In multi-socket server systems, each socket reports its own RAPL values. For example, a two-socket computing system has separate PKG readings for both packages, separate PP0 readings, and so on.
Depending on the CPU manufacturer and series, some of the RAPL domains might be available or not. Please check your hardware.
On macOS, the powermetrics command is used to display various system metrics, including CPU usage statistics, for different samplers like cpu, gpu, thermal, battery, network, and disk at a specific sampling interval.
Depending on the CPU manufacturer and series, some of the samplers might be available or not. Please check your hardware.
To properly run the scripts, you need to download, install, and properly configure the following tools:
| OS | Covered | Tools |
|---|---|---|
| GNU/Linux | Yes | ps, powerstat (i.e., RAPL interface), wrk |
| macOS | Yes | ps, powermetrics, wrk |
| Windows | No | N/A |
Please ensure that you have sudo (root) access; otherwise, the powerstat and powermetrics commands cannot be executed while measuring the energy consumption during tests.
Linux only: On system under test machine, for a smooth tests execution, we recommend extending the default sudo timeout, which is typically a few minutes depending on the Linux distribution, to a higher value (e.g., 1440 minutes) to accommodate the test durations and avoid the sudo prompt from appearing again while tests are running.
$ sudo visudo
# Add the following line to change the sudo timeout
Defaults timestamp_timeout=1440
For more information, please refer to the How to Change Sudo Timeout Period on Linux tutorial.
| JVM Distribution | Build |
|---|---|
| OpenJDK HotSpot VM | Download |
| GraalVM CE | Download |
| Oracle GraalVM | Download |
| Native-Image | Download |
Azul Prime VM (*) |
Download |
| Eclipse OpenJ9 VM | Download |
(*) - License restrictions might apply
-
Clone the wrk repository, a modern HTTP benchmarking tool used for the load testing.
-
Build wrk from sources
# Install the build prerequisites
$ sudo apt-get install build-essential libssl-dev libz-dev make -y
# Clone and build the wrk repository
$ git clone https://github.com/wg/wrk.git
$ cd wrk
$ sudo make
# Move the wrk executable to PATH
$ sudo cp wrk /usr/local/bin Note: Latencies shown by wrk suffer from Coordinated Omission; nevertheless, we use it to measure maximum throughput.
For accurate latency measurements, complement this with wrk2.
In summary, wrk can be used for measuring maximum throughput, while wrk2 is utilized to capture latency numbers under a sustained load.
The following configuration is required before starting any measurement:
-
Open the config.properties file.
-
Update the specific VM_HOME property for the JDK you intend to use. You don't need to update all of them; only the one you plan to use for compiling and running.
OPENJDK_HOTSPOT_VM_HOME="<path_to_openjdk_hotspot>"
GRAAL_VM_CE_HOME="<path_to_graalvm_ce>"
ORACLE_GRAAL_VM_HOME="<path_to_oracle_graalvm>"
GRAAL_VM_NATIVE_IMAGE_HOME="<path_to_graalvm_native_image>"
AZUL_PRIME_VM_HOME="<path_to_azul_prime_vm>"
ECLIPSE_OPEN_J9_VM_HOME="<path_to_eclipse_openj9>"- Update the specific APP_HOME properties that point to the locally downloaded/installed application.
SPRING_PETCLINIC_HOME="<path_to_spring_petclinic>"
QUARKUS_HIBERNATE_ORM_PANACHE_QUICKSTART_HOME="<path_to_quarkus_quickstart>"During power consumption measurements, it is recommended to follow these guidelines:
- Ensure the machine (or laptop) is fully charged and connected to a stable power source.
- Disable all potential sources of instability, such as scheduled updates, scans, and (Wi-Fi) networking, etc.
- Prevent the machine from entering sleep mode during measurements.
- Avoid running any other user intensive applications simultaneously.
- It is usually recommended not to use the machine for any other tasks while conducting measurements.
This set of measurements captures the idle power consumption, and it is used to understand (and remove) the overhead of the hardware system:
$ cd /baseline-idle-os
$ ./run-baseline.sh [--run-identifier=<run-identifier>] [--duration=<duration>]
Please follow the how-to-run instructions.
This set of measurements relies on specific code patterns to identify the most efficient energy-friendly coding paradigm. It includes the following common patterns:
- Logging patterns
- Memory access patterns
- Exception throwing patterns
- String concatenation patterns
- (Sorting) algorithm
- Virtual calls
- Maximum throughput while using virtual/physical threads
$ cd /java-samples
$ ./run-samples.sh [--jvm-identifier=<jvm-identifier>] [--run-identifier=<run-identifier>] [--duration=<duration>] [--skip-os-tuning] [--skip-build]
Please follow the how-to-run instructions.
This set of measurements uses the off-the-shelf Spring PetClinic application.
- Clone the repository spring-petclinic
- On top of the existing code, apply the custom configurations as explained in the application-readme.md
- Launch the JVM application on the system under test machine:
$ cd /spring-petclinic
$ ./run-application.sh [--jvm-identifier=<jvm-identifier>] [--run-identifier=<run-identifier>] [--duration=<duration>] [--enable-pgo-g1gc] [--skip-os-tuning] [--skip-build]
- After the application has successfully started, launch the
wrkon the test client machine:
$ cd /spring-petclinic
$ ./run-wrk.sh --jvm-identifier=<jvm-identifier> [--run-identifier=<run-identifier>] [--jdk-version=<jdk-version>] [--app-base-url=<app-base-url>] [--wrk-duration=<wrk-duration>] [--wrk-threads=<wrk-threads>]
Please follow the how-to-run instructions.
This set of measurements uses the off-the-shelf Quarkus Hibernate ORM Panache quickstart.
- Clone the repository quarkus-quickstarts
- On top of the existing hibernate-orm-panache-quickstart source module, apply the custom configurations as explained in the application-readme.md
- Launch the JVM application on the system under test machine:
$ cd /quarkus-hibernate-orm-panache-quickstart
$ ./run-application.sh [--jvm-identifier=<jvm-identifier>] [--run-identifier=<run-identifier>] [--duration=<duration>] [--enable-pgo-g1gc] [--skip-os-tuning] [--skip-build]
- After the application has successfully started, launch the
wrkon the test client machine:
$ cd /quarkus-hibernate-orm-panache-quickstart
$ ./run-wrk.sh --jvm-identifier=<jvm-identifier> [--run-identifier=<run-identifier>] [--jdk-version=<jdk-version>] [--app-base-url=<app-base-url>] [--wrk-duration=<wrk-duration>] [--wrk-threads=<wrk-threads>]
Please follow the how-to-run instructions.
./plot-results.sh
The plots are saved under results/jdk-$JDK_VERSION/$ARCH/$OS/plot directory.
Please see the LICENSE file for full license.
JVM Energy Consumption
MIT License
Copyright (c) 2023-2024 Ionut Balosin, Ko Turk
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.