This tool will perform a benchmark (see below for more details) against a PostgreSQL or MongoDB database.
MongoDB announced as one of the main features for version 4.0, if not the main one, support for multi-document ACID transactions. The goal of this benchmark is to compare an ACID transactional system by default, PostgreSQL, with MongoDB 4.0 using also transactions.
Given that MongoDB’s support for transactions is quite recent, there are no benchmarks ready to exercise this capability. The OnGres team exercised patching the sysbench benchmark using for the OLTP benchmark adding support with transactions. But the effort was not successful, probably due to limitations in the driver used by this benchmark.
To support this analysis, a new benchmark was created from scratch, and its published as open source on this repository. It has been developed in Java with the idea to elaborate on a test/benchmark already proposed by MongoDB. In particular, it was modeled a similar scenario to the one proposed in Introduction to MongoDB Transactions in Python, which lead to the creation of the this software pymongo-transactions.
This benchmark simulates users buying airline tickets, and generating the appropriate records. Instead of fully synthetic data, some real data (see 1) was used based on the one available on the LSV site. This makes the benchmark more likely to represent real-world scenarios. It uses the most popular Java drivers for MongoDB and PostgreSQL - mongo-java-driver and PgJDBC, respectively. The code for the actual tests lives in two files, MongoFlightBenchmark.java and PostgresFlightBenchmark.java. Both databases are generated using custom scripts and the static data (flight schedules and airplane information) is preloaded automatically, before tests are run.
- The original benchmark generated very simple data. In particular, the flight number was hard-coded to a constant value and the seats assigned were purely random. For the benchmark that was developed, a separate table (or collection in MongoDB) was used to load real data from the LSV site containing flight data, and another one with plane data. Data is still very small (15K rows for the flight schedules, and 200 rows for the planes data).
Java 8 JDK and Maven are required to build this project (you can replace all mvn commands with ./mvnw that will launch a provided wrapper that will download and install Maven for ease of use).
Run following command:
mvn clean package
The command will compile source code and generate an uber-JAR archive in ```cli/target/benchmark-.jar``
- Safer: Slower but safer profile used to look for errors before pushing to SCM
mvn verify -P safer
The integration test suite requires that Docker is installed on the system and available to the user. To launch the integrations tests run the following command:
mvn verify -P integration
To run integration tests with Java debugging enabled on port 8000:
mvn verify -P integration -Dmaven.failsafe.debug="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000"
Go to the root folder of the project and run the following commands:
java -jar cli/target/benchmark-<version>.jar -h