MSalamatov / mongoose

Mongoose is a storage performance testing tool

Contents

1. Overview
2. Features
     2.1. Comparison With Similar Tools
       2.1.1. General
       2.1.2. Purpose
       2.1.3. Scalability
       2.1.4. Input
       2.1.5. Output
       2.1.6. Load Generation Patterns
       2.1.7. Storages Support
     2.2. Scalability
       2.2.1. Vertical
       2.2.2. Horizontal
     2.3. Customization
       2.3.1. Flexible Configuration
       2.3.2. Load Generation Patterns
       2.3.3. Scenarios
       2.3.4. Metrics Reporting
     2.4. Extension
       2.4.1. Load Steps
       2.4.2. Storage Drivers
      2.4.3. Scenario Engine
   
3. Documentation
     3.1. Deployment
     3.2. Usage
     3.2. Troubleshooting
     3.3. Storage Drivers
       3.3.2. S3
       3.3.3. Atmos
       3.3.4. Swift
       3.3.5. FS
     3.4. Dependencies
     3.5. Interfaces
      3.5.1 Input
      3.5.2 Output
      3.5.3 Remote API
       3.5.3.1 Config API
       3.5.3.2 Runs API
       3.5.3.3 Logs API
       3.5.3.4 Metrics API
      3.5.4 Extentions
       3.5.4.1 Load Step Types
       3.5.4.2 Storage Drivers
     3.6. Design
       3.7.1. Architecture
       3.7.2. Distributed Mode
       3.7.3. Installer
       3.7.4. Recycle Mode
       3.7.5. Data Reentrancy
       3.7.6. Byte Range Operations
       3.7.7. Copy Mode
       3.7.8. Pipeline Load
       3.7.9. Weighted Load
     3.8. Contributing
     3.9. Changelog
   

1. Overview

Mongoose is a powerful storage performance testing tool.

It is designed to be used for:

• Load Testing
• Stress Testing
• Soak/Longevity/Endurance Testing
• Volume Testing
• Smoke/Sanity Testing

2. Features

2.1. Comparison With Similar Tools

• COSBench
• LoadRunner
• Locust

2.1.1. General

	Mongoose	COSBench	LoadRunner	Locust
License	MIT License	Apache 2.0	Proprietary	MIT License
Open Source	✔️	✔️	❌	✔️

2.1.2. Purpose

	Mongoose	COSBench	LoadRunner	Locust
Load testing	✔️	✔️	✔️	✔️
Stress testing	✔️	✔️	✔️	TBD
Endurance testing	✔️	TBD	✔️	TBD

2.1.3. Scalability

	Mongoose	COSBench	LoadRunner	Locust
Horizontal (Distributed Mode)	✔️	✔️	TBD	✔️
Vertical (Max sustained concurrency per instance)	1_048_576	1024	TBD	1_000_000

2.1.4. Input

	Mongoose	COSBench	LoadRunner	Locust
GUI	❌	✔️	✔️	✔️
Parameterization	✔️	✔️	TBD	✔️(need to extend the functionality)
Script language	Any JSR-223 compatible	XML	ANSI C, Java, .Net, JS	Python

2.1.5. Output

	Mongoose	COSBench	LoadRunner	Locust
Highest-resolution (per each op) metrics	✔️	❌	TBD	❌
Saturation concurrency measurement	✔️	❌	TBD	❌

2.1.6. Load Generation Patterns

	Mongoose	COSBench	LoadRunner	Locust
Weighted load	✔️	✔️	TBD	❌
Pipeline load	✔️	❌	TBD	❌
Recycle mode	✔️	❌	TBD	❌

2.1.7. Storages Support

• Note: Locust and LoadRunner are not designed for the storage performance testing explicitly so they are excluded from the table below

	Mongoose	COSBench
Supported storages	Amazon S3 EMC Atmos OpenStack Swift Filesystem HDFS	Amazon S3 Amplidata OpenStack Swift Scality Ceph Google Cloud Storage Aliyun OSS
Extensible to support custom storage API	✔️	✔️

2.2. Scalability

2.2.1. Vertical

Using fibers allows to sustain millions of concurrent operations easily without significant performance degradation.

2.2.2. Horizontal

The distributed mode in Mongoose was designed as P2P network. Each peer/node performs independently as much as possible. This eliminates the excess network interaction between the nodes which may be a bottleneck.

2.3. Customization

2.3.1. Flexible Configuration

Supports the parameterization and extension but remains type-safe and structure-safe.

2.3.2. Load Generation Patterns

• CRUD operations and the extensions: Noop, Copy, etc

• Parial Operations

• Composite Operations

• Complex Load Steps

  • Pipeline Load
  • Weighted Load

• Recycle Mode

• Data Reentrancy

  Allows to validate the data read back from the storage successfully even after the data items have been randomly updated multiple times before

• Custom Payload Data

2.3.3. Scenarios

Scenaruis allow to organize the load steps in the required order and reuse the complex performance tests

2.3.4. Metrics Reporting

The metrics reported by Mongoose are designed to be most useful for the performance analysis. The following metrics are available:

• Counts

  • Items
  • Bytes transferred
  • Time
    • Effective
    • Elapsed

• Rates

  • Items per second
  • Bytes per second

• Timing distributions for:

  • Operation durations
  • Network latencies

• Actual concurrency

  It's possible to limit the rate and measure the sustained actual concurrency

The average metrics output is being done periodically while a load step is running. The summary metrics output is done once when a load step is finished. Also, it's possible to obtain the highest precision metrics (for each operation, so called I/O trace records).

2.4. Extension

Mongoose is designed to be agnostic to the particular extensions implementations. This allows to support any storage, scenario language, different load step kinds.

2.4.1. Load Steps

A load step controls the load operations flow. Different load step implementations may do it in the different way. There are the available out-of-the-box:

• Linear The most basic load step. All load operations have the same type.

• Pipeline Executes a sequence of the different load operations for each item independently.

• Weighted Executes the load operations of different types sustaining the specified ratio (weights).

2.4.2. Storage Drivers

The actual load is being executed by the storage drivers. Mongoose supports some storage types out-of-the-box:

• Amazon S3
• EMC Atmos
• OpenStack Swift
• Filesystem

The following additional storage driver implementations are available:

• HDFS

2.4.3. Scenario Engine

Any Mongoose scenario may be written using any JSR-223 compliant scripting language. Javascript support is available out-of-the-box.