Overview ======== MMTests is a configurable test suite that runs performance tests against arbitrary workloads. This is not the only test framework but care is taken to make sure the test configurations are accurate, representative and reproducible. Reporting and analysis is common across all benchmarks. Support exists for gathering additional telemetry while tests are running and hooks exist for more detailed tracing using ftrace or perf. Organisation ============ The top-level directory has a single driver script called run-mmtests.sh which reads a config file that describes how the benchmarks should be configured and executed. In some cases, the same benchmarking tool may be used with different configurations that stresses the scenario. A test name can have any same. A common use case is simply to compare kernel versions but it can be anything -- different compiler, different userspace package, different benchmark configuration etc. Monitors can be optionally configured. A full list is in monitors/ . Care should be taken with monitors as there is a possibility that they introduce overhead of their own. Hence, for some performance sensitive tests it is preferable to have no monitoring. Many of the tests download external benchmarks. An attempt will be made to download from a mirror if it exists. To get an idea where the mirror should be located, grep for MIRROR_LOCATION= in shellpacks/. A basic invocation of the suite is <pre> $ ./bin/autogen-configs $ ./run-mmtests.sh --no-monitor --config configs/config-pagealloc-performance 5.8-vanilla $ ./run-mmtests.sh --no-monitor --config configs/config-pagealloc-performance 5.9-vanilla $ cd work/log $ ../../compare-kernels.sh $ mkdir /tmp/html/ $ ../../compare-kernels.sh --format html --output-dir /tmp/html > /tmp/html/index.html </pre> The first step is optional. Some configurations are auto-generated from a template, particularly the filesystem-specific ones. Note that List::BinarySearch may need to be installed from cpan for the reporting to work. Similarly, R should be installed if attempting to highlight whether performance differences are statistically relevant. Configuration ============= The config file can take many options. Some are universal, others are specific to the test. MMTESTS is a list of what tests will be run SWAP_CONFIGURATION SWAP_PARTITIONS SWAP_SWAPFILE_SIZEMB It's possible to use a different swap configuration than what is provided by default. TESTDISK_RAID_DEVICES TESTDISK_RAID_MD_DEVICE TESTDISK_RAID_OFFSET TESTDISK_RAID_SIZE TESTDISK_RAID_TYPE If the target machine has partitions suitable for configuring RAID, they can be specified here. This RAID partition is then used for all the tests TESTDISK_PARTITION Use this partition for all tests TESTDISK_FILESYSTEM TESTDISK_MKFS_PARAM TESTDISK_MOUNT_ARGS The filesystem, mkfs parameters and mount arguments for the test partitions TESTDISK_DIR A directory passed to the test. If not set, defaults to SHELLPACK_TEMP. The directory is supposed to contain a precreated environment (eg. a specifically created filesystem mounted with desired mount options). STORAGE_CACHE_TYPE STORAGE_CACHING_DEVICE STORAGE_BACKING_DEVICE It's also possible to use storage caching. STORAGE_CACHE_TYPE is either "dm-cache" or "bcache". The devices specified with STORAGE_CACHING_DEVICE and STORAGE_BACKING_DEVICE are used to create the cache device which then is used for all the tests. As MMTests downloads a number of benchmarks it is possible to create a local mirror. The location of the mirror is configured in WEBROOT in shellpacks/common-config.sh. For example, kernbench tries to download $WEBROOT/kernbench/linux-3.0.tar.gz . If this is not available, it is downloaded from the internet. This can add delays in testing and consumes bandwidth so is worth configuring. Organisation ------------ All available configurations are stored in configs/. The install and test scripts are automatically generated from template files stored in shellpack_src/src/. Some have a build suffix indicating that it is only building a supporting tool like a library a benchmark requires. Do not modify the generated test-scripts in shellpacks/ directory as they will simply be overwritten. Reporting ========= For reporting, there is a basic compare-kernels.sh script. It can optionally specify a different baseline and comparison point. By default, table is organised by the time the test was executed. Development =========== Patches should be sent to Mel Gorman <mgorman@techsingularity.net>. While the project is hosted on github, notifications get lost so pull requests there may be missed for quite a long time.