ColinIanKing / PerfSpect

System performance characterization tool based on linux perf

PerfSpect

Quick Start | Output | Deploy in Kubernetes | Requirements | Build from source

PerfSpect is a system performance characterization tool built on top of linux perf. It contains two parts:

perf-collect: Collects hardware events at a 5 second output interval with practically zero overhead since PMU's run in counting mode.

• Collection mode:
  • sudo ./perf-collect default system wide
  • sudo ./perf-collect --socket
  • sudo ./perf-collect --cpu
  • sudo ./perf-collect --pid <process-id>
  • sudo ./perf-collect --cid by default, selects the 5 containers using the most CPU at start of perf-collect. To monitor specific containers provide up to 5 comma separated cids i.e. <cid_1>,<cid_2>
• Duration:
  • sudo ./perf-collect default run until terminated
  • sudo ./perf-collect --timeout 10 run for 10 seconds
  • sudo ./perf-collect --app "myapp.sh myparameter" runs for duration of another process

perf-postprocess: Calculates high level metrics from hardware events

• ./perf-postprocess

Quick start (requires perf installed)

wget -qO- https://github.com/intel/PerfSpect/releases/latest/download/perfspect.tgz | tar xvz
cd perfspect
sudo ./perf-collect --timeout 10
./perf-postprocess

Output

perf-collect outputs:

1. perfstat.csv: raw event counts with system metadata

perf-postprocess outputs:

1. metric_out.sys.average.csv: average metrics
2. metric_out.sys.csv: metric values at every 5 second interval
3. metric_out.html: html view of a few select metrics

Deploy in Kubernetes

Modify the template deamonset.yml to deploy in kubernetes

Requirements

Packages:

• perf - PerfSpect uses the Linux perf tool to collect PMU counters

Minimum supported kernels

Xeon Generation	CentOS 7+	Ubuntu 16.04+
Broadwell	3.10	4.15
Skylake	3.10	4.15
Cascade Lake	3.10	4.15
Ice Lake	3.10	4.15
Sapphire Rapids	5.12	5.12

Supported Operating Systems:

• Ubuntu 16.04 and newer
• CentOS 7 and newer
• Amazon Linux 2
• RHEL 9
• Debian 11

Note: PerfSpect may work on other Linux distributions, but has not been thoroughly tested

Build from source

Requires recent python. On successful build, binaries will be created in dist folder

pip3 install -r requirements.txt
make

Note: Most metrics and events come from perfmon and TMA v4.5