infinitewarp / cloudigrade-perf-test

Summary

This repo just has a one-off script and patches for cloudigrade to test cloudigrade's performance when built over UBI8 versus UBI9. The test specifically focuses on logging because we observed that Watchtower with use_queues=False has catastrophically bad performance when running on UBI9 whereas it had acceptable (but still not great) performance on UBI8.

Results

Output captured on 2022-09-13:

• perf testing ubi8 a63b3b0a (just before first uib9 update)
• perf testing ubi9 da1a2d71 (first problematic uib9 update)
• perf testing ubi8 3145a53c (latest master)
• perf testing ubi9 3145a53c (latest master)

stdout was captured and combined into results/2022-09-13.md.

See also cloudigrade/issues/1314#issuecomment-1249671927.

Usage

This repo has @infinitewarp's personal one-off script designed and tested only on his local system and may not work on yours. Caveat emptor. Hic sunt dracones.

Assumptions/prereqs

1. recent cloudigrade repo lives at ~/projects/cloudigrade
2. script to activate cloudigrade lives at ~/bin/cloudigrade.sh
3. script to export ephemeral-specific env vars lives at ~/.env-files/ephemeral
4. brew install moreutils openshift-cli coreutils httpie
5. Docker or Podman is installed and running.
6. Docker or Podman is logged in with quay.io; it will need to push images.
7. oc CLI is logged in with ephemeral cluster. Request token.
8. Connect to the Red Hat VPN.

Execution

To build and run on the last UBI8 ref before the first UBI9 upgrade:

UBI=8 ./scripts/ubi-perf-test.sh

To build and run on the first ref with UBI9:

UBI=9 ./scripts/ubi-perf-test.sh

To build and run on an arbitrary ref assuming UBI8 (like the latest master):

git -C ~/projects/cloudigrade checkout master && \
git -C ~/projects/cloudigrade pull && \
BASE_REF=$(git -C ~/projects/cloudigrade rev-parse --short HEAD)

UBI=8 BASE_REF="${BASE_REF}" ./scripts/ubi-perf-test.sh

To build and run on an arbitrary ref (like the latest master) but upgrade it to UBI9:

git -C ~/projects/cloudigrade checkout master && \
git -C ~/projects/cloudigrade pull && \
BASE_REF=$(git -C ~/projects/cloudigrade rev-parse --short HEAD)

UBI=9 BASE_REF="${BASE_REF}" ./scripts/ubi-perf-test.sh

Cleanup

Running the script by default always reserves a new ephemeral namespace. You may want to clean these up between runs.

bonfire namespace list --mine
bonfire namespace release -f ephemeral-potato  # etc