akuity / kargo

Application lifecycle orchestration

Home Page:https://kargo.akuity.io/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

e2e tests

krancour opened this issue · comments

I use these besboke examples to validate a broad swath of Kargo functionality prior to release. It's as of fairly recently that I updated the examples such that they can all run concurrently.

I'd like to automate these so they run at least nightly, if not during every CI run.

Proposal:

  1. Update the examples to be something more like example templates so that branch names and such can be set dynamically on a per-test basis. This will avoid us needing to do something more heavy-handed like forking the GitOps repo with the test manifests for every test.
  2. Provision a KinD or k3D cluster in Actions.
  3. Install Kargo and its pre-reqs on that cluster.
    1. This is probably going to require that we start pushing un-validated images to a remote repo as part of the process
  4. Create shell script to assert successful installation
  5. Create a shell script to drive much of the example setup via Kargo CLI.
  6. Assert success two ways:
    1. With shell script using CLI (to maximize e2e coverage of the CLI binary)
    2. With Go tests (because we'll be able to make more detailed assertions this way)
  7. Create a shell script to drive example tear-down via Kargo CLI. We will cover more functionality this way than if we just tear down implicitly via the Action simply concluding and taking the KinD or k3d cluster with it.
  8. Assert success of tear-down two ways:
    1. Shell script
    2. Go tests

This is probably going to require that we start pushing un-validated images to a remote repo as part of the process

You can quite easily load container images into a KinD cluster, which means you do not have to push things to a registry at all.

You can quite easily load container images into a KinD cluster, which means you do not have to push things to a registry at all.

I forgot all about that!

I suppose that will require that the build and e2e tests occur in a single job, but that's ok.