TestStep delete syntax is inconsistent and not validated, may lead to deleting More than Intended Resource

[mstatv]

What happened: Went attempting to test the deletion of a Namespace, if the test author incorrectly identified the object (namespace in this case) to be deleted, KuTTL seems to interpret this as a 'file glob' for lack of a better way to put it. Shown by the error when attempting to delete the default namespace - audit logs will show a 403 occurs - ceasing the TestStep Execution.

What you expected to happen:
For Namespaces or Cluster Scoped resources, the binary would default to validating for specific instances of the k8s API rather than assuming you meant all.

How to reproduce it (as minimally and precisely as possible):

➜ tree
.
├── kuttl-test.yaml
├── setup
└── wrong
    └── 00-test
        └── 00-test-step.yaml

setup

#!/usr/bin/env bash

namespaces=( a-project a-project2 b-project b-project2 c-project c-project2 )

echo "Setting up test namespaces..."
for i in "${namespaces[@]}"
do
  kubectl create namespace ${i}
done

echo "Running Kuttl...."
kubectl kuttl test wrong/ -v 5

kuttl-test.yaml

apiVersion: kuttl.dev/v1beta1
kind: TestSuite
testDirs:
- wrong/
timeout: 120

wrong/00-test/00-test-step.yaml

apiVersion: kuttl.dev/v1beta1
kind: TestStep
delete:
# commenting out namespace: ... produces same result
- apiVersion: v1
  kind: Namespace
  metadata:
    name: a-project
    namespace: a-project

Now for the run and output...

kuttl-error …
➜ setup

Current Namespaces...
NAME              STATUS   AGE
default           Active   21d
kube-node-lease   Active   21d
kube-public       Active   21d
kube-system       Active   21d

Setting up test namespaces...
namespace/a-project created
namespace/a-project2 created
namespace/b-project created
namespace/b-project2 created
namespace/c-project created
namespace/c-project2 created

Running KuTTL...
2022/12/22 16:53:22 kutt-test config testdirs is overridden with args: [ wrong/ ]
=== RUN   kuttl
    harness.go:459: starting setup
    harness.go:250: running tests using configured kubeconfig.
    harness.go:287: Successful connection to cluster at: https://192.168.39.70:8443
    harness.go:355: running tests
    harness.go:73: going to run test suite with timeout of 120 seconds for each step
    harness.go:367: testsuite: wrong/ has 1 tests
=== RUN   kuttl/harness
=== RUN   kuttl/harness/00-test
=== PAUSE kuttl/harness/00-test
=== CONT  kuttl/harness/00-test
    logger.go:42: 16:53:23 | 00-test | Creating namespace: kuttl-test-endless-jawfish
    logger.go:42: 16:53:23 | 00-test/0-test-step | starting test step 0-test-step
    case.go:362: failed in step 0-test-step
    case.go:364: namespaces "default" is forbidden: this namespace may not be deleted
I1222 16:53:24.146376   36209 request.go:601] Waited for 1.047013661s due to client-side throttling, not priority and fairness, request: GET:https://192.168.39.70:8443/apis/scheduling.k8s.io/v1?timeout=32s
    logger.go:42: 16:53:24 | 00-test | Failed to collect events for 00-test in ns kuttl-test-endless-jawfish: no matches for kind "Event" in version "events.k8s.io/v1beta1"
    logger.go:42: 16:53:24 | 00-test | Trying with events eventsv1 API...
    logger.go:42: 16:53:24 | 00-test | 00-test events from ns kuttl-test-endless-jawfish:
    logger.go:42: 16:53:24 | 00-test | Deleting namespace: kuttl-test-endless-jawfish
=== CONT  kuttl
    harness.go:401: run tests finished
    harness.go:510: cleaning up
    harness.go:567: removing temp folder: ""
--- FAIL: kuttl (1.27s)
    --- FAIL: kuttl/harness (0.00s)
        --- FAIL: kuttl/harness/00-test (1.22s)
FAIL

kuttl-error took 14.2s …
➜ kubectl get namespaces
NAME              STATUS   AGE
default           Active   21d
kube-node-lease   Active   21d
kube-public       Active   21d
kube-system       Active   21d

Anything else we need to know?:
I can fix the test and get this to run as intended. I just think that as a tool the default behavior shouldn't be as shown for Namespaces.
The reason for testing Namespace deletion is for Cluster Scoped CRDs in development.

Environment:

• Kubernetes version (use kubectl version): 1.25.2
• KUTTL version (use kubectl kuttl version): 0.13.1 && 0.14.0
• Cloud provider or hardware configuration: Running on VM
• OS (e.g. from /etc/os-release): Fedora release 36, RHEL 8.6
• Kernel (e.g. uname -a): Linux fedora 6.0.14-200.fc36.x86_64 | RHEL 4.18.0-372.9.1
• Install tools: krew and manual
• Others: N/A

[Marcin Owsiany]

I just hit this as well. The reason is that the syntax for resources to be deleted is not:

apiVersion: kuttl.dev/v1beta1
kind: TestStep
delete:
- apiVersion: v1
  kind: Namespace
  metadata:
    name: a-project

It's:

apiVersion: kuttl.dev/v1beta1
kind: TestStep
delete:
- apiVersion: v1
  kind: Namespace
  name: a-project

It's a deviation from the k8s standard way, and furthermore kuttl does not fail on an unexpected metadata field.
It just ignores it, meaning all it sees is:

- apiVersion: v1
  kind: Namespace

and joyfully deletes ALL namespaces.

There are a few ways to fix this, I guess the least intrusive would be to just support metadata.name as well...