fidelity / kraan

Kraan is a Kubernetes Controller that manages the deployment of HelmReleases to a cluster.

Home Page:https://fidelity.github.io/kraan/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

kraan-controller fails to detect Helm Release deployment failures

nab-gha opened this issue · comments

When running tests using new script to run kraan-controller on workstation against local files (https://github.com/fidelity/kraan/pull/35/files#diff-1818c11d00240758e549449086e232c6) Helm release in bootstrap layer are failing, see #36 but kraan-controller marks the layer deployed.

kubectl get addonslayers.kraan.io bootstrap -o json| jq -r '.spec,.status'
{
  "hold": false,
  "interval": "1m",
  "prereqs": {
    "k8sVersion": "v1.16"
  },
  "source": {
    "name": "addons-config",
    "namespace": "gitops-system",
    "path": "./testdata/addons/bootstrap"
  },
  "version": "0.1.01"
}
{
  "conditions": [
    {
      "lastTransitionTime": "2020-09-02T17:01:04Z",
      "message": "The k8sVersion status means the manager has detected that the AddonsLayer needs a higher version of the Kubernetes API than the current version running on the cluster.",
      "reason": "AddonsLayer is waiting for the required K8sVersion",
      "status": "True",
      "type": "K8sVersion",
      "version": "0.1.01"
    },
    {
      "lastTransitionTime": "2020-09-03T08:16:40Z",
      "message": "source directory (/repos/addons-config/./testdata/addons/bootstrap) not found for AddonsLayer bootstrap",
      "reason": "AddonsLayer processsing has failed",
      "status": "True",
      "type": "Failed",
      "version": "0.1.01"
    },
    {
      "lastTransitionTime": "2020-09-03T09:56:14Z",
      "message": "The applying status means the manager is either applying the yaml files or waiting for the HelmReleases to successfully deploy.",
      "reason": "AddonsLayer is being applied",
      "status": "True",
      "type": "Applying",
      "version": "0.1.01"
    },
    {
      "lastTransitionTime": "2020-09-03T10:10:43Z",
      "message": "error from kubectl while parsing source directory (/tmp/kraan-LQ1JMB/addons-config/./testdata/addons/bootstrap) for AddonsLayer bootstrap: error executing kubectl command '/usr/local/bin/kubectl apply -R -f /tmp/kraan-LQ1JMB/addons-config/./testdata/addons/bootstrap -o json' : exit status 1",
      "reason": "AddonsLayer processsing has failed",
      "status": "True",
      "type": "Failed",
      "version": "0.1.01"
    },
    {
      "lastTransitionTime": "2020-09-03T10:20:16Z",
      "message": "The applying status means the manager is either applying the yaml files or waiting for the HelmReleases to successfully deploy.",
      "reason": "AddonsLayer is being applied",
      "status": "True",
      "type": "Applying",
      "version": "0.1.01"
    },
    {
      "lastTransitionTime": "2020-09-03T10:20:17Z",
      "reason": "AddonsLayer is Deployed",
      "status": "True",
      "type": "Deployed",
      "version": "0.1.01"
    },
    {
      "lastTransitionTime": "2020-09-03T10:40:45Z",
      "message": "The pruning status means the manager is pruning objects removed from this layer",
      "reason": "AddonsLayer is being pruned",
      "status": "True",
      "type": "Pruning",
      "version": "0.1.01"
    },
    {
      "lastTransitionTime": "2020-09-03T10:40:48Z",
      "message": "The applying status means the manager is either applying the yaml files or waiting for the HelmReleases to successfully deploy.",
      "reason": "AddonsLayer is being applied",
      "status": "True",
      "type": "Applying",
      "version": "0.1.01"
    },
    {
      "lastTransitionTime": "2020-09-03T10:40:52Z",
      "reason": "AddonsLayer is Deployed",
      "status": "True",
      "type": "Deployed",
      "version": "0.1.01"
    }
  ],
  "state": "Deployed",
  "version": "0.1.01"
}

I think we're just missing the mapping function right now. Since I've added the owner reference index, I should be able to resolve this by re-using that index in the mapping function.

My reading of the code is that

func (a KubectlLayerApplier) ApplyWasSuccessful(ctx context.Context, layer layers.Layer) (applyIsRequired bool, err error) {
should check the helm releases are deployed?

fixed this with a workaround using the golang rest client to list all HelmReleases and returning the ones owned by the addon layer being processed, this fix is in #51
Will pursue implementation of solution based on controller manager caching in #52