redkubes / otomi-core

Application Platform for Kubernetes

Home Page:https://otomi.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Potential race condition for multiple change operations

merll opened this issue · comments

Describe the bug:

Deploying several changes in a row creates multiple Tekton pipeline runs for otomi-pipeline. If an earlier pipeline takes a while, the second pipeline can fail due to conflicting changes.

To Reproduce
Steps to reproduce the behavior:

  1. As a secondary admin user, go to the Apps overview
  2. Activate several apps, such as Harbor, Loki, Grafana
  3. Click Deploy
  4. Open the Istio app and activate Tracing, do the same for Nginx Ingress
  5. Click Deploy once again
  6. Go to the Tekton app / PipelineRuns, see there are two in progress
  7. The second pipeline run eventually fails in the apply step with the message: Error: Failed to render chart: exit status 1: Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

Expected behavior:

The pipelines should work in a sequence, as there can not only be conflicting changes in such pipelines, but also dependent ones (e.g. one app requiring the other). If Tekton cannot handle this, we should consider having a waiting cycle in the pipeline.

Versions: 2.3.1