Improve test stability
foot opened this issue · comments
- Fix flakiness on tests[0][1][2]
[0]
#!/bin/bash -eo pipefail
if [ -z "${CIRCLE_TAG}" -a "${CIRCLE_BRANCH}" == "master" ]; then
export SERVICE_IMAGE="weaveworks/launcher-service:$(docker/image-tag)"
else
export SERVICE_IMAGE=weaveworks/build-tmp-public:launcher-service-$(docker/image-tag)
fi
./integration-tests/tests/install-update-flow.sh
#############################################
### Test installation and creation of WC pods
#############################################
• Set WEAVE_CLOUD_TOKEN if it is not already set
• Start launcher/service on minikube
service "service" created
deployment "service" created
• Wait for launcher/service pod to become ready
• Wait for launcher/service to be fully reachable
• Install Weave Cloud on the minikube cluster
Downloading the Weave Cloud installer from http://10.142.3.61:30080/bootstrap?dist=linux
Preparing for Weave Cloud setup
Checking kubectl & kubernetes versions
Connecting cluster to "Weave Cloud Launcher Integration tests" (id: wandering-star-52) on Weave Cloud
Installing Weave Cloud agents on minikube at https://10.142.3.61:8443
Performing a check of the Kubernetes installation setup.
Successfully installed.
• Wait for weave pods to become ready • Wait for weave/weave-agent
• Wait for weave/kube-state-metrics
• Wait for weave/prom-node-exporter
• Wait for weave/prometheus
• Wait for weave/weave-flux-agent
• Wait for weave/weave-flux-memcached
• Wait for weave/weave-scope-agent
###########################
### Test agent self update
###########################
• Take the current service agent k8s and add a new label to it and reduce the recovery wait to 60s
• Create a configmap for the updated yaml which will be mounted as a volume
Error from server (NotFound): configmaps "agent-k8s" not found
configmap "agent-k8s" created
• Apply the updated service which will use the configmap
service "service" unchanged
deployment "service" configured
• Wait for launcher/service pod to become ready
• Wait for launcher/service to be fully reachable
• Restart the weave-agent pod to force an update
pod "weave-agent-8656f5745-snxn5" deleted
• Wait for the agent to start the self update
• Wait for the agent to finish the self update
• Wait for weave pods to become ready • Wait for weave/weave-agent
• Wait for weave/kube-state-metrics
• Wait for weave/prom-node-exporter
• Wait for weave/prometheus
• Wait for weave/weave-flux-agent
• Wait for weave/weave-flux-memcached
• Wait for weave/weave-scope-agent
• Check the agent updated correctly
##################################
### Test agent self update failure
##################################
• Take the current service agent k8s and set the image to one that does not exist
• Create a configmap for the updated yaml which will be mounted as a volume
NAME DATA AGE
agent-k8s 1 11s
configmap "agent-k8s" deleted
configmap "agent-k8s" created
• Restart the service
pod "service-554667d899-pll5m" deleted
pod "service-64cddd5bfb-s97wk" deleted
• Wait for launcher/service pod to become ready
• Wait for launcher/service to be fully reachable
Waiting, endpoint for service is not ready yet...
• Restart the weave-agent pod to force an update
pod "weave-agent-5c96d6b44c-h7pdw" deleted
pod "weave-agent-6744cc9b9c-qcfmp" deleted
• Wait for the new agent to fail to pull the image
• Wait for the agent to begin recovery
• Wait for the agent to finish recovery
Too long with no output (exceeded 2m0s)
[1]
#!/bin/bash -eo pipefail
if [ -z "${CIRCLE_TAG}" -a "${CIRCLE_BRANCH}" == "master" ]; then
export SERVICE_IMAGE="weaveworks/launcher-service:$(docker/image-tag)"
else
export SERVICE_IMAGE=weaveworks/build-tmp-public:launcher-service-$(docker/image-tag)
fi
./integration-tests/tests/install-update-flow.sh
#############################################
### Test installation and creation of WC pods
#############################################
• Set WEAVE_CLOUD_TOKEN if it is not already set
• Start launcher/service on minikube
service "service" created
deployment "service" created
• Wait for launcher/service pod to become ready
• Wait for launcher/service to be fully reachable
• Install Weave Cloud on the minikube cluster
Downloading the Weave Cloud installer from http://10.142.0.155:30080/bootstrap?dist=linux
Preparing for Weave Cloud setup
Checking kubectl & kubernetes versions
Connecting cluster to "Weave Cloud Launcher Integration tests" (id: wandering-star-52) on Weave Cloud
Installing Weave Cloud agents on minikube at https://10.142.0.155:8443
Performing a check of the Kubernetes installation setup.
There was an error applying the agent: Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
Error from server (BadRequest): the API version in the data (rbac.authorization.k8s.io/v1beta1) does not match the expected API version (rbac.authorization.k8s.io/v1)
Error from server (BadRequest): the API version in the data (rbac.authorization.k8s.io/v1beta1) does not match the expected API version (rbac.authorization.k8s.io/v1)
Full output:
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
namespace "weave" configured
serviceaccount "weave-agent" created
deployment "weave-agent" created
Error from server (BadRequest): the API version in the data (rbac.authorization.k8s.io/v1beta1) does not match the expected API version (rbac.authorization.k8s.io/v1)
Error from server (BadRequest): the API version in the data (rbac.authorization.k8s.io/v1beta1) does not match the expected API version (rbac.authorization.k8s.io/v1)
Rolling back cluster changes
Exited with code 111
[2]
#!/bin/bash -eo pipefail
if [ -z "${CIRCLE_TAG}" -a "${CIRCLE_BRANCH}" == "master" ]; then
export SERVICE_IMAGE="weaveworks/launcher-service:$(docker/image-tag)"
else
export SERVICE_IMAGE=weaveworks/build-tmp-public:launcher-service-$(docker/image-tag)
fi
./integration-tests/tests/install-update-flow.sh
#############################################
### Test installation and creation of WC pods
#############################################
• Set WEAVE_CLOUD_TOKEN if it is not already set
• Start launcher/service on minikube
service "service" created
deployment "service" created
• Wait for launcher/service pod to become ready
• Wait for launcher/service to be fully reachable
• Install Weave Cloud on the minikube cluster
Downloading the Weave Cloud installer from http://10.142.9.218:30080/bootstrap?dist=linux
Preparing for Weave Cloud setup
Checking kubectl & kubernetes versions
Connecting cluster to "Weave Cloud Launcher Integration tests" (id: wandering-star-52) on Weave Cloud
Installing Weave Cloud agents on minikube at https://10.142.9.218:8443
Performing a check of the Kubernetes installation setup.
Successfully installed.
• Wait for weave pods to become ready • Wait for weave/weave-agent
• Wait for weave/kube-state-metrics
• Wait for weave/prom-node-exporter
• Wait for weave/prometheus
• Wait for weave/weave-flux-agent
• Wait for weave/weave-flux-memcached
• Wait for weave/weave-scope-agent
###########################
### Test agent self update
###########################
• Take the current service agent k8s and add a new label to it and reduce the recovery wait to 60s
• Create a configmap for the updated yaml which will be mounted as a volume
Error from server (NotFound): configmaps "agent-k8s" not found
configmap "agent-k8s" created
• Apply the updated service which will use the configmap
service "service" unchanged
deployment "service" configured
• Wait for launcher/service pod to become ready
• Wait for launcher/service to be fully reachable
• Restart the weave-agent pod to force an update
pod "weave-agent-7bc769bcc5-bvpr2" deleted
• Wait for the agent to start the self update
• Wait for the agent to finish the self update
• Wait for weave pods to become ready • Wait for weave/weave-agent
• Wait for weave/kube-state-metrics
• Wait for weave/prom-node-exporter
• Wait for weave/prometheus
• Wait for weave/weave-flux-agent
• Wait for weave/weave-flux-memcached
• Wait for weave/weave-scope-agent
• Check the agent updated correctly
Failed to self update. Label newLabel=foo does not exist.
Exited with code 1