Errors in running "E2E Testing with Kubernetes"
Cbdc2022 opened this issue · comments
Hi,
I tried to run "E2E Testing with Kubernetes" , but all test cases failed with the following error:
Events:
Type Reason Age From Message
Normal Scheduled 7m17s default-scheduler Successfully assigned opencbdc-tx-qg7ep1/shard-0-1-0 to opencbdc
Warning Failed 5m17s (x12 over 7m15s) kubelet Error: ErrImageNeverPull
Warning ErrImageNeverPull 2m6s (x27 over 7m15s) kubelet Container image "opencbdc-tx:latest" is not present with pull policy of Never
Not sure where to get the correct Image, would you please help when have time. thank you.
File "testrun.log" attached.
testrun.log
Yes, I have run the ./scripts/build-docker.sh
FYI:
OS: ubuntu 22.04
tried to run:
E2E Testing with Kubernetes
Requirements
Go (go test library used to run tests)
Minikube
Helm
Kubectl
Running tests
./scripts/test-e2e-minikube.sh
Review results and logs at testruns//
FYI:
./scripts/build-docker.sh
Sending build context to Docker daemon 106.7MB
Step 1/10 : ARG IMAGE_VERSION="ubuntu:20.04"
Step 2/10 : ARG BASE_IMAGE="ghcr.io/mit-dci/opencbdc-tx-base:latest"
Step 3/10 : FROM $IMAGE_VERSION AS base
20.04: Pulling from library/ubuntu
eaead16dc43b: Pull complete
Digest: sha256:450e066588f42ebe1551f3b1a535034b6aa46cd936fe7f2c6b0d72997ec61dbd
Status: Downloaded newer image for ubuntu:20.04
---> 680e5dfb52c7
Step 4/10 : ENV DEBIAN_FRONTEND noninteractive
.......
$ docker images -a --digests
REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE
f217f881db26 20 minutes ago 119MB
5c8c1273d9d3 20 minutes ago 94.7MB
opencbdc-tx-twophase latest b08cde7306d1 20 minutes ago 119MB
87c3318f7a11 20 minutes ago 117MB
d10a8be8a553 20 minutes ago 116MB
f5bef2400852 20 minutes ago 117MB
92f60702b416 20 minutes ago 1.81GB
202277689b37 20 minutes ago 73.6MB
94790781d5dc 20 minutes ago 72.8MB
6b14885d4292 21 minutes ago 1.64GB
opencbdc-tx-base latest 676b02da7baa 21 minutes ago 1.53GB
18a322dc3633 23 minutes ago 72.8MB
65fb0c173c7f 23 minutes ago 72.8MB
ac60d424e32b 23 minutes ago 72.8MB
1932112415d7 23 minutes ago 72.8MB
fea9d7cc6b92 23 minutes ago 72.8MB
1b2a63db5764 23 minutes ago 72.8MB
ubuntu 20.04 sha256:450e066588f42ebe1551f3b1a535034b6aa46cd936fe7f2c6b0d72997ec61dbd 680e5dfb52c7 8 days ago 72.8MB
ghcr.io/mit-dci/opencbdc-tx-base latest sha256:2076bdd505b9a003acebc733c1981ec4ef1ed3d3f2cdea1ba8abb3e1d2250720 49fc6337a4eb 13 days ago 1.54GB
ghcr.io/mit-dci/opencbdc-tx-base sha-16bd61a sha256:2076bdd505b9a003acebc733c1981ec4ef1ed3d3f2cdea1ba8abb3e1d2250720 49fc6337a4eb 13 days ago 1.54GB
opencbdc-tx latest 929938b6244b 2 weeks ago 2.88GB
gcr.io/k8s-minikube/kicbase v0.0.35 sha256:e6f9b2700841634f3b94907f9cfa6785ca4409ef8e428a0322c1781e809d311b 7fb60d0ea30e 4 weeks ago 1.12GB
FYI:
OS: ubuntu 22.04 running on virtual machine - VMware® Workstation 16 Pro (16.2.4 build-20089737)
$ docker version
Client: Docker Engine - Community
Version: 20.10.21
API version: 1.41
Go version: go1.18.7
Git commit: baeda1f
Built: Tue Oct 25 18:01:58 2022
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.21
API version: 1.41 (minimum version 1.12)
Go version: go1.18.7
Git commit: 3056208
Built: Tue Oct 25 17:59:49 2022
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.9
GitCommit: 1c90a442489720eec95342e1789ee8a5e1b9536f
runc:
Version: 1.1.4
GitCommit: v1.1.4-0-g5fd4c4d
docker-init:
Version: 0.19.0
GitCommit: de40ad0
~$ minikube version
minikube version: v1.27.1
commit: fe869b5d4da11ba318eb84a3ac00f336411de7ba
$ helm version
version.BuildInfo{Version:"v3.7.0", GitCommit:"eeac83883cb4014fe60267ec6373570374ce770b", GitTreeState:"clean", GoVersion:"go1.16.8"}
$ kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.25.3
Kustomize Version: v4.5.7
Server Version: v1.25.2
~$ go version
go version go1.18.1 linux/amd64
It works after the following change:
- values.yaml
line 53: repository: ghcr.io/mit-dci/opencbdc-tx --> repositoryVal: opencbdc-tx-twophase - client.yaml
line 11:$.Values.image.repository --> $ .Values.image.repositoryVal - 2pc-component.yaml
line 151, 193:$.Values.image.repository --> $ .Values.image.repositoryVal
The problem is using variable name "repository" will cause issue, maybe "repository" is a keyword or something. I need to change "repository" to "repositoryVal" ( "repository2" also works)
must run ./scripts/build-docker.sh first to create local docker images , then run /scripts/test-e2e-minikube.sh
Interesting. Needing to run ./scripts/build-docker.sh is known and required before running the E2E test suite, but I'm surprised that repository is causing an issue for you in the yaml files (since it's worked for us several times now). Can you tell me more about the setup you have so we can figure out what might be causing that?
sorry for the delay response. not sure what is special on my end. I suspect it is helm. I am using helm with Version:"v3.7.0".
when using variable name "repository", kube pod created for shard (and other components) is looking for image/tag ="opencbdc-tx:latest". However, the image/tag created by "./scripts/build-docker.sh " is "opencbdc-tx-twophase:latest".
Basically, line "repository: opencbdc-tx-twophase" in 'values.yaml" will always create kube pod with image/tag ="opencbdc-tx:latest" regardless what value I assign to variable "repository".
FYI:
trader@trader-virtual-machine:~/Documents/code/opencbdc/opencbdc-tx-trunk$ kubectl describe pod -n opencbdc-tx-lt1ptx shard-0-0-0
Name: shard-0-0-0
Namespace: opencbdc-tx-lt1ptx
Priority: 0
Service Account: opencbdc-tx-updox9-opencbdc-2pc
Node: opencbdc/192.168.49.2
Start Time: Thu, 05 Jan 2023 22:35:38 -0500
Labels: app.kubernetes.io/instance=opencbdc-tx-updox9
app.kubernetes.io/name=shard
controller-revision-hash=shard-0-0-88c8d7f46
opencbdc.hmltn.io/clusterId=0
opencbdc.hmltn.io/id=0
opencbdc.hmltn.io/role=shard
statefulset.kubernetes.io/pod-name=shard-0-0-0
Annotations:
Status: Pending
IP: 172.17.0.36
IPs:
IP: 172.17.0.36
Controlled By: StatefulSet/shard-0-0
Containers:
shard:
Container ID:
Image: opencbdc-tx:latest
Image ID:
Ports: 6666/TCP, 6667/TCP, 6767/TCP
trader@trader-virtual-machine:~/Documents/code/opencbdc/opencbdc-tx-trunk$ docker image ls -a
REPOSITORY TAG IMAGE ID CREATED SIZE
76fa079d53b9 30 minutes ago 119MB
opencbdc-tx-twophase latest 900b435e3bdc 30 minutes ago 119MB
212e859ccdc1 30 minutes ago 117MB
4fd10e616171 30 minutes ago 117MB
7885982b0419 30 minutes ago 116MB
57a719dbc368 31 minutes ago 72.8MB
047ca6a55824 31 minutes ago 2.22GB
2d9bc56ee18b 31 minutes ago 73.6MB
76c166be6354 31 minutes ago 94.7MB
ac8f926e8b6c 32 minutes ago 2.05GB
opencbdc-tx-base latest faf9f9b099b0 32 minutes ago 1.54GB
0bc94695397c 37 minutes ago 72.8MB
e8c395469e4b 37 minutes ago 72.8MB
01811d8d9869 37 minutes ago 72.8MB
298053d1dd17 37 minutes ago 72.8MB
8924c488eb7f 37 minutes ago 72.8MB
f2089c38c8cd 37 minutes ago 72.8MB
ubuntu 20.04 d5447fc01ae6 4 weeks ago 72.8MB
gcr.io/k8s-minikube/kicbase v0.0.35 7fb60d0ea30e 3 months ago 1.12GB
As it's working for me, several other people, and especially because it works in the github CI, I'm very hesitant to change this without knowing the actual cause (or at least knowing for sure that changing the name wouldn't break something else).