Eraser error

Question

Eraser error

whober0521 opened this issue 10 months ago · comments

What steps did you take and what happened:
kubectl apply -f ./"${eraserFile}"

eraserFile:

apiVersion: eraser.sh/v1alpha1
kind: ImageList
metadata:
  name: imagelist
spec:
  # POAM hotfix
  images:
    - arosvc.azurecr.io/genevasecpackinstall@sha256:40448ad550751e2255c86d814228e5654422faba1d5fb774ce15ccae7e24e89d
    - arosvc.azurecr.io/genevasecpackinstall@sha256:d174c3b97cc9fd95fad3d518bb3f7e72f4a5e1c7e436b34fb9525977c8633e05

after running above command, it's still in the crictl image list

# crictl images
IMAGE                                                                         TAG                                    IMAGE ID            SIZE
arosvc.azurecr.io/acskeyvaultagent                                            mariner_20230706.2                     3e170db1052ed       118MB
arosvc.azurecr.io/distroless/genevamdm                                        2.2023.721.1630-e50918-20230721t1737   247206bad4692       23MB
arosvc.azurecr.io/distroless/genevamdsd                                       mariner_20230719.1                     a83452a979ef5       93.9MB
arosvc.azurecr.io/genevasecpackinstall                                        master_20230706.1                      0af62431131aa       52MB
arosvc.azurecr.io/oss/fluent/fluent-bit                                       2.0.14-6                               32f6ed5db233e       33.4MB
arosvc.azurecr.io/shared/prom-mdm-converter                                   distroless-2.0.master.20230611.1       9e9b79aa9e357       8.91MB

What did you expect to happen:
it should no longer show in crictl image list

Anything else you would like to add:
checked logs in eraser controller, it shows the following error:
{"level":"error","ts":1691210574.7192776,"msg":"Reconciler error","controller":"imagejob-controller","object":{"name":"imagejob-bpxzn"},"namespace":"","name":"imagejob-bpxzn","reconcileID":"ab5fd3a0-adcf-43f6-802d-48ababc0bbec","error":"reconcile new: PodTemplate \"imagejob-bpxzn\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:235"}

Environment:
Production

Eraser version:
Image: mcr.microsoft.com/oss/eraser/eraser-manager:v1.1.1
Kubernetes version: (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.4", GitCommit:"fa3d7990104d7c1f16943a67f11b154b71f6a132", GitTreeState:"clean", BuildDate:"2023-07-19T12:20:54Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.6", GitCommit:"94c50547e633f1db5d4c56b2b305670e14987d59", GitTreeState:"clean", BuildDate:"2023-06-12T18:46:30Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}

Jenny Chen · Answer 1 · Tue Aug 08 2023 05:59:46 GMT+0800 (China Standard Time)

full logs

kubectl logs eraser-controller-manager-544fb6b45d-pkqqv -n kube-system
{"level":"info","ts":1691118810.5730023,"logger":"setup","msg":"setting up manager","userAgent":"eraser/manager/v1.1.1 (linux/amd64) 8307320/1688163047"}
{"level":"info","ts":1691118820.621814,"logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8889"}
{"level":"info","ts":1691118820.6221611,"logger":"setup","msg":"setup controllers"}
{"level":"info","ts":1691118820.6222107,"logger":"controllerRuntimeLogger","msg":"set up with manager"}
{"level":"info","ts":1691118820.6223655,"logger":"controller","msg":"add collector controller","process":"imagecollector-controller"}
{"level":"info","ts":1691118820.6224978,"logger":"setup","msg":"starting manager"}
{"level":"info","ts":1691118820.6225588,"logger":"controller","msg":"Queueing first ImageCollector reconcile...","process":"imagecollector-controller"}
{"level":"info","ts":1691118820.6229966,"msg":"Starting server","path":"/metrics","kind":"metrics","addr":"[::]:8889"}
{"level":"info","ts":1691118820.623009,"msg":"Starting server","kind":"health probe","addr":"[::]:8081"}
{"level":"info","ts":1691118820.6232395,"msg":"Starting EventSource","controller":"imagelist-controller","source":"kind source: *v1.ImageList"}
{"level":"info","ts":1691118820.6234748,"msg":"Starting EventSource","controller":"imagelist-controller","source":"kind source: *v1.ImageJob"}
{"level":"info","ts":1691118820.6232398,"msg":"Starting EventSource","controller":"imagelist-controller","source":"kind source: *v1.ConfigMap"}
{"level":"info","ts":1691118820.6235461,"msg":"Starting Controller","controller":"imagelist-controller"}
{"level":"info","ts":1691118820.6235657,"msg":"Starting Controller","controller":"imagelist-controller"}
{"level":"info","ts":1691118820.62341,"msg":"Starting EventSource","controller":"imagejob-controller","source":"kind source: *v1.ImageJob"}
{"level":"info","ts":1691118820.6233,"msg":"Starting EventSource","controller":"imagecollector-controller","source":"kind source: *v1.ImageJob"}
{"level":"info","ts":1691118820.6236458,"msg":"Starting EventSource","controller":"imagejob-controller","source":"kind source: *v1.Pod"}
{"level":"info","ts":1691118820.6237454,"msg":"Starting EventSource","controller":"imagejob-controller","source":"kind source: *v1.PodTemplate"}
{"level":"info","ts":1691118820.6237268,"msg":"Starting EventSource","controller":"imagecollector-controller","source":"channel source: 0xc0002a4dc0"}
{"level":"info","ts":1691118820.6237729,"msg":"Starting Controller","controller":"imagejob-controller"}
{"level":"info","ts":1691118820.6237967,"msg":"Starting Controller","controller":"imagecollector-controller"}
{"level":"info","ts":1691118820.7241788,"msg":"Starting workers","controller":"imagelist-controller","worker count":1}
{"level":"info","ts":1691118820.7245595,"msg":"Starting workers","controller":"imagecollector-controller","worker count":1}
{"level":"info","ts":1691118820.7249093,"logger":"controller","msg":"ImageCollector Reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691118820.725826,"msg":"Starting workers","controller":"imagejob-controller","worker count":1}
{"level":"info","ts":1691118820.7258694,"msg":"Starting workers","controller":"imagelist-controller","worker count":1}
{"level":"info","ts":1691118820.725948,"logger":"controller","msg":"Ignoring unsupported imagelist name","process":"imagelist-controller","name":"acskeyvault"}
{"level":"info","ts":1691118820.7329838,"logger":"KubeAPIWarningLogger","msg":"v1alpha1 of the eraser API has been deprecated. Please migrate to v1."}
{"level":"info","ts":1691118820.7431626,"logger":"controller","msg":"Successfully created collector ImageJob","process":"imagecollector-controller","job":"imagejob-qllq4"}
{"level":"info","ts":1691118820.7431872,"logger":"controller","msg":"done reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691118820.858418,"logger":"controller","msg":"Started collector pod on node","process":"imagejob-controller","job":"imagejob-qllq4","node":"aks-system-48383240-vmss00000p","nodeName":"aks-system-48383240-vmss00000p"}
{"level":"info","ts":1691118820.8767493,"logger":"controller","msg":"Started collector pod on node","process":"imagejob-controller","job":"imagejob-qllq4","node":"aks-system-48383240-vmss000013","nodeName":"aks-system-48383240-vmss000013"}
{"level":"info","ts":1691118820.8917067,"logger":"controller","msg":"Started collector pod on node","process":"imagejob-controller","job":"imagejob-qllq4","node":"aks-system-48383240-vmss00001c","nodeName":"aks-system-48383240-vmss00001c"}
{"level":"info","ts":1691118820.9047143,"logger":"controller","msg":"Started collector pod on node","process":"imagejob-controller","job":"imagejob-qllq4","node":"aks-worker-48383240-vmss000005","nodeName":"aks-worker-48383240-vmss000005"}
{"level":"info","ts":1691118820.9210706,"logger":"controller","msg":"Started collector pod on node","process":"imagejob-controller","job":"imagejob-qllq4","node":"aks-worker-48383240-vmss000006","nodeName":"aks-worker-48383240-vmss000006"}
{"level":"info","ts":1691118820.933727,"logger":"controller","msg":"Started collector pod on node","process":"imagejob-controller","job":"imagejob-qllq4","node":"aks-worker-48383240-vmss000007","nodeName":"aks-worker-48383240-vmss000007"}
{"level":"info","ts":1691119065.853171,"logger":"controller","msg":"Marking job as failed","process":"imagejob-controller","success ratio":1,"actual ratio":0}
{"level":"info","ts":1691119065.8610322,"logger":"controller","msg":"ImageCollector Reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691119065.8610723,"logger":"controller","msg":"failed phase","process":"imagecollector-controller"}
{"level":"info","ts":1691119065.8734987,"logger":"controller","msg":"done reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691119065.8735695,"logger":"controller","msg":"ImageCollector Reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691119065.8735979,"logger":"controller","msg":"failed phase","process":"imagecollector-controller"}
{"level":"info","ts":1691119065.8736167,"logger":"controller","msg":"Delaying imagejob delete","process":"imagecollector-controller","job":"imagejob-qllq4","deleteAter":"2023-08-05 03:17:45 +0000 UTC"}
{"level":"info","ts":1691119065.8737254,"logger":"controller","msg":"done reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691196486.1711106,"logger":"controller","msg":"ImageCollector Reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691196486.1712375,"logger":"controller","msg":"failed phase","process":"imagecollector-controller"}
{"level":"info","ts":1691196486.171251,"logger":"controller","msg":"Delaying imagejob delete","process":"imagecollector-controller","job":"imagejob-qllq4","deleteAter":"2023-08-05 03:17:45 +0000 UTC"}
{"level":"info","ts":1691196486.1713245,"logger":"controller","msg":"done reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691205465.0014846,"logger":"controller","msg":"ImageCollector Reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691205465.0015907,"logger":"controller","msg":"failed phase","process":"imagecollector-controller"}
{"level":"info","ts":1691205465.001602,"logger":"controller","msg":"Deleting imagejob","process":"imagecollector-controller","job":"imagejob-qllq4"}
{"level":"info","ts":1691205465.0108783,"logger":"controller","msg":"end job deletion","process":"imagecollector-controller"}
{"level":"info","ts":1691205465.0109048,"logger":"controller","msg":"done reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691205465.016293,"logger":"controller","msg":"ImageCollector Reconcile","process":"imagecollector-controller"}
{"level":"error","ts":1691205465.0225906,"msg":"Reconciler error","controller":"imagejob-controller","object":{"name":"imagejob-tsfjr"},"namespace":"","name":"imagejob-tsfjr","reconcileID":"9b1b6a50-4320-4f15-8d0a-eade5e4afc24","error":"reconcile new: PodTemplate "imagejob-tsfjr" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:235"}
{"level":"error","ts":1691205465.028301,"msg":"Reconciler error","controller":"imagejob-controller","object":{"name":"imagejob-tsfjr"},"namespace":"","name":"imagejob-tsfjr","reconcileID":"c14ba1a1-e7f9-45c6-8b02-bdebbd26da44","error":"reconcile new: PodTemplate "imagejob-tsfjr" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:235"}
{"level":"info","ts":1691205465.0390232,"logger":"controller","msg":"Successfully created collector ImageJob","process":"imagecollector-controller","job":"imagejob-tsfjr"}
{"level":"info","ts":1691205465.039312,"logger":"controller","msg":"done reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691205465.0708961,"logger":"controller","msg":"Started collector pod on node","process":"imagejob-controller","job":"imagejob-tsfjr","node":"aks-system-48383240-vmss000013","nodeName":"aks-system-48383240-vmss000013"}
{"level":"info","ts":1691205465.08421,"logger":"controller","msg":"Started collector pod on node","process":"imagejob-controller","job":"imagejob-tsfjr","node":"aks-system-48383240-vmss00001c","nodeName":"aks-system-48383240-vmss00001c"}
{"level":"info","ts":1691205465.0984986,"logger":"controller","msg":"Started collector pod on node","process":"imagejob-controller","job":"imagejob-tsfjr","node":"aks-worker-48383240-vmss000005","nodeName":"aks-worker-48383240-vmss000005"}
{"level":"info","ts":1691205465.1133313,"logger":"controller","msg":"Started collector pod on node","process":"imagejob-controller","job":"imagejob-tsfjr","node":"aks-worker-48383240-vmss000006","nodeName":"aks-worker-48383240-vmss000006"}
{"level":"info","ts":1691205465.126368,"logger":"controller","msg":"Started collector pod on node","process":"imagejob-controller","job":"imagejob-tsfjr","node":"aks-worker-48383240-vmss000007","nodeName":"aks-worker-48383240-vmss000007"}
{"level":"info","ts":1691205465.1392424,"logger":"controller","msg":"Started collector pod on node","process":"imagejob-controller","job":"imagejob-tsfjr","node":"aks-system-48383240-vmss00000p","nodeName":"aks-system-48383240-vmss00000p"}
{"level":"info","ts":1691205919.1619987,"logger":"controller","msg":"ImageCollector Reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691205919.1620567,"logger":"controller","msg":"completed phase","process":"imagecollector-controller"}
{"level":"info","ts":1691205919.1680925,"logger":"controller","msg":"done reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691205919.1682913,"logger":"controller","msg":"ImageCollector Reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691205919.1683352,"logger":"controller","msg":"completed phase","process":"imagecollector-controller"}
{"level":"info","ts":1691205919.1683445,"logger":"controller","msg":"Delaying imagejob delete","process":"imagecollector-controller","job":"imagejob-tsfjr","deleteAter":"2023-08-05 03:35:19 +0000 UTC"}
{"level":"info","ts":1691205919.1683638,"logger":"controller","msg":"done reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691206519.001185,"logger":"controller","msg":"ImageCollector Reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691206519.0012894,"logger":"controller","msg":"completed phase","process":"imagecollector-controller"}
{"level":"info","ts":1691206519.0013635,"logger":"controller","msg":"Deleting imagejob","process":"imagecollector-controller","job":"imagejob-tsfjr"}
{"level":"info","ts":1691206519.010455,"logger":"controller","msg":"end job deletion","process":"imagecollector-controller"}
{"level":"info","ts":1691206519.010478,"logger":"controller","msg":"done reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691210574.7187757,"logger":"controller","msg":"creating imagejob","process":"imagelist-controller","job":"imagejob-bpxzn"}
{"level":"error","ts":1691210574.7192776,"msg":"Reconciler error","controller":"imagejob-controller","object":{"name":"imagejob-bpxzn"},"namespace":"","name":"imagejob-bpxzn","reconcileID":"ab5fd3a0-adcf-43f6-802d-48ababc0bbec","error":"reconcile new: PodTemplate "imagejob-bpxzn" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:235"}
{"level":"error","ts":1691210574.7254024,"msg":"Reconciler error","controller":"imagejob-controller","object":{"name":"imagejob-bpxzn"},"namespace":"","name":"imagejob-bpxzn","reconcileID":"86b39188-a391-4ef6-a925-35c4ce62f489","error":"reconcile new: PodTemplate "imagejob-bpxzn" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:235"}
{"level":"info","ts":1691210574.753271,"logger":"controller","msg":"Started remover pod on node","process":"imagejob-controller","job":"imagejob-bpxzn","node":"aks-worker-48383240-vmss000005","nodeName":"aks-worker-48383240-vmss000005"}
{"level":"info","ts":1691210574.7623234,"logger":"controller","msg":"Started remover pod on node","process":"imagejob-controller","job":"imagejob-bpxzn","node":"aks-worker-48383240-vmss000006","nodeName":"aks-worker-48383240-vmss000006"}
{"level":"info","ts":1691210574.7737796,"logger":"controller","msg":"Started remover pod on node","process":"imagejob-controller","job":"imagejob-bpxzn","node":"aks-worker-48383240-vmss000007","nodeName":"aks-worker-48383240-vmss000007"}
{"level":"info","ts":1691210574.7837982,"logger":"controller","msg":"Started remover pod on node","process":"imagejob-controller","job":"imagejob-bpxzn","node":"aks-system-48383240-vmss00000p","nodeName":"aks-system-48383240-vmss00000p"}
{"level":"info","ts":1691210574.7944186,"logger":"controller","msg":"Started remover pod on node","process":"imagejob-controller","job":"imagejob-bpxzn","node":"aks-system-48383240-vmss000013","nodeName":"aks-system-48383240-vmss000013"}
{"level":"info","ts":1691210574.8041985,"logger":"controller","msg":"Started remover pod on node","process":"imagejob-controller","job":"imagejob-bpxzn","node":"aks-system-48383240-vmss00001c","nodeName":"aks-system-48383240-vmss00001c"}
{"level":"info","ts":1691210579.0639145,"logger":"controller","msg":"Delaying imagejob delete","process":"imagelist-controller","job":"imagejob-bpxzn","deleteAter":"2023-08-05 04:52:59 +0000 UTC"}
{"level":"info","ts":1691211179.007673,"logger":"controller","msg":"Deleting imagejob","process":"imagelist-controller","job":"imagejob-bpxzn"}
{"level":"info","ts":1691292319.0110085,"logger":"controller","msg":"ImageCollector Reconcile","process":"imagecollector-controller"}
{"level":"error","ts":1691292319.0194101,"msg":"Reconciler error","controller":"imagejob-controller","object":{"name":"imagejob-jqjnm"},"namespace":"","name":"imagejob-jqjnm","reconcileID":"cb2e5c7c-95ad-441f-b8b7-2687d13424db","error":"reconcile new: PodTemplate "imagejob-jqjnm" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:235"}
{"level":"error","ts":1691292319.025499,"msg":"Reconciler error","controller":"imagejob-controller","object":{"name":"imagejob-jqjnm"},"namespace":"","name":"imagejob-jqjnm","reconcileID":"39ff5d2c-3d9c-4ca6-82a3-cfd67e276cd7","error":"reconcile new: PodTemplate "imagejob-jqjnm" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:235"}
{"level":"info","ts":1691292319.0288517,"logger":"controller","msg":"Successfully created collector ImageJob","process":"imagecollector-controller","job":"imagejob-jqjnm"}
{"level":"info","ts":1691292319.028875,"logger":"controller","msg":"done reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691292319.0576613,"logger":"controller","msg":"Started collector pod on node","process":"imagejob-controller","job":"imagejob-jqjnm","node":"aks-system-48383240-vmss00000p","nodeName":"aks-system-48383240-vmss00000p"}
{"level":"info","ts":1691292319.0709596,"logger":"controller","msg":"Started collector pod on node","process":"imagejob-controller","job":"imagejob-jqjnm","node":"aks-system-48383240-vmss000013","nodeName":"aks-system-48383240-vmss000013"}
{"level":"info","ts":1691292319.084194,"logger":"controller","msg":"Started collector pod on node","process":"imagejob-controller","job":"imagejob-jqjnm","node":"aks-system-48383240-vmss00001c","nodeName":"aks-system-48383240-vmss00001c"}
{"level":"info","ts":1691292319.0966063,"logger":"controller","msg":"Started collector pod on node","process":"imagejob-controller","job":"imagejob-jqjnm","node":"aks-worker-48383240-vmss000005","nodeName":"aks-worker-48383240-vmss000005"}
{"level":"info","ts":1691292319.1078837,"logger":"controller","msg":"Started collector pod on node","process":"imagejob-controller","job":"imagejob-jqjnm","node":"aks-worker-48383240-vmss000006","nodeName":"aks-worker-48383240-vmss000006"}
{"level":"info","ts":1691292319.1190374,"logger":"controller","msg":"Started collector pod on node","process":"imagejob-controller","job":"imagejob-jqjnm","node":"aks-worker-48383240-vmss000007","nodeName":"aks-worker-48383240-vmss000007"}
{"level":"info","ts":1691292515.1607168,"logger":"controller","msg":"ImageCollector Reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691292515.1607692,"logger":"controller","msg":"completed phase","process":"imagecollector-controller"}
{"level":"info","ts":1691292515.1664617,"logger":"controller","msg":"done reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691292515.1666987,"logger":"controller","msg":"ImageCollector Reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691292515.166765,"logger":"controller","msg":"completed phase","process":"imagecollector-controller"}
{"level":"info","ts":1691292515.1667757,"logger":"controller","msg":"Delaying imagejob delete","process":"imagecollector-controller","job":"imagejob-jqjnm","deleteAter":"2023-08-06 03:38:35 +0000 UTC"}
{"level":"info","ts":1691292515.1668193,"logger":"controller","msg":"done reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691293115.0004036,"logger":"controller","msg":"ImageCollector Reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691293115.0004935,"logger":"controller","msg":"completed phase","process":"imagecollector-controller"}
{"level":"info","ts":1691293115.0005002,"logger":"controller","msg":"Deleting imagejob","process":"imagecollector-controller","job":"imagejob-jqjnm"}
{"level":"info","ts":1691293115.009217,"logger":"controller","msg":"end job deletion","process":"imagecollector-controller"}
{"level":"info","ts":1691293115.0092475,"logger":"controller","msg":"done reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691378915.0104795,"logger":"controller","msg":"ImageCollector Reconcile","process":"imagecollector-controller"}
{"level":"error","ts":1691378915.0172753,"msg":"Reconciler error","controller":"imagejob-controller","object":{"name":"imagejob-tfdfx"},"namespace":"","name":"imagejob-tfdfx","reconcileID":"90fc30d1-99d6-4dd1-a6cc-e0bcc81e6118","error":"reconcile new: PodTemplate "imagejob-tfdfx" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:235"}
{"level":"error","ts":1691378915.0232081,"msg":"Reconciler error","controller":"imagejob-controller","object":{"name":"imagejob-tfdfx"},"namespace":"","name":"imagejob-tfdfx","reconcileID":"dff35b26-bfa4-43dc-8387-8dddc743d8d7","error":"reconcile new: PodTemplate "imagejob-tfdfx" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:235"}
{"level":"info","ts":1691378915.0280108,"logger":"controller","msg":"Successfully created collector ImageJob","process":"imagecollector-controller","job":"imagejob-tfdfx"}
{"level":"info","ts":1691378915.028047,"logger":"controller","msg":"done reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691378915.0557888,"logger":"controller","msg":"Started collector pod on node","process":"imagejob-controller","job":"imagejob-tfdfx","node":"aks-worker-48383240-vmss000005","nodeName":"aks-worker-48383240-vmss000005"}
{"level":"info","ts":1691378915.0702238,"logger":"controller","msg":"Started collector pod on node","process":"imagejob-controller","job":"imagejob-tfdfx","node":"aks-worker-48383240-vmss000006","nodeName":"aks-worker-48383240-vmss000006"}
{"level":"info","ts":1691378915.0839422,"logger":"controller","msg":"Started collector pod on node","process":"imagejob-controller","job":"imagejob-tfdfx","node":"aks-worker-48383240-vmss000007","nodeName":"aks-worker-48383240-vmss000007"}
{"level":"info","ts":1691378915.095335,"logger":"controller","msg":"Started collector pod on node","process":"imagejob-controller","job":"imagejob-tfdfx","node":"aks-system-48383240-vmss00000p","nodeName":"aks-system-48383240-vmss00000p"}
{"level":"info","ts":1691378915.1071675,"logger":"controller","msg":"Started collector pod on node","process":"imagejob-controller","job":"imagejob-tfdfx","node":"aks-system-48383240-vmss000013","nodeName":"aks-system-48383240-vmss000013"}
{"level":"info","ts":1691378915.119608,"logger":"controller","msg":"Started collector pod on node","process":"imagejob-controller","job":"imagejob-tfdfx","node":"aks-system-48383240-vmss00001c","nodeName":"aks-system-48383240-vmss00001c"}
{"level":"info","ts":1691379365.5571933,"logger":"controller","msg":"ImageCollector Reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691379365.5572739,"logger":"controller","msg":"completed phase","process":"imagecollector-controller"}
{"level":"info","ts":1691379365.5631778,"logger":"controller","msg":"done reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691379365.5632555,"logger":"controller","msg":"ImageCollector Reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691379365.5633008,"logger":"controller","msg":"completed phase","process":"imagecollector-controller"}
{"level":"info","ts":1691379365.5633287,"logger":"controller","msg":"Delaying imagejob delete","process":"imagecollector-controller","job":"imagejob-tfdfx","deleteAter":"2023-08-07 03:46:05 +0000 UTC"}
{"level":"info","ts":1691379365.5633597,"logger":"controller","msg":"done reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691379965.0012538,"logger":"controller","msg":"ImageCollector Reconcile","process":"imagecollector-controller"}
{"level":"info","ts":1691379965.0013316,"logger":"controller","msg":"completed phase","process":"imagecollector-controller"}
{"level":"info","ts":1691379965.0013433,"logger":"controller","msg":"Deleting imagejob","process":"imagecollector-controller","job":"imagejob-tfdfx"}
{"level":"info","ts":1691379965.0127265,"logger":"controller","msg":"end job deletion","process":"imagecollector-controller"}
{"level":"info","ts":1691379965.012781,"logger":"controller","msg":"done reconcile","process":"imagecollector-controller"}

Sertaç Özercan · Answer 2 · Tue Aug 08 2023 06:20:22 GMT+0800 (China Standard Time)

can you post the relevant output of ctr -n k8s.io images list too?

did you try with an image with a tag instead of only sha?

do you see a remover pod after you deploy an imagelist? if so, please post logs

Jenny Chen · Answer 3 · Tue Aug 08 2023 06:46:59 GMT+0800 (China Standard Time)

ctr -n k8s.io images list

REF TYPE DIGEST SIZE PLATFORMS LABELS
arosvc.azurecr.io/genevasecpackinstall:master_20230706.1 application/vnd.docker.distribution.manifest.v2+json sha256:40448ad550751e2255c86d814228e5654422faba1d5fb774ce15ccae7e24e89d 49.6 MiB linux/amd64 io.cri-containerd.image=managed
arosvc.azurecr.io/genevasecpackinstall@sha256:40448ad550751e2255c86d814228e5654422faba1d5fb774ce15ccae7e24e89d application/vnd.docker.distribution.manifest.v2+json sha256:40448ad550751e2255c86d814228e5654422faba1d5fb774ce15ccae7e24e89d 49.6 MiB linux/amd64 io.cri-containerd.image=managed

Jenny Chen · Answer 4 · Tue Aug 08 2023 06:47:44 GMT+0800 (China Standard Time)

did you try with an image with a tag instead of only sha?

yes, we tried in INT, it works both using tag and sha
and it works in central us euap too

Jenny Chen · Answer 5 · Tue Aug 08 2023 06:49:29 GMT+0800 (China Standard Time)

do you see a remover pod after you deploy an imagelist? if so, please post logs

we ran it using pipeline, we didn't check the logs in remover pod

Sertaç Özercan · Answer 6 · Tue Aug 08 2023 07:30:34 GMT+0800 (China Standard Time)

@whober0521 sounds like this issue is about removing with just SHA? It behaves as expected using a tag (or tag+sha), and removes images successfully.

what is your use case to just remove using sha but not tag?

Jenny Chen · Answer 7 · Tue Aug 08 2023 07:39:02 GMT+0800 (China Standard Time)

No, we removed it using just SHA successfully in central us euap.
We uses SHA because that's what shown on POAM dashboard

Sertaç Özercan · Answer 8 · Tue Aug 08 2023 09:00:32 GMT+0800 (China Standard Time)

If you are able to succesfully remove using sha, i am not sure if this would be a bug on eraser side. Please make sure to check your environment and remover pod logs. If you can provide a repro, that would be great.

Sertaç Özercan · Answer 9 · Sat Sep 30 2023 03:23:19 GMT+0800 (China Standard Time)

@ashnamehrotra can we close this with v1.2.2?

Ashna Mehrotra · Answer 10 · Sat Sep 30 2023 04:21:42 GMT+0800 (China Standard Time)

fixed in #875