Self reporting progress does not work when `workflows.argoproj.io/progress` is set

[jswxstw]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?

The workflow sample in docs: https://argo-workflows.readthedocs.io/en/stable/progress/ does not work.

#8000 this creates a WorkflowTaskResult instead of doing a pod patch. However, podReconciliation still reconcile node progress from annotation workflows.argoproj.io/progress, so node progress reconciled from WorkflowTaskResult is overrided by the initial progress annotation on the pod: 0/100.

Version

latest

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: progress-
spec:
  entrypoint: main
  templates:
    - name: main
      dag:
        tasks:
          - name: progress
            template: progress
    - name: progress
      metadata:
        annotations:
          workflows.argoproj.io/progress: 0/100
      container:
        image: alpine
        imagePullPolicy: IfNotPresent
        command: [ "/bin/sh", "-c" ]
        args:
          - |
            for i in `seq 1 10`; do sleep 10; echo "$(($i*10))"'/100' > $ARGO_PROGRESS_FILE; done

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

Logs from in your workflow's wait container

time="2024-05-16T03:16:59.905Z" level=info msg="Starting Workflow Executor" version=untagged
time="2024-05-16T03:16:59.910Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2024-05-16T03:16:59.911Z" level=info msg="Executor initialized" deadline="2024-05-16 03:21:58 +0000 UTC" includeScriptOutput=false namespace=argo podName=progress-ckdv7-progress-2738936019 templateName=progress version="&Version{Version:untagged,BuildDate:2024-05-09T02:07:19Z,GitCommit:6b16f24c97e4b9c792e18835304e517f2d0ddd0f,GitTag:untagged,GitTreeState:clean,GoVersion:go1.21.10,Compiler:gc,Platform:linux/amd64,}"
time="2024-05-16T03:16:59.917Z" level=debug msg="Create workflowtaskresults 201"
time="2024-05-16T03:16:59.918Z" level=info msg="Starting deadline monitor"
time="2024-05-16T03:17:06.923Z" level=debug msg="Create workflowtaskresults 409"
time="2024-05-16T03:17:06.926Z" level=debug msg="Patch workflowtaskresults 200"
time="2024-05-16T03:17:11.919Z" level=info progress=10/100
time="2024-05-16T03:17:13.923Z" level=debug msg="Create workflowtaskresults 409"
time="2024-05-16T03:17:13.927Z" level=debug msg="Patch workflowtaskresults 200"
time="2024-05-16T03:17:20.922Z" level=debug msg="Create workflowtaskresults 409"
time="2024-05-16T03:17:20.926Z" level=debug msg="Patch workflowtaskresults 200"
time="2024-05-16T03:17:20.926Z" level=info progress=20/100
time="2024-05-16T03:17:27.924Z" level=debug msg="Create workflowtaskresults 409"
time="2024-05-16T03:17:27.927Z" level=debug msg="Patch workflowtaskresults 200"
time="2024-05-16T03:17:32.919Z" level=info progress=30/100
time="2024-05-16T03:17:34.925Z" level=debug msg="Create workflowtaskresults 409"
time="2024-05-16T03:17:34.928Z" level=debug msg="Patch workflowtaskresults 200"
time="2024-05-16T03:17:41.923Z" level=debug msg="Create workflowtaskresults 409"
time="2024-05-16T03:17:41.926Z" level=debug msg="Patch workflowtaskresults 200"
time="2024-05-16T03:17:41.926Z" level=info progress=40/100
time="2024-05-16T03:17:48.923Z" level=debug msg="Create workflowtaskresults 409"
time="2024-05-16T03:17:48.927Z" level=debug msg="Patch workflowtaskresults 200"
time="2024-05-16T03:17:50.919Z" level=info progress=50/100
time="2024-05-16T03:17:55.924Z" level=debug msg="Create workflowtaskresults 409"
time="2024-05-16T03:17:55.927Z" level=debug msg="Patch workflowtaskresults 200"
time="2024-05-16T03:18:02.923Z" level=debug msg="Create workflowtaskresults 409"
time="2024-05-16T03:18:02.927Z" level=debug msg="Patch workflowtaskresults 200"
time="2024-05-16T03:18:02.927Z" level=info progress=60/100
time="2024-05-16T03:18:09.926Z" level=debug msg="Create workflowtaskresults 409"
time="2024-05-16T03:18:09.929Z" level=debug msg="Patch workflowtaskresults 200"
time="2024-05-16T03:18:11.918Z" level=info progress=70/100
time="2024-05-16T03:18:16.922Z" level=debug msg="Create workflowtaskresults 409"
time="2024-05-16T03:18:16.926Z" level=debug msg="Patch workflowtaskresults 200"
time="2024-05-16T03:18:20.919Z" level=info progress=80/100
time="2024-05-16T03:18:23.924Z" level=debug msg="Create workflowtaskresults 409"
time="2024-05-16T03:18:23.927Z" level=debug msg="Patch workflowtaskresults 200"
time="2024-05-16T03:18:30.922Z" level=debug msg="Create workflowtaskresults 409"
time="2024-05-16T03:18:30.926Z" level=debug msg="Patch workflowtaskresults 200"
time="2024-05-16T03:18:32.918Z" level=info progress=90/100
time="2024-05-16T03:18:37.923Z" level=debug msg="Create workflowtaskresults 409"
time="2024-05-16T03:18:37.927Z" level=debug msg="Patch workflowtaskresults 200"
time="2024-05-16T03:18:40.967Z" level=info msg="Main container completed" error="<nil>"
time="2024-05-16T03:18:40.967Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2024-05-16T03:18:40.967Z" level=info msg="No output parameters"
time="2024-05-16T03:18:40.967Z" level=info msg="No output artifacts"
time="2024-05-16T03:18:40.967Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: progress-ckdv7/progress-ckdv7-progress-2738936019/main.log"
time="2024-05-16T03:18:40.967Z" level=info msg="Creating minio client using static credentials" endpoint="minio:9000"
time="2024-05-16T03:18:40.968Z" level=info msg="Saving file to s3" bucket=my-bucket endpoint="minio:9000" key=progress-ckdv7/progress-ckdv7-progress-2738936019/main.log path=/tmp/argo/outputs/logs/main.log
time="2024-05-16T03:18:40.981Z" level=info msg="Save artifact" artifactName=main-logs duration=13.623371ms error="<nil>" key=progress-ckdv7/progress-ckdv7-progress-2738936019/main.log
time="2024-05-16T03:18:40.981Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2024-05-16T03:18:40.981Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2024-05-16T03:18:40.987Z" level=debug msg="Create workflowtaskresults 409"
time="2024-05-16T03:18:40.991Z" level=debug msg="Patch workflowtaskresults 200"
time="2024-05-16T03:18:40.991Z" level=info msg="Alloc=7200 TotalAlloc=13324 Sys=20325 NumGC=5 Goroutines=10"
time="2024-05-16T03:18:40.995Z" level=debug msg="Patch workflowtaskresults 200"