failed to upload child commit to pulp server
yih-redhat opened this issue · comments
Describe the bug
ostree pulp test case https://github.com/virt-s1/rhel-edge/blob/main/ostree-pulp.sh failed frequently (after I added child commit case because pulp bug pulp/pulp_ostree#289 is fixed). If this test case failed, re-run can pass without error, but we still need to find out why.
All failures happened during upload of child commit, here is the job logs:
- https://github.com/virt-s1/rhel-edge/actions/runs/7176103800/job/19540481049 (waiting for task "/pulp/api/v3/tasks/018c5c15-6731-77e8-ad2e-4a72d87cdf4f/" to finish)
- https://github.com/virt-s1/rhel-edge/actions/runs/7178624152/job/19554398570 (waiting for task "/pulp/api/v3/tasks/018c5e17-c37b-7285-9aae-59832ff26f16/" to finish)
- https://github.com/virt-s1/rhel-edge/actions/runs/7254239003/job/19772602470 (failure details : [Code: 11, Reason: failed to upload file "/var/cache/osbuild-worker/output/ceb50129-914d-4ed0-ad9e-6c20245d3ae9-3083257830/commit-archive/commit.tar": 502 Bad Gateway)
From my verify result for pulp bug pulp/pulp_ostree#289, it takes 9 minutes to upload child commit. And for these failed jobs, I think maybe it took long time so osbuild-composer timed out.
Environment
- OS version (
/etc/os-releaseand/etc/redhat-release): RHEL 9.4 - osbuild-composer version (
rpm -qi osbuild-composer): Latest
To Reproduce
Steps to reproduce the behavior:
- for example starting osbuild-composer.service or running composer-cli command, don't forget to include all configuration files you created
Expected behavior
A clear and concise description of what you expected to happen.
Additional context
Add any other context about the problem here.
@achilleas-k @lubosmj Could you please help to take a look of this issue when you have time?
Can we get our hands on the task which has not been finished? Can we increase the timeout a bit?
Regarding the "502 Bad Gateway" error, I am not sure if this is a problem on the Pulp's side.
Dec 19 01:40:54 runner-rhos-01-rhel-9-4-large-9609 osbuild-worker[22699]: time="2023-12-19T01:40:54-05:00" level=info msg="uploading ostree commit to pulp"
+ true
+ sudo composer-cli --json compose info 3a45427f-5166-4c2f-8d5f-ce4724c8c253
+ tee /tmp/tmp.sgiIl0Atqx/compose-info-ostree-commit-519b5f5a-8e2f-4560-9a8e-b195e312e75c.json
++ jq -r '.[0].body.queue_status' /tmp/tmp.sgiIl0Atqx/compose-info-ostree-commit-519b5f5a-8e2f-4560-9a8e-b195e312e75c.json
+ COMPOSE_STATUS=RUNNING
+ [[ RUNNING != RUNNING ]]
+ sleep 5
+ true
+ tee /tmp/tmp.sgiIl0Atqx/compose-info-ostree-commit-519b5f5a-8e2f-4560-9a8e-b195e312e75c.json
+ sudo composer-cli --json compose info 3a45427f-5166-4c2f-8d5f-ce4724c8c253
++ jq -r '.[0].body.queue_status' /tmp/tmp.sgiIl0Atqx/compose-info-ostree-commit-519b5f5a-8e2f-4560-9a8e-b195e312e75c.json
+ COMPOSE_STATUS=RUNNING
+ [[ RUNNING != RUNNING ]]
+ sleep 5
+ true
+ sudo composer-cli --json compose info 3a45427f-5166-4c2f-8d5f-ce4724c8c253
+ tee /tmp/tmp.sgiIl0Atqx/compose-info-ostree-commit-519b5f5a-8e2f-4560-9a8e-b195e312e75c.json
++ jq -r '.[0].body.queue_status' /tmp/tmp.sgiIl0Atqx/compose-info-ostree-commit-519b5f5a-8e2f-4560-9a8e-b195e312e75c.json
+ COMPOSE_STATUS=RUNNING
+ [[ RUNNING != RUNNING ]]
+ sleep 5
Dec 19 01:41:07 runner-rhos-01-rhel-9-4-large-9609 osbuild-worker[22699]: time="2023-12-19T01:41:07-05:00" level=error msg="osbuild job failed: at least one target failed" jobId=ceb50129-914d-4ed0-ad9e-6c20245d3ae9
Dec 19 01:41:07 runner-rhos-01-rhel-9-4-large-9609 osbuild-worker[22699]: time="2023-12-19T01:41:07-05:00" level=error msg="failure details : [Code: 11, Reason: failed to upload file \"/var/cache/osbuild-worker/output/ceb50129-914d-4ed0-ad9e-6c20245d3ae9-3083257830/commit-archive/commit.tar\": 502 Bad Gateway (<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.22.1</center>\r\n</body>\r\n</html>\r\n), Details: org.osbuild.pulp.ostree]" jobId=ceb50129-914d-4ed0-ad9e-6c20245d3ae9
Yes, let's just focus on unfinished task error, what's the timeout for uploading commit task?
@achilleas-k Could you please take a look of this when you have time?
A minimal reproducer would be nice. The test script is a bit too big to iterate on to experiment with this. I'll try to find some time to look into this sometime next week.