oss artifact: get 400 error from OSS server when OSSLifecycleRule is set
AlbeeSo opened this issue · comments
Pre-requisites
- I have double-checked my configuration
- I can confirm the issue exists when I tested with
:latest
- I have searched existing issues and could not find a match for this bug
- I'd like to contribute the fix myself (see contributing guide)
What happened/what did you expect to happen?
OSS does not support to set different rules at a time for a same prefix and that's why I received a 400 error.
And Lifecycle rules of OSS is a little bit complicated which always also need to consider if the bucket is Versioning enabled. (refers to https://www.alibabacloud.com/help/en/oss/developer-reference/manage-versioning-3?spm=a2c63.p38356.0.0.e8367781VCXnwa)
A rule can contains 5 different configurations:
(Refers to https://github.com/aliyun/aliyun-oss-go-sdk/blob/3804c1371ab84adf9bacf7d940281b3fb39deb75/oss/type.go#L70C1-L85C2)
- Expiration: used to configure when to delete the objects (or turn to non-current version if Versioning enabled).
- Transitions: used to configure when to convert the storage type of current version objects.(like Infrequent Access)
- NonVersionExpiration: similar to Expiration but for non-current version objects.
- NonVersionTransitions: similar to Transitions but for non-current version objects.
- AbortMultipartUpload: used to configure when to abort (or delete) the uploaded parts created by a interrupted multipart upload due to network or something. (Parts will reuse if retry to upload again, but will be storaged in OSS if the upload task is given up.)
So, I suggest to modified the rule and only keep the Expiration, AbortMultipartUpload, and Transitions configurations as IMO, if users enable BucketVersioning, deleting the non-version objects by markDeletionAfterDays is unexpected.
Version
latest
Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: set-lifecycle-
spec:
entrypoint: main
templates:
- name: main
metadata:
annotations:
k8s.aliyun.com/eci-extra-ephemeral-storage: "5Gi"
k8s.aliyun.com/eci-use-specs : "ecs.g7.xlarge"
container:
image: alpine:latest
command:
- sh
- -c
args:
- |
mkdir -p /out
dd if=/dev/random of=/out/lifecycle.txt bs=2M count=1024
echo "created files!"
outputs:
artifacts:
- name: out
path: /out/lifecycle.txt
oss:
endpoint: http://oss-cn-zhangjiakou-internal.aliyuncs.com
bucket: my-argo-workflow
key: argo-workflows/test/lifecycle
accessKeySecret:
name: my-argo-workflow-credentials
key: accessKey
secretKeySecret:
name: my-argo-workflow-credentials
key: secretKey
lifecycleRule:
markInfrequentAccessAfterDays: 1
markDeletionAfterDays: 3
Logs from the workflow controller
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}
Logs from in your workflow's wait container
time="2024-04-09T16:04:23.147Z" level=info msg="Save artifact" artifactName=out duration=1.270302632s error="oss: service returned error: StatusCode=400, ErrorCode=InvalidRequest, ErrorMessage=\"Found same prefix in different rules\", RequestId=6615670711BF5732361C11E2, Ec=0014-00000085" key=argo-workflows/test/lifecycletime="2024-04-09T16:04:23.147Z" level=error msg="executor error: oss: service returned error: StatusCode=400, ErrorCode=InvalidRequest, ErrorMessage=\"Found same prefix in different rules\", RequestId=6615670711BF5732361C11E2, Ec=0014-00000085"
Sounds good! Great to know that you are using OSSLifecycleRule
that I added a long time ago.