Giters

cortexproject / cortex

A horizontally scalable, highly available, multi-tenant, long term Prometheus.

Home Page:https://cortexmetrics.io/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Cloudflare R2 Header 'x-amz-checksum-algorithm' with value 'CRC32C' not implemented

mpavlovicbb opened this issue · comments

commented

Describe the bug
I am running my Cortex deployment in combination with Cloudflare R2 bucket.
In Ingesters we have following logs:

ts=2024-03-25T11:25:24.379913283Z caller=shipper.go:325 level=error org_id=rxgqq9xn96 msg="shipping failed" block=01HST6JB4PKRJ2NEGRPQN3JSS9 err="upload chunks: upload file /data/rxgqq9xn96/thanos/upload/01HST6JB4PKRJ2NEGRPQN3JSS9/chunks/000001 as 01HST6JB4PKRJ2NEGRPQN3JSS9/chunks/000001: upload s3 object: Header 'x-amz-checksum-algorithm' with value 'CRC32C' not implemented" |  
ts=2024-03-25T11:25:24.377531977Z caller=bucket_client.go:182 level=warn msg="skip upload retry as reader is not seekable" file=c95nsmcjh9/01HSTK3QGYGVBECS5GP0KZBAP3/chunks/000001 err="upload s3 object: Header 'x-amz-checksum-algorithm' with value 'CRC32C' not implemented" |  
ts=2024-03-25T11:25:24.336389491Z caller=bucket_client.go:182 level=warn msg="skip upload retry as reader is not seekable" file=j67mpw76kd/01HST6JGATCSY9CXR0JHYSMHF9/chunks/000001 err="upload s3 object: Header 'x-amz-checksum-algorithm' with value 'CRC32C' not implemented" |  
ts=2024-03-25T11:25:24.315175576Z caller=bucket_client.go:182 level=warn msg="skip upload retry as reader is not seekable" file=d66xdbw5mf/01HSTK3T0N2PERDWMVJNFKH18J/chunks/000001 err="upload s3 object: Header 'x-amz-checksum-algorithm' with value 'CRC32C' not implemented" |  
ts=2024-03-25T11:25:24.313750159Z caller=bucket_client.go:182 level=warn msg="skip upload retry as reader is not seekable" file=w6a16f481/01HSSZPD1TQMHR44YXBHBHVFW6/chunks/000001 err="upload s3 object: Header 'x-amz-checksum-algorithm' with value 'CRC32C' not implemented" |  
ts=2024-03-25T11:25:24.302724654Z caller=shipper.go:325 level=error org_id=c95nsmcjh9 msg="shipping failed" block=01HSTDDV6KCMCQTK7PRZB8RJZ0 err="upload chunks: upload file /data/c95nsmcjh9/thanos/upload/01HSTDDV6KCMCQTK7PRZB8RJZ0/chunks/000001 as 01HSTDDV6KCMCQTK7PRZB8RJZ0/chunks/000001: upload s3 object: Header 'x-amz-checksum-algorithm' with value 'CRC32C' not implemented" 
ts=2024-03-25T11:25:24.262010696Z caller=bucket_client.go:182 level=warn msg="skip upload retry as reader is not seekable" file=xnpxx8mnsz/01HSTK3STNQKV3182KGPE8X4HQ/chunks/000001 err="upload s3 object: Header 'x-amz-checksum-algorithm' with value 'CRC32C' not implemented"

To Reproduce
Steps to reproduce the behavior:

  1. Distributed Cortex installation in Kubernetes with 5+ tenants
  2. Cloudflare R2 bucket to send metrics to

Expected behavior
Without errors in logs.

Environment:

  • Infrastructure: Kubernetes
  • Deployment tool: Helm
  • Cortex version: 1.16
commented

to implement this, we just need to expose

cortex/vendor/github.com/thanos-io/objstore/providers/s3/s3.go

Line 116 in 99d8c3e

SendContentMd5: true, // Default to using MD5.
to the configuration, so users can disable this