ContentLength is not set when I access a S3 compatible storage

[Manuel Guggi]

I try to access a S3 compatible storage via etcd-operator. I found the error occurs when the ContentLength will be checked. In my case this is always nil for the GetObject call.

When I use curl and print the Header in a file the Content-Length key is available.

I extracted the s3 code and the output of the test program looks like:

/tmp/s3-example k8s etcd.backup
2019/12/17 12:46:29 DEBUG: Request s3/GetObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /k8s/etcd.backup HTTP/1.1
Host: cloudstor.example.com
User-Agent: aws-sdk-go/1.13.8 (go1.13.1; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=ZGDFASFE3VW3ESZ7GXTN/20191217/default/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=ba5a3ed05015dac984528a0303e3b76bd9ad9058fef94efea868f38ca98804a
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca4418191b7852b855
X-Amz-Date: 20191217T114629Z
X-Amz-Security-Token: n/a
Accept-Encoding: gzip
 
 
-----------------------------------------------------
2019/12/17 12:46:29 DEBUG: Response s3/GetObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Accept-Ranges: bytes
Connection: KEEP-ALIVE
Content-Type: application/xml
Date: Tue, 17 Dec 2019 11:46:29 GMT
Etag: "1e2095efe15fef45aef9225e038a5b92"
Last-Modified: Tue, 17 Dec 2019 10:16:49 GMT
Server: StorageGRID/11.1.1.3
X-Amz-Request-Id: 1532667193
X-Amz-Version-Id: NTY0QUY1ODktMjBCNi0xMUVBLTlGQjctODJENDAwQkRERTE4
 
 
-----------------------------------------------------
resp ->  {
  AcceptRanges: "bytes",
  Body: buffer(0xc00023e120),
  ContentType: "application/xml",
  ETag: "\"1e2095efe15fef45aef9225e038a5b92\"",
  LastModified: 2019-12-17 10:16:49 +0000 UTC,
  Metadata: {
 
  },
  VersionId: "NTY0QUY1ODktMjBCNi0xMUVBLTlGQjctODJENDAwQkRERTE4"
}
resp.ContentLength -> <nil>
failed to compute s3 object size

When I try the same with curl, the Content-Length key is set as expected. Below is the header from my curl call:

HTTP/1.1 200 OK
Date: Tue, 17 Dec 2019 13:34:19 GMT
Connection: KEEP-ALIVE
Server: StorageGRID/11.1.1.3
x-amz-request-id: 1084366077
Content-Length: 7983136
ETag: "3bf60b995e992580a68ba46e2a1d38ce-2"
Content-Type: binary/octet-stream
Last-Modified: Tue, 17 Dec 2019 12:29:55 GMT
x-amz-version-id: RUUwRkEyRDQtMjBDOC0xMUVBLThCQjAtM0ZGNzAwQjg1RUJE
Accept-Ranges: bytes

The status of my backup is:

Status:
  Reason:             failed to save snapshot (failed to write snapshot (failed to compute s3 object size))
  Last Success Date:  <nil>
  Succeeded:          false

but the file was written.