Error when using Helm and S3

[tanora]

Describe the bug
ForcePathStyle is not included in the chart. I have tried setting ForcePathStyle to true and false.

    region: eu-west-1
    bucket: bucket-name
    useDefaultConfiguration: true
    ForcePathStyle: true

Error Message
Logs from pod

INFO[12:04PM]: incoming request	http-method=GET http-path=/readyz http-status=500
INFO[12:04PM]: incoming request	http-method=GET http-path=/healthz http-status=200
INFO[12:04PM]: incoming request	http-method=GET http-path=/readyz http-status=500

When describing pod:

  Normal   Created    4m2s                  kubelet, ip-10-10-29-157.eu-west-1.compute.internal  Created container
  Normal   Started    4m1s                  kubelet, ip-10-10-29-157.eu-west-1.compute.internal  Started container
  Warning  Unhealthy  39s (x21 over 3m58s)  kubelet, ip-10-10-29-157.eu-west-1.compute.internal  Readiness probe failed: HTTP probe failed with statuscode: 500

To Reproduce
Using Kube2iam and ingress is enabled.

Environment (please complete the following information):

• Storage (fs/mongodb/s3 etc.) : S3

[Teo Cheng Lim]

Same problem here.

# curl -v http://localhost:3000/readyz
*   Trying 127.0.0.1:3000...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 3000 (#0)
> GET /readyz HTTP/1.1
> Host: localhost:3000
> User-Agent: curl/7.67.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 500 Internal Server Error
< Content-Type: application/json
< Date: Tue, 02 Mar 2021 07:21:29 GMT
< Content-Length: 0
<

[Aaron Schlesinger]

@teochenglim @tanora do either of you see anything in logs that shows an error specific to S3 or similar?

[Teo Cheng Lim]

@arschles That's no any helpful error message from the kubernetes pod. All we see is bunch of 200 (/healthz) and 500 (/readyz) status code because of helm charts's readiness. A more verbose debug log option would be helpful.

i think it is from https://github.com/gomods/athens/blob/main/charts/athens-proxy/templates/deployment.yaml#L59-L70

INFO[12:04PM]: incoming request	http-method=GET http-path=/readyz http-status=500
INFO[12:04PM]: incoming request	http-method=GET http-path=/healthz http-status=200
INFO[12:04PM]: incoming request	http-method=GET http-path=/readyz http-status=500

The other possible question is when the endpoint /readyz is not ready, how do we debug it?

[Teo Cheng Lim]

@arschles

indeed the /readyz seems from this code and it only return the http status
https://github.com/gomods/athens/blob/main/cmd/proxy/actions/readiness.go#L12

If some log is written somewhere would be good.

Or maybe also return the "err" on what ready happen would be so helpful.
https://github.com/gomods/athens/blob/main/pkg/storage/s3/s3.go#L70

The problem can range from permission error, the connection established error would be prefect.