Giters

google / go-cloud

The Go Cloud Development Kit (Go CDK): A library and tools for open cloud development in Go.

Home Page:https://gocloud.dev/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

blob/s3: Add content length parameter in signing URL

mrth1995 opened this issue · comments

commented

Hi team, I'd like to give idea to introduce new field ContentLength *int64 in SignedURLOptions. I know that the content length is automatically calculated when sending the request to signed URL. The idea adding this field in order to add validation in server side that control how big the file uploaded using signed URL.

Sample Use Case
I'm using go-cloud especially blob module to generate signed URL upload and download file to AWS S3. But I'm struggle because I cannot control size of file uploaded to S3 bucket (since client upload directly to S3 via signed URL. I've explore aws-go-sdk related to S3 and realize that I can set content-length in PutObjectInput. In my exploration, if I set content length in PutObjectInput the content length is included in signing process. If client upload a file with different content length, it'll be rejected. So this can be used as pre-upload validation for client that using upload URL.

I already explore the code base but looks like this can be supported for s3blob and fileblob.

I'll be happy if i could introduce this small feature to go-cloud.

Please let me know if you have suggestion.
Thanks!

commented

I explored this a bit:

  • It looks like AWS allows you to set ContentLength in PutObjectInput as you say. I haven't tested it. Does it require an exact match, or is a maximum length?
  • GCS allows a range, via storage.GenerateSignedPostPolicyV4: https://stackoverflow.com/q/63230403.
  • I don't see any way to do this for Azure.
  • I think fileblob could support an exact match.

This will be hard for you to contribute, because in order to test it we'd want to add a driver test, which will update all of our golden files and requires access to our test projects, but if we can find reasonable semantics that are supported across the providers I can add it.

commented

Hi vangent thanks for the response!

As per my exploration on AWS, its exact content length.
So I guess only Azure that doesn't support this? Are you okay if we add this feature to go-cloud?
It'll be helpful for given usecase I think.

Thanks!

commented

I looked into this some more. GCS doesn't support this either, at least not in the same way; it has https://pkg.go.dev/cloud.google.com/go/storage#GenerateSignedPostPolicyV4, but that's for POST, not PUT.

Given the lack of consistency across providers, I don't think it makes sense to add to Go CDK.

You can use As (https://gocloud.dev/concepts/as/) and SignedURLOptions.BeforeSign (https://pkg.go.dev/gocloud.dev/blob#SignedURLOptions) to set the PutObjectInput.ContentLength for S3. Here's an example using As for writes that should be similar: https://pkg.go.dev/gocloud.dev/blob#example-WriterOptions.