gaul / s3proxy

S3Proxy does not currently support POST uploads, triggering many s3-tests failures:

ERROR: s3tests.functional.test_s3.test_post_object_anonymous_request
ERROR: s3tests.functional.test_s3.test_post_object_authenticated_request_bad_access_key
ERROR: s3tests.functional.test_s3.test_post_object_set_success_code
ERROR: s3tests.functional.test_s3.test_post_object_set_invalid_success_code
FAIL: s3tests.functional.test_s3.test_post_object_authenticated_request
FAIL: s3tests.functional.test_s3.test_post_object_upload_larger_than_chunk
FAIL: s3tests.functional.test_s3.test_post_object_set_key_from_filename
FAIL: s3tests.functional.test_s3.test_post_object_ignored_header
FAIL: s3tests.functional.test_s3.test_post_object_case_insensitive_condition_fields
FAIL: s3tests.functional.test_s3.test_post_object_escaped_field_values
FAIL: s3tests.functional.test_s3.test_post_object_success_redirect_action
FAIL: s3tests.functional.test_s3.test_post_object_invalid_date_format
FAIL: s3tests.functional.test_s3.test_post_object_no_key_specified
FAIL: s3tests.functional.test_s3.test_post_object_missing_signature
FAIL: s3tests.functional.test_s3.test_post_object_user_specified_header
FAIL: s3tests.functional.test_s3.test_post_object_condition_is_case_sensitive
FAIL: s3tests.functional.test_s3.test_post_object_expires_is_case_sensitive
FAIL: s3tests.functional.test_s3.test_post_object_missing_expires_condition
FAIL: s3tests.functional.test_s3.test_post_object_missing_conditions_list
FAIL: s3tests.functional.test_s3.test_post_object_upload_size_limit_exceeded
FAIL: s3tests.functional.test_s3.test_post_object_missing_content_length_argument
FAIL: s3tests.functional.test_s3.test_post_object_invalid_content_length_argument
FAIL: s3tests.functional.test_s3.test_post_object_upload_size_below_minimum

@auguster I implemented basic support for POST uploads -- can you test to see if it resolves your issue? Note that S3Proxy still lacks support for policies.

Thanks you for the news !
Unfortunatly I'll be away for the whole coming week, so I might not be able to test that before the week after...

Is it planned to be merged?

S3Proxy has partial POST upload support although there are a lot of extra bits. Do you need one feature in particular?

We currently migrating from S3 to Azure Blob Storage and to upload files from the client, our API creates a PostObject, which seems to work with the proxy (at least I get similar "form fields"). But when I submit the form I get a "Not Implemented" error.

Could you share some example source code? Also maybe dump logs via -DLOG_LEVEL=trace although it might not capture enough information to help.

How I use the php aws sdk:

$options = [
    ['acl' => 'public-read'],
    ['bucket' => $bucket],
    ['starts-with', '$key', $key],
    ['content-length-range', 0, 1024 * 1024 * 1024],
];

/** @var \Aws\S3\S3Client $s3 */
$s3 = app('s3');
$postObject = new PostObjectV4($s3, $bucket, $inputs, $options);
$attributes = $postObject->getFormAttributes();
$fields = $postObject->getFormInputs();

this is an example response:

{
    "attributes": {
        "action": "http://localhost:8080",
        "method": "POST",
        "enctype": "multipart/form-data"
    },
    "fields": {
        "acl": "public-read",
        "key": "filename.ext",
        "X-Amz-Credential": "*******",
        "X-Amz-Algorithm": "AWS4-HMAC-SHA256",
        "X-Amz-Date": "20180418T063726Z",
        "Policy": "**************",
        "X-Amz-Signature": "************"
    }
}

The above mentions field are used to submit the form with a file called filename.ext.

This is the exact error message:

<?xml version='1.0' encoding='UTF-8'?>
<Error>
    <Code>NotImplemented</Code>
    <Message>A header you provided implies functionality that is not implemented.</Message>
    <RequestId>4442587FB7D0A2F9</RequestId>
</Error>

Unfortunately I can't or I don't know how to set the DLOG_LEVEL. I am using docker and if I am adding it after ENTRYPOINT ./target/s3proxy --properties ./s3proxy.conf the proxy will not start because of the unknown option.

any updates? :/

The Docker image should take a LOG_LEVEL environment variable. I really need a self-contained test case to make progress on this; otherwise I would have to implement the entire POST handling functionality which I would like to avoid for now.

@ck86 I'm facing this issue at the moment, did you manage to find a work around?

@simonbowen unfortunately not. I switched to another proxy and used the signed URLs to upload a file.

@ck86 Can you tell me the name of the other proxy you used? I'm looking at using LocalStack instead at the moment.

minio

Thanks

As suggested in #73 (comment), if someone can provide a self-contained test I can add missing functionality but I don't want to commit to all possible features of POST uploads right now.

@gaul Thanks for the response. Also thanks for this great package. I'm not a Java developer and writing a test for this use case is going to be beyond me. I had dropped this into my development environment via docker and it worked great up until the HTTP POST upload.

@simonbowen If you can provide a self-contained test, either via unit test, curl, or whatever, I can likely add the required functionality easily. Unfortunately figuring out how to even run something like #73 (comment) is likely more work than the fix.

@gaul I'm not sure I can just give you an example curl request, since I have to make a request to get the fields (via a request) and then use those fields in a subsequent request. All I'd be doing is converting above into a bash script. As I said already, I am not capable of writing a unit test in Java (sorry).

Try turning logging up to trace level and see which parameter fails?

Here's a simple failure

files_1          | [s3proxy] D 02-09 03:03:57.200 S3Proxy-Jetty-41 o.gaul.s3proxy.S3ProxyHandler:299 |::] request: Request(POST //files/philomena/avatars/2022/2/9/e9158f14-8954-11ec-a0c5-0242ac130006.png?uploads=1)@5c2e0896
files_1          | [s3proxy] D 02-09 03:03:57.201 S3Proxy-Jetty-41 o.gaul.s3proxy.S3ProxyHandler:324 |::] header: Authorization: AWS4-HMAC-SHA256 Credential=local-identity/20220209/us-east-1/s3/aws4_request,SignedHeaders=content-length;host;x-amz-acl;x-amz-content-sha256;x-amz-date,Signature=d900da74357e968e29ea2cfd0f0afef09052bc8df0b4f45f9c011e3727612822
files_1          | [s3proxy] D 02-09 03:03:57.201 S3Proxy-Jetty-41 o.gaul.s3proxy.S3ProxyHandler:324 |::] header: x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
files_1          | [s3proxy] D 02-09 03:03:57.201 S3Proxy-Jetty-41 o.gaul.s3proxy.S3ProxyHandler:324 |::] header: x-amz-date: 20220209T030357Z
files_1          | [s3proxy] D 02-09 03:03:57.202 S3Proxy-Jetty-41 o.gaul.s3proxy.S3ProxyHandler:324 |::] header: x-amz-acl: public-read
files_1          | [s3proxy] D 02-09 03:03:57.202 S3Proxy-Jetty-41 o.gaul.s3proxy.S3ProxyHandler:324 |::] header: host: files
files_1          | [s3proxy] D 02-09 03:03:57.202 S3Proxy-Jetty-41 o.gaul.s3proxy.S3ProxyHandler:324 |::] header: User-Agent: hackney/1.18.0
files_1          | [s3proxy] D 02-09 03:03:57.202 S3Proxy-Jetty-41 o.gaul.s3proxy.S3ProxyHandler:324 |::] header: Content-Length: 0
files_1          | [s3proxy] D 02-09 03:03:57.202 S3Proxy-Jetty-41 o.gaul.s3proxy.S3ProxyHandler:324 |::] header: Content-Type: application/octet-stream
files_1          | [s3proxy] E 02-09 03:03:57.204 S3Proxy-Jetty-41 o.gaul.s3proxy.S3ProxyHandler:769 |::] Unknown method POST with URI /philomena/avatars/2022/2/9/e9158f14-8954-11ec-a0c5-0242ac130006.png
files_1          | [s3proxy] D 02-09 03:03:57.204 S3Proxy-Jetty-41 o.gaul.s3proxy.S3ProxyHandler:2923 |::] sendSimpleErrorResponse: 501 NotImplemented A header you provided implies functionality that is not implemented. {}

because the code tests if the parameter value is empty rather than if the parameter is present

s3proxy/src/main/java/org/gaul/s3proxy/S3ProxyHandler.java

Line 719 in 11868c2

} else if ("".equals(request.getParameter("uploads"))) {

@liamwhite Which S3 client do you use to access S3Proxy? The uploads=1 parameter does not exactly match the AWS documented CreateMultipartUpload RPC. I prefer to fix the client but will also accept an S3Proxy PR to be more liberal with all the parameter checks, e.g., getParameter() != null.

Note that this issue tracks browser-based POST uploads and your logs correspond to the unrelated CreateMultipartUpload RPC which confusingly has a POST HTTP verb and an uploads parameter. Let's discuss this is a separate issue or PR.

This is coming from the ExAws client.
https://github.com/ex-aws/ex_aws_s3/blob/b132398e6e4b7fb5fecda84d885955503d554f8a/lib/ex_aws/s3.ex#L1060-L1071
https://github.com/ex-aws/ex_aws/blob/08cbbd2aef4ebf52796e48761d1351b5c87c4c5e/lib/ex_aws/operation/s3.ex#L72-L76
I found this issue and didn't realize it was for browser POSTs.

Two behaviour differences I’ve noticed compared to the real S3 service when testing browser POSTs are:

${filename} in the key field is not substituted with the filename from the file field’s Content-Disposition header; and
The response should be a 201 Created with a Location header pointing to the URL of the created object. The path seems to be always URL encoded. e.g. https://example.s3.amazonaws.com/uploads%2Ftest.csv. There’s also a PostResponse XML object in the body but I’ve never used it.