Invalid signature

Question

Invalid signature

justinmchase opened this issue 3 years ago · comments

Describe the bug

I'm trying to connect to minio running in local docker, but it seems to have a problem with the signature generated by this library.

The request signature we calculated does not match the signature you provided. Check your key and signing method

Your environment

SDK version number

@aws_sdk/client-s3@v3.22.0-1

Details of the deno version

deno 1.11.5 (release, x86_64-apple-darwin)
v8 9.1.269.35
typescript 4.3.2

Steps to reproduce

Install docker and enable kubernetes
Install minio into local kubernetes

# Randomly generated fake key for testing
ACCESS_KEY=AjAOk2gNRU
SECRET_KEY=Wk1HVyV8WP2Nh3O9QfLvTW9dOwR0ysqthZrP2Smf
helm repo add bitnami https://charts.bitnami.com/bitnami
helm upgrade \
  --install \
  --kube-context docker-desktop \
  --namespace docs \
  --set accessKey.password=$ACCESS_KEY \
  --set secretKey.password=$SECRET_KEY \
  minio bitnami/minio &

Run with deno

import { S3 } from 'https://deno.land/x/aws_sdk@v3.22.0-1/client-s3/mod.ts'
const s3 = new S3({
  endpoint: "http://localhost:9000",
  forcePathStyle: true,
  region: 'us-east-1',
  credentials: { 
    accessKeyId: 'AjAOk2gNRU',
    secretAccessKey: 'Wk1HVyV8WP2Nh3O9QfLvTW9dOwR0ysqthZrP2Smf',
  },
})
await s3.createBucket({
  Bucket: "uploads"
})

Observed behavior

The server produces an error about an invalid signature:

error: Uncaught (in promise) SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
  return Promise.reject(Object.assign(new Error(message), response));
                                      ^
    at deserializeAws_restXmlCreateBucketCommandError (https://deno.land/x/aws_sdk@v3.22.0-1/client-s3/protocols/Aws_restXml.ts:5216:39)
    at async https://deno.land/x/aws_sdk@v3.22.0-1/middleware-serde/deserializerMiddleware.ts:18:20
    at async https://deno.land/x/aws_sdk@v3.22.0-1/middleware-signing/middleware.ts:26:22
    at async StandardRetryStrategy.retry (https://deno.land/x/aws_sdk@v3.22.0-1/middleware-retry/StandardRetryStrategy.ts:83:38)
    at async https://deno.land/x/aws_sdk@v3.22.0-1/middleware-logger/loggerMiddleware.ts:22:22
    at async S3Service.init (file:///app/src/services/s3/S3Service.ts:32:5)
    at async initServices (file:///app/src/services/mod.ts:14:3)
    at async init (file:///app/src/mod.ts:6:20)
    at async file:///app/main.ts:4:20

Expected behavior

The call is expected to succeed.

Additional context

Justin Chase · Answer 1 · Wed Jul 28 2021 02:11:51 GMT+0800 (China Standard Time)

I just went through the process of switching my code to node and used the node library with the identical code and it works, so it is a valid configuration.

Christoph Gysin · Answer 2 · Wed Jul 28 2021 03:25:57 GMT+0800 (China Standard Time)

Can you provide the actual example code you tried? The above doesn't even compile.

Christoph Gysin · Answer 3 · Wed Jul 28 2021 03:27:02 GMT+0800 (China Standard Time)

Also, does it work against AWS instead of minio?

Justin Chase · Answer 4 · Wed Jul 28 2021 04:25:41 GMT+0800 (China Standard Time)

Sorry! Copy paste error, fixed in original post.

And no if I remove endpoint and forcePathStyle from the config then use credentials for my cloud account it does not work either, same error.

Justin Chase · Answer 5 · Wed Jul 28 2021 04:27:33 GMT+0800 (China Standard Time)

Also this is a seperate issue but s3.headBucket has an error also:

error: Uncaught (in promise) TypeError: Cannot read property 'getReader' of null
  const reader = stream.getReader();
                        ^
    at collectStream (https://deno.land/x/aws_sdk@v3.22.0-1/fetch-http-handler/stream-collector.ts:21:25)
    at Object.streamCollector (https://deno.land/x/aws_sdk@v3.22.0-1/fetch-http-handler/stream-collector.ts:10:10)
    at collectBody (https://deno.land/x/aws_sdk@v3.22.0-1/client-s3/protocols/Aws_restXml.ts:14705:18)
    at collectBodyString (https://deno.land/x/aws_sdk@v3.22.0-1/client-s3/protocols/Aws_restXml.ts:14710:3)
    at parseBody (https://deno.land/x/aws_sdk@v3.22.0-1/client-s3/protocols/Aws_restXml.ts:14720:3)
    at deserializeAws_restXmlHeadBucketCommandError (https://deno.land/x/aws_sdk@v3.22.0-1/client-s3/protocols/Aws_restXml.ts:7623:17)
    at deserializeAws_restXmlHeadBucketCommand (https://deno.land/x/aws_sdk@v3.22.0-1/client-s3/protocols/Aws_restXml.ts:7608:12)
    at deserialize (https://deno.land/x/aws_sdk@v3.22.0-1/client-s3/commands/HeadBucketCommand.ts:103:12)
    at https://deno.land/x/aws_sdk@v3.22.0-1/middleware-serde/deserializerMiddleware.ts:18:26
    at async https://deno.land/x/aws_sdk@v3.22.0-1/middleware-signing/middleware.ts:26:22

Justin Chase · Answer 6 · Wed Jul 28 2021 04:36:01 GMT+0800 (China Standard Time)

Now using node js, exact same config, exact same API's and it works.

// index.ts
const { S3 } = require("@aws-sdk/client-s3")
const config = {
  endpoint: "http://localhost:9000",
  forcePathStyle: true,
  region: 'us-east-1',
  credentials: {
    accessKeyId: 'AjAOk2gNRU',
    secretAccessKey: 'Wk1HVyV8WP2Nh3O9QfLvTW9dOwR0ysqthZrP2Smf',
  }
}
const s3 = new S3(config)
const args = {
  Bucket: "uploads"
}
async function main() {
  const bucket = await s3.headBucket(args)
  if (!bucket) {
    await s3.createBucket(args)
  }

  const buckets = await s3.listBuckets({})
  console.log(buckets)
}

main().then().catch(err => console.log(err))

$ npm i @aws-sdk/client-s3
$ node -r node-tsc index.ts
{
  '$metadata': {
    httpStatusCode: 200,
    requestId: undefined,
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Buckets: [ { Name: 'uploads', CreationDate: 2021-07-27T18:08:58.437Z } ],
  Owner: {
    DisplayName: 'minio',
    ID: '02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4'
  }
}

Justin Chase · Answer 7 · Wed Jul 28 2021 04:43:45 GMT+0800 (China Standard Time)

Maybe this is helpful, but if you create a pre-signed url in both apps they create different signatures also:

// main.ts
import { getSignedUrl } from "https://deno.land/x/aws_sdk@v3.22.0-1/s3-request-presigner/mod.ts";
import { S3Client } from "https://deno.land/x/aws_sdk@v3.22.0-1/client-s3/S3Client.ts";
import { PutObjectCommand } from "https://deno.land/x/aws_sdk@v3.22.0-1/client-s3/commands/PutObjectCommand.ts";

const client = new S3Client({
  region: "us-east-1",
  endpoint: "http://localhost:9000",
  forcePathStyle: true,
  credentials: {
    accessKeyId: 'AjAOk2gNRU',
    secretAccessKey: 'Wk1HVyV8WP2Nh3O9QfLvTW9dOwR0ysqthZrP2Smf',
  },
});
const command = new PutObjectCommand({
  Bucket: 'uploads',
  Key: 'test123',
});
const url = await getSignedUrl(client, command, { expiresIn: 3600 });
console.log(url)

// index.ts
const { S3Client, PutObjectCommand } = require("@aws-sdk/client-s3")
const { getSignedUrl } = require("@aws-sdk/s3-request-presigner")

const client = new S3Client({
  region: "us-east-1",
  endpoint: "http://localhost:9000",
  forcePathStyle: true,
  credentials: {
    accessKeyId: 'AjAOk2gNRU',
    secretAccessKey: 'Wk1HVyV8WP2Nh3O9QfLvTW9dOwR0ysqthZrP2Smf',
  },
});
const command = new PutObjectCommand({
  Bucket: 'uploads',
  Key: 'test123',
});
async function main() {
  const url = await getSignedUrl(client, command, { expiresIn: 3600 });
  console.log(url)
}

main().then().catch(err => console.log(err))

deno run --unstable -A main.ts && node -r node-tsc index.ts
http://localhost:9000/uploads/test123?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AjAOk2gNRU%2F20210727%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210727T204007Z&X-Amz-Expires=3600&X-Amz-Signature=4c1f2d8d5128bdff37b09d4f0a539e887640b60b1fd7e1065b8a49eecc49a005&X-Amz-SignedHeaders=host&x-id=PutObject
http://localhost:9000/uploads/test123?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AjAOk2gNRU%2F20210727%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210727T204008Z&X-Amz-Expires=3600&X-Amz-Signature=f79dda2a587e821ad95affbd8de0dadee39098bb4804443ee83908ff535971e0&X-Amz-SignedHeaders=host&x-id=PutObject

Christoph Gysin · Answer 8 · Fri Jul 30 2021 15:24:29 GMT+0800 (China Standard Time)

I'm unable to reproduce this. The reason you got different signatures above is because the timestamp is part of the signature, and the two runs were not started within the same second:

...&X-Amz-Date=20210727T204007Z&...
...&X-Amz-Date=20210727T204008Z&...

Justin Chase · Answer 9 · Sat Jul 31 2021 07:20:23 GMT+0800 (China Standard Time)

You're right about the signatures. I'm not sure why its not working for me I think it has to do with the host differing between my localhost curl and the docker to docker internal hostname the api is using.

Braden MacDonald · Answer 10 · Fri Oct 22 2021 08:39:45 GMT+0800 (China Standard Time)

I have encountered this issue as well, and I think it is a real bug. The code is working with a remote S3 endpoint (HTTPS), but my local MinIO endpoint (http://localhost:9000) is giving this signature error.

Here is an easy repro case.

Run MinIO with

docker run --rm -e MINIO_ACCESS_KEY=AKIA_DEV -e MINIO_SECRET_KEY=secretkey -e MINIO_REGION_NAME=dev-region -p 9000:9000 --entrypoint /bin/sh minio/minio:RELEASE.2020-05-08T02-40-49Z -c 'mkdir -p /data/dev-bucket && /usr/bin/minio server /data'

Then put this into deno-s3-test.ts

import {
    S3Client,
    DeleteObjectsCommand,
    ListObjectsV2Command,
    PutObjectCommand,
} from "https://deno.land/x/aws_sdk@v3.23.0-1/client-s3/mod.ts";
import { S3Bucket } from "https://deno.land/x/s3@0.4.1/mod.ts";

const config = {
    objStoreEndpointURL: "http://localhost:9000/",
    objStoreRegion: "dev-region",
    objStoreBucketName: "dev-bucket",
    objStoreAccessKey: "AKIA_DEV",
    objStoreSecretKey: "secretkey",
    objStorePublicUrlPrefix: "http://localhost:9000/dev-bucket",
};

const simpleClient = new S3Bucket({
    bucket: config.objStoreBucketName,
    endpointURL: config.objStoreEndpointURL,
    accessKeyID: config.objStoreAccessKey,
    secretKey: config.objStoreSecretKey,
    region: config.objStoreRegion,
});

const awsClient = new S3Client({
    endpoint: config.objStoreEndpointURL,
    region: config.objStoreRegion,
    credentials: {
        accessKeyId: config.objStoreAccessKey,
        secretAccessKey: config.objStoreSecretKey,
    },
    bucketEndpoint: false,
    forcePathStyle: true,  // Fix: "TypeError: error sending request for url (http://dev-bucket.localhost:9000/"
});

await simpleClient.listObjects();
console.log("It worked.");
await awsClient.send(new ListObjectsV2Command({Bucket: config.objStoreBucketName}));
console.log("It worked too.");

And run it with

deno run --allow-net --allow-env --allow-read --unstable deno-s3-test.ts

For me, the other client works, but this client gives SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.

Note that I have to run the command twice, due to an unrelated type error that shows up on the first attempt.

Christoph Gysin · Answer 11 · Fri Oct 22 2021 22:09:38 GMT+0800 (China Standard Time)

@bradenmacdonald Does it work with JavaScript and the aws-sdk-js-v3? Can you adjust the examples above to your use case, and show that the signature differs from aws-sdk-v3?

Braden MacDonald · Answer 12 · Sun Oct 24 2021 06:37:49 GMT+0800 (China Standard Time)

@christophgysin Yes, it does work with the official aws-sdk-js-v3, I just tried using Node.

Actually I believe I have found the bug: when a port number is specified, e.g. using http://localhost:9000 as the endpoint, then this deno version doesn't work - MinIO rejects the signature. However, the upstream AWS API still works when called identically, and if I instead run MinIO on port 80 and remove the port number from the endpoint URL, this deno version works too. So I believe that the Deno version is somehow incorrectly including or not including the port number in the Host when computing the hash.

Here's the identical code that I used with node to test the JS/upstream version:

const { S3Client, ListObjectsV2Command } = require("@aws-sdk/client-s3");

const config = {
    objStoreEndpointURL: "http://localhost:9000/",
    objStoreRegion: "dev-region",
    objStoreBucketName: "dev-bucket",
    objStoreAccessKey: "AKIA_DEV",
    objStoreSecretKey: "secretkey",
    objStorePublicUrlPrefix: "http://localhost:9000/dev-bucket",
};

const awsClient = new S3Client({
    endpoint: config.objStoreEndpointURL,
    region: config.objStoreRegion,
    credentials: {
        accessKeyId: config.objStoreAccessKey,
        secretAccessKey: config.objStoreSecretKey,
    },
    bucketEndpoint: false,
    forcePathStyle: true,  // Fix: "TypeError: error sending request for url (http://dev-bucket.localhost:9000/"
});

awsClient.send(new ListObjectsV2Command({Bucket: config.objStoreBucketName})).then(r => {
    console.log("It worked too.");
});

Christoph Gysin · Answer 13 · Sun Oct 24 2021 21:06:00 GMT+0800 (China Standard Time)

So I created two minimized examples:

node

const { S3Client, ListObjectsV2Command } = require("@aws-sdk/client-s3")
const { getSignedUrl } = require("@aws-sdk/s3-request-presigner")

const client = new S3Client({
  endpoint: "http://localhost:9000/",
  region: "dev-region",
  credentials: {
    accessKeyId: 'AKIA_DEV',
    secretAccessKey: 'secretkey',
  },
  bucketEndpoint: false,
  forcePathStyle: true,
})

const command = new ListObjectsV2Command({
  Bucket: 'dev-bucket',
})

async function main() {
  const url = await getSignedUrl(client, command, { expiresIn: 3600 });
  console.log(url)
}

main().then().catch(err => console.log(err))

deno

import { getSignedUrl } from "https://deno.land/x/aws_sdk@v3.22.0-1/s3-request-presigner/mod.ts";
import { S3Client } from "https://deno.land/x/aws_sdk@v3.22.0-1/client-s3/S3Client.ts";
import { ListObjectsV2Command } from "https://deno.land/x/aws_sdk@v3.22.0-1/client-s3/commands/ListObjectsV2Command.ts";

const client = new S3Client({
  endpoint: "http://localhost:9000/",
  region: "dev-region",
  credentials: {
    accessKeyId: 'AKIA_DEV',
    secretAccessKey: 'secretkey',
  },
  bucketEndpoint: false,
  forcePathStyle: true,
})

const command = new ListObjectsV2Command({
  Bucket: 'dev-bucket',
})

const url = await getSignedUrl(client, command, { expiresIn: 3600 });
console.log(url)

When both are executed within the same second, they produce an identical signature:

$ deno run --unstable -A deno.ts & node -r node-tsc node.ts
http://localhost:9000/dev-bucket?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA_DEV%2F20211024%2Fdev-region%2Fs3%2Faws4_request&X-Amz-Date=20211024T130115Z&X-Amz-Expires=3600&X-Amz-Signature=a1608f441494387157ef1eb7a44ac4c3e311b1f66163289b918ff35e234ad1f9&X-Amz-SignedHeaders=host&list-type=2
http://localhost:9000/dev-bucket?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA_DEV%2F20211024%2Fdev-region%2Fs3%2Faws4_request&X-Amz-Date=20211024T130115Z&X-Amz-Expires=3600&X-Amz-Signature=a1608f441494387157ef1eb7a44ac4c3e311b1f66163289b918ff35e234ad1f9&X-Amz-SignedHeaders=host&list-type=2

So it seems that the problem lies somewhere else.

Please let me know if you find a way to show how the deno implementation differs from the node implementation of aws-sdk-js-v3.

Braden MacDonald · Answer 14 · Mon Oct 25 2021 03:13:19 GMT+0800 (China Standard Time)

@christophgysin Thanks for your close attention here. I believe you are right, that the problem lies elsewhere: not in the signature generation (which I haven't tested myself, but seems identical), but in the actual Host header sent over the wire to MinIO.

Here you can see the actual HTTP request and response logged by MinIO for the ListObjectsV2 request from my examples. The first request is from deno, and it fails. The second is from Node, and it succeeds. Both are using aws-sdk-js-v3, and both are configured identically. Although I cannot verify that the signatures are computed in an identical manner, I can see that the clients are behaving differently - the Deno implementation sends localhost:9000 as its Host header, while the Node.js implementation sends localhost only. So if they are computing their signatures in the same way, as you say, then it's clear that one of the the requests will fail because the actual Host header that gets sent is different, while the signature calculation is the same.

See the host header in green in this screenshot:

BTW the command to get this debug output is:

mc admin alias set localdebug http://localhost:9000 AKIA_DEV secretkey
mc admin trace --verbose --all localdebug

Christoph Gysin · Answer 15 · Mon Oct 25 2021 04:20:05 GMT+0800 (China Standard Time)

I managed to reproduce your issue, and I think the problem is in the http-handler. The node implementation uses node-http-handler, whereas deno uses the fetch-http-handler.

As far as I can see the input data and signature are correctly going into both handlers, but only the node-http-handler makes a successful request. I'll investigate.

Christoph Gysin · Answer 16 · Mon Oct 25 2021 04:40:31 GMT+0800 (China Standard Time)

Minimal code to reproduce:

const response = await fetch('http://localhost:8000/get')
const data = await response.json()
console.log(data.headers.Host)

$ docker run -p 8000:80 kennethreitz/httpbin
$ deno run --allow-net fetch.ts 
localhost:8000

Christoph Gysin · Answer 17 · Mon Oct 25 2021 04:47:33 GMT+0800 (China Standard Time)

AFAICT the fetch implementation is actually doing the right thing by including the port in the Host header. But since the signature is calculated with the hostname only, it doesn't match. I don't think we can control the header in fetch(), so the only solution is to modify signature calculation specifically for the deno port.

I'm curious though why the node-http-handler doesn't include the port in the host header?

Braden MacDonald · Answer 18 · Mon Oct 25 2021 05:23:24 GMT+0800 (China Standard Time)

I agree with your assessment. And I can confirm at least with MinIO that it works fine that way (port in the host header) as long as the signature has been calculated accordingly.

Christoph Gysin · Answer 19 · Mon Oct 25 2021 15:37:57 GMT+0800 (China Standard Time)

I can reproduce this with aws-sdk-js-v3 in the browser, where it also uses the fetch-http-handler. So this is really a bug that should be fixed upstream.

Braden MacDonald · Answer 20 · Tue Oct 26 2021 03:08:52 GMT+0800 (China Standard Time)

Ah, yes, it seems that there are some upstream reports already: aws#1941 aws#1930

In the meantime, the workaround reported in that first issue will resolve this: instead of specifying the endpoint as a string, specify it like:

endpoint: {
  protocol: 'http',
  hostname: '127.0.0.1:9000',
  path: '/',
},

Christoph Gysin · Answer 21 · Mon Dec 27 2021 21:44:13 GMT+0800 (China Standard Time)

Closing, as the bug is tracked upstream, and a workaround exists.