endOfInput error when calling ListObjects against localstack/moto mock of S3 service
rcook opened this issue · comments
I don't know if this is a bug in amazonka or a bug in the emulation of S3 by localstack/moto, so I apologize in advance if this bug is going to the wrong place.
Anyway, I've been using localstack for local development of my AWS client apps. This platform's S3 implementation is, I believe, built on moto. It's quite a common platform for local/dev testing of AWS apps and I've run into this problem trying to use the ListObjects
API against it.
Repro program
See https://gist.github.com/rcook/21ec658bce70cc3841de2e591de5b79f#file-main-hs. This full project can be cloned and built locally as follows:
git clone https://gist.github.com/553ae2f596bbad3852f5f6d2edd3a257.git amazonka-localstack-repro
cd amazonka-localstack-repro
stack build
stack exec s3-app
Repro steps
- Repro steps work on Ubuntu and Linux Mint
- Follow instructions at https://github.com/localstack/localstack to install localstack
- Start localstack with
localstack
- Once localstack is running, running repro program.
- Run repro program with
stack exec s3-app
.
Here's the output:
CreateBucket
[Client Request] {
host = localhost:4572
secure = False
method = PUT
target = Nothing
timeout = ResponseTimeoutMicro 70000000
redirects = 0
path = /mybucket
query =
headers = host: localhost; x-amz-date: 20171218T183702Z; x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855; authorization: AWS4-HMAC-SHA256 Credential=AKIAJXT7OAEJFXCG7QOA/20171218/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=983e522d9b1abc00bed3aa8f2c03714ac69fcfde58be022b2d8ea7eec4dd914a
body =
}
[Client Response] {
status = 200 OK
headers = server: BaseHTTP/0.3 Python/2.7.12; date: Mon, 18 Dec 2017 18:37:02 GMT; content-type: text/html; charset=utf-8; content-length: 162; server: Werkzeug/0.13 Python/2.7.12; date: Mon, 18 Dec 2017 18:37:02 GMT; access-control-allow-origin: *; access-control-allow-methods: HEAD,GET,PUT,POST,DELETE,OPTIONS,PATCH; access-control-allow-headers: authorization,content-type,content-md5,x-amz-content-sha256,x-amz-date,x-amz-security-token,x-amz-user-agent
}
[Client Request] {
host = localhost:4572
secure = False
method = HEAD
target = Nothing
timeout = ResponseTimeoutMicro 70000000
redirects = 0
path = /mybucket
query =
headers = host: localhost; x-amz-date: 20171218T183702Z; x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855; authorization: AWS4-HMAC-SHA256 Credential=AKIAJXT7OAEJFXCG7QOA/20171218/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=2abc727f586521b7ed33c67b5e982b688a80692a6899eb242d2a76200a394255
body =
}
[Client Response] {
status = 200 OK
headers = server: BaseHTTP/0.3 Python/2.7.12; date: Mon, 18 Dec 2017 18:37:02 GMT; content-type: text/html; charset=utf-8; content-length: 0; server: Werkzeug/0.13 Python/2.7.12; date: Mon, 18 Dec 2017 18:37:02 GMT; access-control-allow-origin: *; access-control-allow-methods: HEAD,GET,PUT,POST,DELETE,OPTIONS,PATCH; access-control-allow-headers: authorization,content-type,content-md5,x-amz-content-sha256,x-amz-date,x-amz-security-token,x-amz-user-agent
}
[Await BucketExists] Success after 1 attempts.
ListObjects
[Client Request] {
host = localhost:4572
secure = False
method = GET
target = Nothing
timeout = ResponseTimeoutMicro 70000000
redirects = 0
path = /mybucket
query = ?list-type%3D2=
headers = host: localhost; x-amz-date: 20171218T183702Z; x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855; authorization: AWS4-HMAC-SHA256 Credential=AKIAJXT7OAEJFXCG7QOA/20171218/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=11ffdb07eb5ea708ec382159188fcdbb693fa881986ccd15e8c4ad25b5e2215e
body =
}
[Client Response] {
status = 200 OK
headers = server: BaseHTTP/0.3 Python/2.7.12; date: Mon, 18 Dec 2017 18:37:02 GMT; content-type: text/html; charset=utf-8; content-length: 248; server: Werkzeug/0.13 Python/2.7.12; date: Mon, 18 Dec 2017 18:37:02 GMT; access-control-allow-origin: *; access-control-allow-methods: HEAD,GET,PUT,POST,DELETE,OPTIONS,PATCH; access-control-allow-headers: authorization,content-type,content-md5,x-amz-content-sha256,x-amz-date,x-amz-security-token,x-amz-user-agent
}
[Raw Response Body] {
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>mybucket</Name><Prefix>None</Prefix><MaxKeys>1000</MaxKeys><Delimiter>None</Delimiter><IsTruncated>false</IsTruncated></ListBucketResult>
}
[SerializeError] {
service = S3
status = 200 OK
message = endOfInput
body = Just <?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>mybucket</Name><Prefix>None</Prefix><MaxKeys>1000</MaxKeys><Delimiter>None</Delimiter><IsTruncated>false</IsTruncated></ListBucketResult>
}
s3-app: SerializeError (SerializeError' {_serializeAbbrev = Abbrev "S3", _serializeStatus = Status {statusCode = 200, statusMessage = "OK"}, _serializeBody = Just "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<ListBucketResult xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"><Name>mybucket</Name><Prefix>None</Prefix><MaxKeys>1000</MaxKeys><Delimiter>None</Delimiter><IsTruncated>false</IsTruncated></ListBucketResult>", _serializeMessage = "endOfInput"})
You'll see that this fails with an endOfInput
error. This example works just fine against a real AWS S3 instance.
Richard,
I'm also getting into trying to test amazonka based code with localstack.
I have not really tried it yet, but putting myself in your shoes, I would likely note the raw queries that amazonka makes and come up with a minimal example in python (or somesuch) that I would post in the localstack's git issues.
Sorry to hijack a bit, but I was wondering whether you could help me set up the amazonka to call localstack instead of the real AWS, as it does not seem to be trivial. Can you post and link me to a simple example of that here on github?
EDIT: read too fast, just saw the link to gist, thanks
I investigated this thoroughly. The problem is that the ListObjectsResponse contains Delimiter which is Char type and so it's expecting either the field to be absent or to contain precisely one char. I haven't dived in enough to find precisely where a longer input (or one in entity form such as �); is rejected, but that's the message. It's saying "I expected end of input, got something else" basically.
To make matters worse, localstack replies contain None when you don't specify delimiter, but s3 replies contain entities if the char is unprintable (I tried '\0') (this is a correct thing to do). That's the cause anyway.
This sounds to me like both the following are true:
- Localstack is wrong.
- We should be parsing entities into a single character
This is low-enough priority that it can wait until 2.0 and possibly after.