Issue with chunk-signature when using proxy
dcarrion87 opened this issue · comments
This is an extension of #164 that I thought someone might have thoughts on.
When using aws-sigv4-proxy with this load testing tool we get:
warp: <ERROR> unexpected download size. want:10485760, got:10500246
warp: <ERROR> unexpected download size. want:10485760, got:10500246
warp: <ERROR> unexpected download size. want:10485760, got:10500246
I believe it's to do with this chunk signature ending up in the file on the way. E.g.:
10000;chunk-signature=7cc8934a07a23a0b069244e0f418202c055291dffeb9cf8bfa3b92df546b4aeb
Only seems to be an issue when using warp + aws-sigv4-proxy as when using aws s3 direct it's not an issue.
Any thoughts why it would be getting added with this combination?
The proxy isn't meant to proxy already signed requests AFAICT, so it doesn't parse already signed data correctly.
@klauspost thanks for the response. Are you saying the proxy service should be checking for a signing condition based on how warp is interacting with the s3 service? Is this related to streaming?
@dcarrion87 AWS 4 signatures look like chunked data, but contains extra signatures.
So if a proxy doesn't expect them and they parse the chunks, it may lead to data corruption.
OK thanks @klauspost.
I'll need to do some more digging why I'm only getting it with warp+proxy but not an issue with AWS SDK clients + proxy. E.g. using aws s3 CLI operations with the proxy endpoint doesn't result in those signature lines ending up in files.
How strange...