Token expiry exception with ResultStreamer with IAM roles
opened this issue · comments
Getting the following token expiry exception when streaming results of a large query from S3 using ResultStreamer:
com.amazonaws.services.s3.model.AmazonS3Exception: The provided token has expired. (Service: Amazon S3; Status Code: 400; Error Code: ExpiredToken; Request ID: <redacted>), S3 Extended Request ID: <redacted>
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:805)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:428)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:234)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3569)
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1130)
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:998)
at com.qubole.qds.sdk.java.client.ResultStreamer$1.getObject(ResultStreamer.java:142)
at com.qubole.qds.sdk.java.client.ResultStreamer$PathReader.loadNextReader(ResultStreamer.java:224)
at com.qubole.qds.sdk.java.client.ResultStreamer$PathReader.read(ResultStreamer.java:170)
at com.qubole.qds.sdk.java.client.ResultStreamer$PathReader.read(ResultStreamer.java:185)
at com.qubole.qds.sdk.java.client.ResultStreamer$PathReader.read(ResultStreamer.java:178)
I think the ResultStreamer should catch this exception and re-initialise the connection with S3 to continue streaming the rest of the files.
Thanks for reporting. We will take a look.
@abhisheksr8 Sending it your way. Please look at this.
@nimbudew if possible, can you please try with this branch once and let us know if you see the issue again.
thanks
@abhisheksr8 hi, I'm unable to reproduce the issue even with the 0.8.0 release, probably because of changes in the validity of the IAM role token.
@nimbudew Hi, was earlier stack trace from 0.8.0 itself ?
@amoghmargoor Yes, the posted stacktrace is from 0.8.0 release. There have been some changes to the environment around IAM roles, and the issue is non-reproducible since.
@amoghmargoor @abhisheksr8 The issue resurfaced with 0.8.0, will deploy a new JAR with a pull from the abhisheksr8:SDK-154
and see if the issue is fixed.
@nimbudew We could test this on our end and have committed the fix to master which you can check on your end too. Thanks for raising it with us. We are closing this issue for now, but feel free to reopen it if you find any more issues.
@amoghmargoor @abhisheksr8 I no longer see the issue with the added fix. Thanks.