Token expiry exception with ResultStreamer with IAM roles

Question

Token expiry exception with ResultStreamer with IAM roles

opened this issue 8 years ago · comments

Getting the following token expiry exception when streaming results of a large query from S3 using ResultStreamer:

com.amazonaws.services.s3.model.AmazonS3Exception: The provided token has expired. (Service: Amazon S3; Status Code: 400; Error Code: ExpiredToken; Request ID: <redacted>), S3 Extended Request ID: <redacted>
        at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:805)
        at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:428)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:234)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3569)
        at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1130)
        at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:998)
        at com.qubole.qds.sdk.java.client.ResultStreamer$1.getObject(ResultStreamer.java:142)
        at com.qubole.qds.sdk.java.client.ResultStreamer$PathReader.loadNextReader(ResultStreamer.java:224)
        at com.qubole.qds.sdk.java.client.ResultStreamer$PathReader.read(ResultStreamer.java:170)
        at com.qubole.qds.sdk.java.client.ResultStreamer$PathReader.read(ResultStreamer.java:185)
        at com.qubole.qds.sdk.java.client.ResultStreamer$PathReader.read(ResultStreamer.java:178)

I think the ResultStreamer should catch this exception and re-initialise the connection with S3 to continue streaming the rest of the files.

Amogh Margoor · Answer 1 · Sun Sep 25 2016 09:34:05 GMT+0800 (China Standard Time)

Thanks for reporting. We will take a look.

@abhisheksr8 Sending it your way. Please look at this.

Abhishek Srivastava · Answer 2 · Tue Sep 27 2016 13:58:35 GMT+0800 (China Standard Time)

@nimbudew if possible, can you please try with this branch once and let us know if you see the issue again.
thanks

Deleted user · Answer 3 · Wed Sep 28 2016 15:21:45 GMT+0800 (China Standard Time)

@abhisheksr8 hi, I'm unable to reproduce the issue even with the 0.8.0 release, probably because of changes in the validity of the IAM role token.

Amogh Margoor · Answer 4 · Wed Sep 28 2016 15:27:08 GMT+0800 (China Standard Time)

@nimbudew Hi, was earlier stack trace from 0.8.0 itself ?

Deleted user · Answer 5 · Wed Sep 28 2016 22:14:53 GMT+0800 (China Standard Time)

@amoghmargoor Yes, the posted stacktrace is from 0.8.0 release. There have been some changes to the environment around IAM roles, and the issue is non-reproducible since.

Deleted user · Answer 6 · Sun Oct 02 2016 03:06:04 GMT+0800 (China Standard Time)

@amoghmargoor @abhisheksr8 The issue resurfaced with 0.8.0, will deploy a new JAR with a pull from the abhisheksr8:SDK-154 and see if the issue is fixed.

Amogh Margoor · Answer 7 · Tue Oct 04 2016 13:18:06 GMT+0800 (China Standard Time)

@nimbudew We could test this on our end and have committed the fix to master which you can check on your end too. Thanks for raising it with us. We are closing this issue for now, but feel free to reopen it if you find any more issues.

Deleted user · Answer 8 · Sun Oct 16 2016 16:49:18 GMT+0800 (China Standard Time)

@amoghmargoor @abhisheksr8 I no longer see the issue with the added fix. Thanks.