GCP retry mechanism

Question

GCP retry mechanism

cometta opened this issue 2 years ago · comments

i'm using version 6. May i know any example on how to enable retry on flaky internet connection when reading stream from GCP bucket ?

Michael Penkov · Answer 1 · Fri Jul 29 2022 13:34:13 GMT+0800 (China Standard Time)

There is no existing example code I'm aware of that would help in your case.

@petedannemann may be able to help here.

Peter Dannemann · Answer 2 · Wed Aug 03 2022 07:35:17 GMT+0800 (China Standard Time)

smart-open's GCS reader uses google.cloud.storage.Blob under the hood. The network call when reading is done by the Blob's download_as_bytes method in smart open's code here. We do not specify a retry in smart-open so we use the default one provided by download_as_bytes which you can read more about in the first link provided. If that is not sufficient for your use case, please submit a PR allow the user to provide a retry to the download_as_bytes calls used in smart-open.

comet · Answer 3 · Wed Aug 03 2022 09:09:33 GMT+0800 (China Standard Time)

Thank you for responding @petedannemann @mpenkov got it. need to pass retry object to download_as_bytes to enable retry.

Peter Dannemann · Answer 4 · Tue Dec 13 2022 04:50:47 GMT+0800 (China Standard Time)

With #744 this is now supported by passing in a Retry object or ConditionalRetryPolicy as a blob_open_kwarg with a key of retry. You can read more about this here. I think we can close this issue now.