Thread safety
asantos82 opened this issue · comments
Hi @asantos82 !
Thank you for bringing this up and opening this issue!
So... judging from the documentation you have linked, I would understand that boto3 clients are generally thread-safe, except for the listed caveats. I would not see a requirement in the docs though that clients must not be initialized within threads. What the documentation shows, in my interpretation, is the one case that could potentially be dangerous/unclear, where a single client is initialized outside and then shared with multiple threads. (But it's demonstrated that this case is still fine with boto3.)
In the aws-list-resources case, each thread has their own client object that is not shared with other threads. Due to a lack of sharing of an object, I would also not see a thread safety issue then?
Thank you and cheers,
Michael
Hi @asantos82 !
I continued reading on the topic and it's quite a rabbit hole, but very interesting. In the end, the following resource was most helpful:
https://medium.com/@life-is-short-so-enjoy-it/aws-boto3-misunderstanding-about-thread-safe-a7261d7391fd
As it turns out, boto3 clients are thread-safe. But the call to create client objects ("boto3.client(...)") is not, because there is a shared boto3 session object used underneith.
I thus changed the implementation of aws-list-resources: Each thread now creates their own boto3 session object and clients are only derived from the thread-local session object.
If you have the time, I'm happy if you take a look.
Cheers,
Michael
Hi @michael-kirchner-at
Thanks for the quick turn around time.
I can take a look at the code. Where can I find it? I can not find a PR o branch with it
It's already on the main branch :) I ran couple of tests to ensure that the code before and after still produce the same output, which was the case. Thank you!
It looks ok.
Thanks for the fix
Thank you for your checks and for opening the issue initially!