welldone-cloud / aws-list-resources

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Thread safety

asantos82 opened this issue · comments

Hi,
I was looking at the code and it does not look thread safe.
Per boto3 documentation Boto3Docs the client should be initiated outside of the thred.
Unless I'm missing something, here the client is being initialized within the thread L24 making is thread unsafe.

Hi @asantos82 !
Thank you for bringing this up and opening this issue!
So... judging from the documentation you have linked, I would understand that boto3 clients are generally thread-safe, except for the listed caveats. I would not see a requirement in the docs though that clients must not be initialized within threads. What the documentation shows, in my interpretation, is the one case that could potentially be dangerous/unclear, where a single client is initialized outside and then shared with multiple threads. (But it's demonstrated that this case is still fine with boto3.)
In the aws-list-resources case, each thread has their own client object that is not shared with other threads. Due to a lack of sharing of an object, I would also not see a thread safety issue then?
Thank you and cheers,
Michael

Hi @asantos82 !

I continued reading on the topic and it's quite a rabbit hole, but very interesting. In the end, the following resource was most helpful:
https://medium.com/@life-is-short-so-enjoy-it/aws-boto3-misunderstanding-about-thread-safe-a7261d7391fd
As it turns out, boto3 clients are thread-safe. But the call to create client objects ("boto3.client(...)") is not, because there is a shared boto3 session object used underneith.

I thus changed the implementation of aws-list-resources: Each thread now creates their own boto3 session object and clients are only derived from the thread-local session object.

If you have the time, I'm happy if you take a look.

Cheers,
Michael

Hi @michael-kirchner-at
Thanks for the quick turn around time.
I can take a look at the code. Where can I find it? I can not find a PR o branch with it

It's already on the main branch :) I ran couple of tests to ensure that the code before and after still produce the same output, which was the case. Thank you!

It looks ok.
Thanks for the fix

Thank you for your checks and for opening the issue initially!