Reuse S3 session

Question

Reuse S3 session

wouterzwerink opened this issue 4 months ago · comments

🚀 Feature Request

Currently when I use S3 with an IAM role, I see StreamingDataset fetch new credentials for every shard:

There is a never ending stream of credential logs after this

That's quite inefficient, getting credentials from IAM roles is not that fast. Would be nicer to reuse credentials until they expire

Motivation

Faster is better!

[Optional] Implementation

I think it would work to just reuse the S3 Session object per thread

Additional context

Saaketh Narayan · Answer 1 · Thu Mar 21 2024 23:24:56 GMT+0800 (China Standard Time)

Hey! If it's not too much of a hassle, mind submitting a PR with your proposed change? I'd be happy to review

Wouter Zwerink · Answer 2 · Fri Mar 22 2024 22:18:22 GMT+0800 (China Standard Time)

Hey! If it's not too much of a hassle, mind submitting a PR with your proposed change? I'd be happy to review

Sure! I made a fix for this that worked earlier, but will need to clean it up a bit before submitting. Will take a look somewhere next week

Saaketh Narayan · Answer 3 · Mon Apr 01 2024 23:53:13 GMT+0800 (China Standard Time)

Perfect, thank you @wouterzwerink! Feel free to tag me when the PR is up.

Saaketh Narayan · Answer 4 · Tue Apr 09 2024 04:26:51 GMT+0800 (China Standard Time)

@wouterzwerink Hey, just wanted to follow up on this, mind submitting a quick PR if/when you have some time? Thanks!!

Xuan (Sean) Hu · Answer 5 · Thu Jul 18 2024 17:47:54 GMT+0800 (China Standard Time)

I am interested in this issue (actually we need it for potential performance improvement). I think the problem is in which level we want to keep a boto3 seesion. Maybe keep one seesion for each stream? If so, I suppose to create an s3 client in stream and reuse it when trigger download_file() in Stream._download_file(). Any comments?