C2FO / vfs

Pluggable, extensible virtual file system for Go

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Remove unnecessary write to temp when reading in S3 backend

funkyshu opened this issue · comments

Is your feature request related to a problem? Please describe.
Currently, in the S3 backend, anytime you call Read() or Seek(), the source file is copied locally. This was likely originally done to accommodate easy use of os.Seek() when reading. However, this means we can't truly stream from an s3.File.
This should improve copy time by removing the need to copy the contents locally.

Describe the solution you'd like
Remove the need for calling tempfile by tracking cursor positions when read, write, or seek operations are called. Use cursor position to set GetObjectInput.Range value to "seek" on Reads.

Describe alternatives you've considered
n/a

Additional context
Add any other context or screenshots about the feature request here.