Multipart Upload
This is an example of how to perform S3 Multipart Upload with Python. The procedure is as follows:
- Inform S3 that a multipart upload is incoming with
create_multipart_upload
. - Select a part_size (e.g. 6 MiB) and perform separate
upload_part
operations of a large file. This is done sequentially in this example, but at scale this should be parallelized. - Inform S3 that the upload is complete via
complete_multipart_upload
. S3 will assemble the parts into a single S3 object.
Note that incomplete multipart uploads incur storage costs; this can be minimized with a lifecycle policy.
Dependencies
This project assumes pipenv
is available on the system.
Install the dependencies and activate the shell:
pipenv install
pipenv shell
Running the Example
Select a file larger than 6MiB and provide the full path for multipart upload. This should be run in an environment with access to AWS credentials and permissioned with write access to the chosen bucket.
./script.py --file /Users/dliggat/Downloads/large-file.txt --bucket dliggat-multi
# INFO:__main__:Namespace(bucket='dliggat-multi', file='/Users/dliggat/Downloads/large-file.txt')
# INFO:__main__:Sending part 1 of 7
# INFO:__main__:Sending part 2 of 7
# INFO:__main__:Sending part 3 of 7
# INFO:__main__:Sending part 4 of 7
# INFO:__main__:Sending part 5 of 7
# INFO:__main__:Sending part 6 of 7
# INFO:__main__:Sending part 7 of 7
# INFO:__main__:Multipart upload of /Users/dliggat/Downloads/large-file.txt to s3://dliggat-multi complete