thevickypedia / s3-downloader

Download all the objects in an S3 bucket

Versions Supported

Language Stats

Repo Stats

Activity

Build Status

S3 Download

Python module to download objects in an S3 bucket.

Installation

pip install s3-downloader

Usage

Download objects in parallel

import s3

if __name__ == '__main__':
    wrapper = s3.Downloader(bucket_name='BUCKET_NAME')
    wrapper.run_in_parallel(threads=10)  # Defaults to 5

Download objects in sequence

import s3

if __name__ == '__main__':
    wrapper = s3.Downloader(bucket_name='BUCKET_NAME')
    wrapper.run()

Mandatory arg

• bucket_name - Name of the s3 bucket.

Optional kwargs

• prefix - Prefix to filter the objects based on their path. Defaults to None
• logger - Bring your own custom pre-configured logger. Defaults to on-screen logging.
• download_dir - Name/path of the directory where the objects have to be stored. Defaults to bucket_name at current working directory.
  
  
• region_name - AWS region name. Defaults to the env var AWS_DEFAULT_REGION
• profile_name - AWS profile name. Defaults to the env var PROFILE_NAME
• aws_access_key_id - AWS access key ID. Defaults to the env var AWS_ACCESS_KEY_ID
• aws_secret_access_key - AWS secret access key. Defaults to the env var AWS_SECRET_ACCESS_KEY

AWS values are loaded from env vars or the default config at ~/.aws/config / ~/.aws/credentials

Coding Standards

Docstring format: Google
Styling conventions: PEP 8
Clean code with pre-commit hooks: flake8 and isort

Release Notes

Requirement

python -m pip install gitverse

Usage

gitverse-release reverse -f release_notes.rst -t 'Release Notes'

Linting

pre-commit will ensure linting, run pytest, generate runbook & release notes, and validate hyperlinks in ALL markdown files (including Wiki pages)

Requirement

pip install sphinx==5.1.1 pre-commit recommonmark

Usage

pre-commit run --all-files

Pypi Package

https://pypi.org/project/s3-downloader/

License & copyright

© Vignesh Rao

Licensed under the MIT License