Enterprise-Group-Ltd / aws-s3-size

Reports S3 bucket size, including each storage type Standard, Infrequent Access, and Reduced Redundancy

EGL AWS S3 Bucket Size Utility

This shell script reports the the size of AWS S3 buckets and writes it to JSON and CSV files

This utility produces output that:

• Answer the question: "What size are the AWS S3 buckets in our multiple AWS accounts?"
• Create an audit trail of AWS S3 bucket size
• Is suitable for use in databases, spreadsheets, or reports

This utility provides S3 bucket size functionality unavailable in the AWS console or directly via the AWS CLI API.

This utility can:

• Capture the S3 bucket size for any time period in a single or all AWS accounts
• Write the S3 bucket sizes to JSON and CSV files in bytes, megabytes, gigabytes, and terabytes for each storage type (note: CSV is only the sum of all storage types, JSON includes sum and each storage type)

Getting Started

1. Instantiate a local or EC2 Linux instance
2. Install or update the AWS CLI utilities
   • The AWS CLI utilities are pre-installed on AWS EC2 Linux instances
   • To update on an AWS EC2 instance: $ sudo pip install --upgrade awscli
3. Create an AWS CLI named profile that includes the required IAM permissions
   • See the "Prerequisites" section for the required IAM permissions
   • To create an AWS CLI named profile: $ aws configure --profile MyProfileName
   • AWS CLI named profile documentation is here: Named Profiles
4. Install the bash shell
   • The bash shell is included in most distributions and is pre-installed on AWS EC2 Linux instances
5. Install jq
   • To install jq on AWS EC2: $ sudo yum install jq -y
6. Download this utility script or create a local copy and run it on the local or EC2 Linux instance
   • Example: $ bash ./aws-s3-size.sh -p AWS_CLI_profile -s 2018-02-12T02:00:00 -e 2018-02-12T03:59:59

Prerequisites

• bash - Linux shell
• jq - JSON wrangler
• AWS CLI - command line utilities (pre-installed on AWS AMIs)
• AWS CLI profile with IAM permissions for the AWS CLI commands:
  • aws cloudwatch get-metric-statistics (used to pull S3 bucket size )
  • aws sts get-caller-identity (used to pull account number )
  • aws iam list-account-aliases (used to pull account alias )

Deployment

To execute the utility:

• Example: $ bash ./aws-s3-size.sh -p AWS_CLI_profile -s start timestamp -e end timestamp

To directly execute the utility:

1. Set the execute flag: $ chmod +x aws-s3-size.sh
2. Execute the utility
   • Example: $ ./aws-s3-size.sh -p AWS_CLI_profile -s start timestamp -e end timestamp

Output

• JSON 'S3 bucket overall and each storage type size' per account file
• CSV 'S3 bucket overall size' per account file
• Info log (execute with the -g y parameter)
  • Example: $ bash ./aws-s3-size.sh -p AWS_CLI_profile -s start timestamp -e end timestamp -g y
• Console verbose mode (execute with the -b y parameter)
  • Example: $ bash ./aws-s3-size.sh -p AWS_CLI_profile -s start timestamp -e end timestamp -b y

Contributing

Please read CONTRIBUTING.md for the process for submitting pull requests to us.

Versioning

We use SemVer for versioning.

Authors

• Douglas Hackney - dhackney

License

This project is licensed under the MIT License - see the LICENSE.md file for details

Acknowledgments

• Key jq answers by jq170727
• Progress bar
• Dynamic headers fprint
• Menu
• Countless other jq and bash/shell man pages, Q&A, posts, examples, tutorials, etc. from various sources