GS: A minimalistic Google Storage client

gs is a command line interface (CLI) and Python library that provides a set of essential commands for Google Cloud Storage. It is modeled after the AWS CLI's aws s3 command.

gs was first developed in 2018 when Google Cloud's SDKs still did not support Python 3, and were not modular so developers had to install gigabytes of additional libraries to do something as simple as talking to Google Storage. Since then, the SDKs have improved, but gs remains small, fast, and useful as a reference implementation of a standalone Google Cloud API client with full support for crc32c checksum composition. Its features are:

A minimalistic set of dependencies
A tiny footprint
Intuitive convention-driven configuration of API credentials without browser login prompts
Checksum validation to ensure end-to-end data integrity in uploads and downloads
Progress bars for long-running upload and download operations
Resumable uploads and downloads
Multithreaded directory sync and batch delete, capable of handling large numbers of objects
An attractive paging and table layout interface
A JSON object metadata output mode for feeding data to other utilities

Installation

pip install gs

Synopsis

Usage:

gs [OPTIONS] COMMAND [ARGS]...

Options:

--version Show the version and exit. --help Show this message and exit.

Commands:

`gs configure`	Set gs config options, including the API key.
`gs ls`	List buckets or objects in a bucket/prefix.
`gs cp`	Copy files to, from, or between buckets.
`gs mv`	Move files to, from, or between buckets.
`gs mb`	Create a new Google Storage bucket.
`gs rb`	Permanently delete an empty bucket.
`gs rm`	Delete objects (files) from buckets.
`gs sync`	Sync a directory of files with bucket/prefix.
`gs api`	Use httpie to perform a raw HTTP API request.
`gs presign`	Get a pre-signed URL for accessing an object.

Run gs configure to configure Google service account access credentials that will be used by the gs command. You can create a new service account key at https://console.cloud.google.com/iam-admin/serviceaccounts.

Credentials

Before making API calls, gs ingests API credentials in the following order of priority:

First, gs checks if a GOOGLE_APPLICATION_CREDENTIALS environment variable is set. If so, it attempts to load and use credentials from the service account credentials filename referenced by the variable.
If that varible is not set, gs attempts to load service account credentials previously configured with gs configure (stored in ~/.config/gs/config.json).
If that fails, gs attempts to load a service account API token from Google instance metadata.
If that fails, gs prints a warning and attempts to make API requests anonymously.

Using the Python library interface

from gs import GSClient
client = GSClient()
object_meta = client.get("b/my-bucket/o/my-object")
with client.get("b/my-bucket/o/my-object", params=dict(alt="media"), stream=True) as res:
    object_bytes = res.raw.read()
presigned_url = client.get_presigned_url("my-bucket", "my-object", expires_at=time.time()+3600)

Authors

Andrey Kislyuk

Bugs

Please report bugs, issues, feature requests, etc. on GitHub.

License

Licensed under the terms of the MIT License.

kislyuk / gs