Read and write Python objects from/to S3, caching them on your hard drive to avoid unnecessary IO. Special care given to pandas dataframes.
import s3bp
s3bp.save_object(name_to_id_dict, filepath, 'user-data-bucket')
last_week_dataset = s3bp.load_object(second_filepath, 'my-dataset-s3-bucket')
s3bp uses the following packages:
- boto3
- botocore (instaled with boto3)
- dateutil (a.k.a. python-dateutil)
- pyyaml
- pandas
- feather-format
The boto3 package itself requires that you have an AWS config file at ~/.aws/config
with your AWS account credentials to successfully communicate with AWS. Read here on how you can configure it.
You can install s3bp using:
pip install s3bp
Save an object to your bucket with:
import s3bp
name_to_id_dict = {'Dan': 8382, 'Alon': 2993}
s3bp.save_object(name_to_id_dict, '~/Documents/data_files/name_to_id_map', 'user-data-bucket')
File upload is done asynchronously and in the background by default, only printing exceptions (and not throwing them). If you'd like to wait on your upload, and/or for a failed upload to raise an exception rather than print one, set wait=True
:
s3bp.save_object(name_to_id_dict, '~/Documents/data_files/name_to_id_map', 'user-data-bucket', wait=True)
Load an object from your bucket with:
name_to_id_dict = s3bp.load_object('~/Documents/data_files/name_to_id_map', 'user-data-bucket')
Notice that if the most updated version is already on your hard drive, it will be loaded from disk. If, however, a more updated version is found on the S3 (determined by comparing modification time), or if the file is not present, it will be downloaded from S3. Furthermore, any missing directories on the path will be created.
Objects are saved as Python pickle files by default. You can change the way objects are serialized by providing a different serializer when calling save_object
. A serializer is a callable that takes two positonal arguments - a Python object and a path to a file - and dumps the object to the given file. It doesn't have to serialize all Python objects successfully.
For example:
def pandas_df_csv_serializer(pyobject, filepath):
pyobject.to_csv(filepath)
import pandas as pd
df1 = pd.Dataframe(data=[[1,3],[6,2]], columns=['A','B'], index=[1,2])
s3bp.save_object(df1, '~/Documents/data_files/my_frame.csv', 'user-data-bucket', serializer=pandas_df_csv_serializer)
Notice that a corresponding deserializer will have to be provided when loading the object by providing load_object
with a deserializing callable through the deserializer
keyword argument.
You can set a default bucket with:
s3bp.set_default_bucket('user-data-bucket')
You can now load and save objects without specifying a bucket, in which case the default bucket will be used:
profile_dict = s3bp.load_object('~/Documents/data_files/profile_map')
Once set, your configuration will presist through sessions. If you'd like to unset the default bucket - making operations with no bucket specification fail - use s3bp.unset_default_bucket()
.
You can set a specific directory as a base directory, mapping it to a specific bucket, using:
s3bp.map_base_directory_to_bucket('~/Desktop/labels', 'my-labels-s3-bucket')
Now, saving or loading objects from files in that directory - including sub-directories - will automatically use the mapped bucket, unless a different bucket is given explicitly. Furthermore, the files uploaded to the bucket will not be keyed by their file name, but by the sub-path rooted at the given base directory.
This effectively results in replicating the directory tree rooted at this directory on the bucket. For example, given the above mapping, saving an object to the path ~/Desktop/labels/user_generated/skunks.csv
will also create a labels
folder on the my-labels-s3-bucket
, a user_generated
folder inside it and will upload the file into labels/user_generated
.
You can add as many base directories as you want, and can map several to the same bucket, or each to a different one.
This can be used both to automatocally backup entire folders (and their sub-folder structure) to S3 and to synchronize these kind of folders over different machines reading and writing Dataframes into them at different times.
Special care is given to pandas Dataframe objects, for which a couple of dedicated wrapper methods and several serializers are already defined. To save a dataframe use:
import s3bp
import pandas as pd
df1 = pd.Dataframe(data=[[1,3],[6,2]], columns=['A','B'], index=[1,2])
s3bp.save_dataframe(df1, '~/Desktop/datasets/weasels.csv', 'my-datasets-s3-bucket')
This will use the default CSV serializer to save the dataframe to disk. Similarly, you can load a dataframe from your bucket with:
df1 = s3bp.load_dataframe('~/Desktop/datasets/weasels.csv', 'my-datasets-s3-bucket')
To use another format assign the corresponding string to the format
keyword:
s3bp.save_dataframe(df1, '~/Desktop/datasets/weasels.csv', 'my-datasets-s3-bucket', format='feather')
Suported pandas Dataframes serialization formats:
- CSV
- Excel
- Feather (see the feather package)