sensiml / modelstore

🏬 modelstore is a Python library that allows you to version, export, and save a machine learning model to your filesystem or a cloud storage provider.

modelstore

modelstore is a Python library that allows you to version, export, save and download machine learning models in your choice of storage.

For more details, please refer to the documentation.

Features

No tracking server required

• Pip install and get going!
• Support for multiple clouds (AWS, GCP, Azure)

Upload: version all your models

• Model versioning on each upload
• Replaces all the boiler plate code you need to save models

Manage models by domains and states

• List models in a domain
• Create model states and manage which state a model is in

Download and load

• Download model archives by id
• Load models straight from your storage back into memory

Use as a command line tool

• Download models from the command line

Installation

pip install modelstore

Supported storage types

• AWS S3 Bucket (example)
• Azure Blob Storage (example)
• Google Cloud Storage Bucket (example)
• A filesystem directory (example)

Supported machine learning libraries

• Annoy
• Catboost
• Fast.AI
• Gensim
• Keras
• LightGBM
• Mxnet
• Onnx
• Prophet
• PyTorch
• PyTorch Lightning
• Scikit Learn
• Skorch
• Shap
• Tensorflow
• Transformers
• XGBoost

Is there a machine learning framework that is missing?

• Save your model and then upload it as a raw file.
• Feel free to open an issue

Example Usage

Colab Notebook

There is a full example in this Colab notebook.

Python Script

from modelstore import ModelStore
# And your other imports

# Train your model
clf = RandomForestClassifier(n_estimators=10)
clf = clf.fit(X, Y)

# Create a model store that uses a one of the storage options
# In this example, the model store is created with a GCP bucket
model_store = ModelStore.from_gcloud(
   project_name="my-project",
   bucket_name="my-bucket",
)

# Upload the archive to your model store
domain = "example-model"
meta_data = model_store.upload(domain, model=clf)

# Print the meta-data about the model
print(json.dumps(meta_data, indent=4))

# Download the model back!
target = f"downloaded-{model_type}-model"
os.makedirs(target, exist_ok=True)
model_path = model_store.download(
   local_path=target,
   domain=model_domain,
   model_id=meta["model"]["model_id"],
)

Recorded demo from Data Talks Club

Note the talk below is based on an older version of modelstore and the API has been simplified since then.

This interview and demo, recorded with Alexey from the Data Talks Club in July 2021, is based on modelstore==0.0.6. Click here to watch on YouTube.

License

Copyright 2020 Neal Lathia

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.