pgagarinov / pytorch-hyperlight

The ML micro-framework built as a thin wrapper around PyTorch-Lightning and Ray Tune frameworks to push the boundaries of simplicity even further.

PyTorch Hyperlight

The ML micro-framework built on top of PyTorch-Lightning and Ray Tune to push the boundaries of simplicity even further.

PyTorch Hyperlight provides a growing set of reusable components for running train-validate-test cycles and hyper-parameter tunning for your models with less amount of repeated source code. Just inherit from one of the base classes provided by PyTorch-Hyperlight, inject your models into the superclass constructor and concentrate on improving your model quality, not on the boilerplace code!

Start by cloning one of the examples below and modifying it for your needs!

📜 Natural Language Processing	🔍 Image Classification	🌀 Semantic Segmentation	➿ Neural Style Transfer
		source: https://cs.stanford.edu/~roozbeh/pascal-context/

PyTorch Hyperlight key principles

• No wheel reinvention. Parts of PyTorch Lightning or Ray Tune that already provide simple enough APIs are used as is. PyTorch Hyperlight just makes use of those frameworks easier by minimizing an amount of boilerplate code.
• Opinionated approach that doesn't try to be flexible for every possible task. Instead PyTorch Hyperlight tries to address fewer usecase better by providing pre-configured integrations and functionaly out of the box.
• Minimalistic user-facing API allows to do research by using a single Runner class that is similar to PyTorch Lightning's Trainer but is a higher-level abstraction.
• Expects both model and data as definitions (classes), not as objects. All this is done to minimize problems with data serialization in Ray Tune which runs trials in separate processes. For
  • training/validation data this means that Hyperlight API expects a user to provide a function that returns DataLoaders, not ready-to-use DataLoader. Of course you can attach data to your functions with functools.partion but this is not recommended;
  • model it means that PyTorch Hyperlight expect the class representing your PyTorch-Lightning module, not the instantiated module object.

Features

• All hyper-parameters are defined in a single dictionary.
• Plotting of training, validation and testing stage metrics during and after training.
• Trial comparison reports in form of both tables and graphs.
• Pre-configured integration with Ray Tune for ASHA scheduler and HyperOpt optimization algorithm for out of the box hyper-parameter tuning.
• Logging the training progress on console (via tabule library)
• Pre-configured integration with WandB that works for both single runs and hyper-parameter optimization runs (via Ray Tune)
• A growing collection of re-usable base classes (tasks) for different ML domains that you can inherit your PyTorch-Lightning modules from.
• A growing collection of tested Jupyter notebooks demonstrating various PyTorch-Hyperlight usage scenarious.

Assumptions

As most of opinionated frameworks PyTorch Hyperlight makes few assumptions about the way you organize your code:

• You are familiar with PyTorch-Lightning, if not - refer to PyTorch Lightning awesome documentation.

• Metrics that you log from your PyTorch Lightning module should have pre-defined prefixes and suffixes:

  • "val", "train" or "test" ("val_f1_step" for example) as a prefix
  • "epoch" or "step" ("train_f1_epoch" for example) as a suffix

• DataLoaders should be returned by a function as a dictionary. The function should have "batch_size" as a regular parameter and "n_workers" as a key word parameter. They reason PyTorch Hyperlight doesn't rely on LightningDataModule from PyTorch Lightning is LightningDataModule might contains some data that would have to be serialized in the master process and de-serialized in each Ray Tune worker (workers are responsible for running hyper-parameter search trials).

• WandB API key is placed in ~/.wandb_api_key file.

Examples

Jupyter notebooks

Image classification

1. Boring MNIST
2. Boring MNIST model comparison
3. Vision Transformer (ViT) for facial image classification based on gender
4. Hybrid Vision Transformer (ViT) for facial image classification based on gender
5. Hybrid Vision Transformer (ViT) for CIFAR100 image classification

Semantic segmentation

1. Semantic segmentation model comparison

Neural style transfer

1. Plain simple NST
2. Multi-style NST
3. NST Telegram bot

Natural language processing

1. BERT finetuning on a subset of SST2

Installation

PyTorch Lightning doesn't have a pip package just yet so please run the following command to install it directly from git

Pip

Just run pip install pytorch_hyperlight.