Outfit is a lightweight library to tidy up your machine learning experiments in a simple way.
The idea of Outfit is to store in your Wardrobe
your parameters, output file, scores and features in order to be able to make a request and find out which are your best experimentation according to a given criterion.
PyPI:
pip install outfit
Dev version:
git clone https://github.com/AdilZouitine/outfit
cd outfit
pip install -r requirements.txt
pip install -e .
- Tutorial 1: Build a simple model management pipeline with outfit for a CNN with Pytorch on MNIST dataset
import datetime
# Here import all the libraries you need for your experiment
from outfit import Wardrobe, getlog
wardrobe = Wardrobe(db_path='foo/bar/mnist.db')
exp = {
'experiment_name': 'ResNet18',
'comment': 'Use differential learning rate',
'date_experiment': datetime.datetime.now()
}
wardrobe.add_experiment(**exp)
param = {
'dropout': 0.20,
'kernel_size': '3x3',
'conv_block_1_lr': 0.01,
'conv_block_2_lr': 0.001
}
# Create the instance of your model here with your parameters
wardrobe.add_dict_parameter(param)
# Do your training phase here.
output = {'training log': '/result/training_log_resnet18.txt'}
@getlog(filepath=output['training log'])
def train_model(model, loaders, loss, lr_scheduler, n_epoch):
...
output.update({
'tensorboard': '/result/event.tb',
'model': 'diff_lr_resnet18.pth'
})
wardrobe.add_dict_output(output)
score = {
'train acc': 0.96,
'train loss': 0.430,
'val acc': 0.94,
'val loss': 0.460
}
wardrobe.add_dict_score(score)
wardrobe.tidy() # commit your experiment in database
# If you want to get the best experiments
for exp in wardrobe.get_best_scores(mode='max',on_score='val acc'):
'''
Verbose is true by default and will print on the console
at each iteration the parameters, output file,
features and scores in a table format.
Also returns in dictionary the parameters, output file, features and scores.
'''
...
Output:
ββββββββββββββββββββ
β TOP 1 EXPERIMENT β
ββββββββββββββββββββ
Table : Experiment
ββββββ€ββββββββββββββββββ€ββββββββββββββββββββ€βββββββββββββββββββββββββββββββββ€βββββββββββββββββββββ
β β id_experiment β experiment_name β comment β date_experiement β
ββββββͺββββββββββββββββββͺββββββββββββββββββββͺβββββββββββββββββββββββββββββββββͺβββββββββββββββββββββ‘
β 0 β 1 β ResNet18 β Use differential learning rate β β
ββββββ§ββββββββββββββββββ§ββββββββββββββββββββ§βββββββββββββββββββββββββββββββββ§βββββββββββββββββββββ
Table : Parameter
ββββββ€βββββββββββββββββ€βββββββββββββββββββ€ββββββββββββββ€βββββββββββββββ
β β id_parameter β parameter_name β parameter β experiment β
ββββββͺβββββββββββββββββͺβββββββββββββββββββͺββββββββββββββͺβββββββββββββββ‘
β 0 β 1 β dropout β 0.2 β 1 β
ββββββΌβββββββββββββββββΌβββββββββββββββββββΌββββββββββββββΌβββββββββββββββ€
β 1 β 2 β kernel_size β 3x3 β 1 β
ββββββΌβββββββββββββββββΌβββββββββββββββββββΌββββββββββββββΌβββββββββββββββ€
β 2 β 3 β conv_block_1_lr β 0.01 β 1 β
ββββββΌβββββββββββββββββΌβββββββββββββββββββΌββββββββββββββΌβββββββββββββββ€
β 3 β 4 β conv_block_2_lr β 0.001 β 1 β
ββββββ§βββββββββββββββββ§βββββββββββββββββββ§ββββββββββββββ§βββββββββββββββ
Table : Output
ββββββ€ββββββββββββββ€ββββββββββββββββ€ββββββββββββββββββββββββββββββββββββ€βββββββββββββββ
β β id_output β type_output β path_output β experiment β
ββββββͺββββββββββββββͺββββββββββββββββͺββββββββββββββββββββββββββββββββββββͺβββββββββββββββ‘
β 0 β 1 β training log β /result/training_log_resnet18.txt β 1 β
ββββββΌββββββββββββββΌββββββββββββββββΌββββββββββββββββββββββββββββββββββββΌβββββββββββββββ€
β 1 β 2 β tensorboard β /result/event.tb β 1 β
ββββββΌββββββββββββββΌββββββββββββββββΌββββββββββββββββββββββββββββββββββββΌβββββββββββββββ€
β 2 β 3 β model β diff_lr_resnet18.pth β 1 β
ββββββ§ββββββββββββββ§ββββββββββββββββ§ββββββββββββββββββββββββββββββββββββ§βββββββββββββββ
Table : Score
ββββββ€βββββββββββββ€βββββββββββββββ€ββββββββββ€βββββββββββββββ
β β id_score β type_score β score β experiment β
ββββββͺβββββββββββββͺβββββββββββββββͺββββββββββͺβββββββββββββββ‘
β 0 β 1 β train acc β 0.96 β 1 β
ββββββΌβββββββββββββΌβββββββββββββββΌββββββββββΌβββββββββββββββ€
β 1 β 2 β train loss β 0.43 β 1 β
ββββββΌβββββββββββββΌβββββββββββββββΌββββββββββΌβββββββββββββββ€
β 2 β 3 β val acc β 0.94 β 1 β
ββββββΌβββββββββββββΌβββββββββββββββΌββββββββββΌβββββββββββββββ€
β 3 β 4 β val loss β 0.46 β 1 β
ββββββ§βββββββββββββ§βββββββββββββββ§ββββββββββ§βββββββββββββββ
Table : Feature
These solutions are great, they also offer a user interface and have many more options than my library however for a simple use where you only want to organize your experimentation and make a simple query. Both solutions seem to be overkill.