OverBoard

OverBoard is a lightweight yet powerful dashboard to monitor your experiments.

Load remote experiments in real-time through SSH/FTP and others.

Custom visualisations (tensors and any custom plot with familiar MatPlotLib syntax).

A sortable table of hyper-parameters with Python-syntax filtering.

Plot percentile intervals (shaded plots), custom X/Y axes, and hyper-parameters (bubble plots).

Installation

You can install the dependencies with:

With Conda: conda install pyqt=5.12 pyqtgraph=0.11 -c conda-forge
With pip: pip install pyqt5==5.12 pyqtgraph==0.11

Finally, OverBoard itself can be installed with: pip install fs==2.4 overboard (Conda seems to be too strict when installing PyFileSystem/fs, so pip should be preferred.)

Optional:

pip install fs.sshfs to support remote files through SSH.
PyOpenGL 3.1 (either through conda or pip) if you intend to use custom 3D plots with PyQtGraph.
MatPlotLib if you intend to use it for custom plots.

Installation - logger only

Your scripts can log data without installing the full GUI and its dependencies (so your remote GPU cluster does not need PyQt at all).

Just use: pip install overboard_logger

And remember to import overboard_logger instead of overboard in your scripts.

Usage

Main interface: python3 -m overboard <logs-directory>
Logging experiments is simple:

from overboard import Logger

with Logger('./logs') as logger:
  for iteration in range(100):
    logger.append({'loss': 0, 'error': 0})

You can also pass in a meta keyword argument, which can be a dict with hyper-parameters names and values (or other meta-data), to help organize your experiments. These will be displayed in a handy table, which supports sorting and filtering. The meta data can also be an argparse.Namespace, which is useful if your hyper-parameters are command-line arguments parsed with argparse.

By default a unique folder (using the current timestamp) is created for the logs. For full documentation on initialization arguments and other methods, type pydoc overboard on the command-line (Python built-in doc viewer).

You can also check the examples directory:

examples/basic.py: A minimal example. Generates some test logs.
examples/mnist.py: The mandatory MNIST example. Also shows a custom MatPlotLib plot (a confusion matrix).
examples/example_2d.py: Example custom 2D plot, with PyQtGraph (faster than MatPlotLib).
examples/example_3d.py: Example custom 3D plot, with PyQtGraph. Requires PyOpenGL.

A note about importing: You can either import the Logger class from overboard or from overboard_logger. If you installed the "logger only" version as described above (no dependencies), then you can only import from overboard_logger.

Remote experiments

It's as simple as:

python -m overboard ssh://username:password@hostname/path-to-experiments

...replacing your SSH user name, password, host name/server, and the directory where OverBoard should look for experiments. The default port is 22, which you can override with :23 or another port number after the host name.

There are a number of other settings, which you can append after the path. For example, append ?keepalive=60&config_path=~/.ssh/config to set the keepalive packets interval to 60 seconds, and specify an OpenSSH configuration file path.

Other remote file systems are available by using appropriate prefixes (e.g. ftp://).

Interface tips & tricks

Click a plot title to hide it (moving it to the bottom) or to show it again.
Click a plot line to select that experiment in the table, and vice-versa.
Use the mouse wheel to zoom and drag to pan in a plot or custom visualization.
Click the "A" button in the bottom-left of a plot to reset the zoom/pan.
Choose an hyper-parameter as the "X axis" in the sidebar to compare different hyper-parameters graphically (bubble plot). Click a bubble to highlight that experiment. If the hyper-parameter is numerical but has wildly different orders of magnitude, check "Treat X as categorical". By default, each bubble's Y coordinate (e.g. accuracy) is taken from the last iteration; in "Scalar display" you can choose the maximum or minimum across iterations.
If you have multiple runs with different random seeds (stored as hyper-parameter "seed" for example), select "Merge: seed" in the sidebar to merge them into shaded plots (you can then select whether to show the mean, median, range or standard deviations).
You can split all experiments into different plots by selected "Panels: One per run" in the sidebar, and selecting a metric to plot in "Y axis". Similarly, they can be split by a hyper-parameter value (e.g. a different algorithm in each panel, but each has multiple plots/runs).

Author

João Henriques, Visual Geometry Group (VGG), University of Oxford

jotaf98 / overboard