Singularity Support

[Akide Liu]

Proposal Summary

The proposed feature is to provide integrated support for Singularity in ClearML. Singularity is a powerful tool for packaging and running reproducible computational workloads. The aim is to allow users to create, manage, and execute Singularity containers directly through ClearML, just like they would with Docker. It includes being able to specify Singularity images in the ClearML configuration files and the system automatically handling the creation, management and execution of Singularity containers.

Motivation

ClearML already supports Docker for managing containers. However, in many academic, scientific, and high-performance computing (HPC) contexts, Singularity is more prevalent. It provides several features that are important in these environments, like compatibility with shared file systems, user privileges control and the ability to run containers without root access.

Adding Singularity support to ClearML would help to simplify the computational workflows of these users. They would be able to easily package their experiments in Singularity containers, run them via ClearML, and have experiment tracking and resource monitoring fully integrated.

Moreover, for organizations that have restrictions on Docker use due to security considerations, Singularity provides an excellent alternative for containerization. It would be beneficial to offer these users the flexibility of choosing the containerization technology that best suits their needs and organizational policies.

Related Discussion

[Noam Wasersprung]

Thanks for suggesting @AkideLiu

While this is not in our immediate plans, we'll definitely add this as a potential roadmap item if ClearML's user community provides a strong demand.