This repository contains a collection of slides, exercises, and files for the Container Tutorial presented at various conferences and other venues. Below is an abstract of what the Tutorial covers and descriptions for each event. A directory exists for most events we have presented at, with some descriptions of how we presented at the event below.
In recent years, using containers has been rapidly gaining traction as a solution to lower the barriers to using more software on HPC and cloud resources. However, significant barriers still exist to actually doing this in practice, particularly for well-established community codes which expect to run on a particular operating system version or resource. Additional barriers exist for researchers unfamiliar with containerization technologies. While many beginner tutorials are available for building containers, they often stop short of covering the complexities that can arise when containerizing scientific computing software. The goal of this full-day tutorial is to demonstrate and work through building and running non-trivial containers with users. We will containerize community scientific software, exhibit how to share with a larger community via a container registry, and then run on a completely separate HPC resource, with and without the use of a Science Gateway. The subject matter will be approachable for intermediate to advanced users, and is expected to be of interest to a diverse audience including researchers, support staff, and teams building science gateways.
Details coming soon.
The tutorial was presented at Gateways 2020 as a full day (~8 hrs) in total split up across 2 days. The materials in the Gateways2020 directory are organized in accordance with the schedule of events at the conference, as detailed in the list below. Supplemental materials, such as Dockerfiles and Exercise instructions, have been included as well.
-
Introduction to Containers by Jeremy Fischer, Indiana University
- Discussion of how containers can help Gateways
- What is Docker?
- Docker Hub
- What is Singularity?
- SingularityHub
- Differences / relationship to each other
-
Environment Introduction by Eric Coulter, Indiana University
- Distributed login credentials for tutorial resources (not available outside of conference attendance)
- Overview of container development environment
-
Simple (Docker) Container Creation by Sanjana Sudarshan, Indiana University
- Build via Dockerfile
- Exploring Dockerfile structure/conventions
- Container examples
-
Exercise - Docker Build by Stephen Bird, Indiana University
- Build simple Docker container
-
Singularity Containers by Eric Coulter, Indiana University
- Definition Files
- Creating Singularity containers (from scratch)
- Singularity user environment
-
Docker To Singularity Conversion by Peter Vaillancourt, Cornell University
- Best Practices for conversion
- Example conversion
-
Exercise - Singularity by Eric Coulter, Indiana University
- Convert to Singularity
- Build via definition file
- Upload to registry
- Run via SLURM
-
Gateways Tie-in by Suresh Marru, Indiana University
- Overview of components in a Gateway lifecycle
- Big picture in a gateways context
- Discuss Part 4 Bucket choice
-
Open Q&A by Jeremy Fischer, Indiana University
- Catchup and unresolved questions
-
Exercise - MATLAB Container Build by Stephen Bird, Indiana University
- Dockerfile walkthrough
- Container build and run
- Conversion
-
Best Practices & Advanced Topics by Peter Vaillancourt, Cornell University
- Lifecycle of Containers
- Development vs. Production
- Reducing container sizes
- Data management
- Reproducibility
- Security
- Container Orchestration
-
Containers for Gateways with Exercises by Suresh Marru, Indiana University
- Intro to Gateways
- Sign in to a gateway and run a pre-packaged containerized application with pre-provided inputs
- Execute applications with user provided input files and plot outputs
- Add a Containerized application to the Gateway
- Goals to Cover:
- What is a Gateway for?
- What features do Gateways add when using containers?
- How can Gateways now enable the community with these containers?
- Run/provide your app via a gateway!
-
Bucket of Topics (Open to Audience Input) by Eric Coulter and Suresh Marru, Indiana University
- Gateway Hosting
- Container Orchestration and management
- CI/CD Overview (Code -> Gateways pipelines)
- Container Registries (Building or usage of public registries)
- Decisions about container vs. host for software/dependency locality
All tutorials at PEARC20 were shortened to half-day (~4 hrs), so the resulting tutorial is shorter than our planned full-day (~8 hrs). If you are interested in the full-day tutorial, we presented it at Gateways2020 and the materials are shared here also. The materials in the PEARC20 directory have been organized in order of how they were originally presented:
- Introduction to Containers by Jeremy Fischer, Indiana University
- Simple Container Creation by Sanjana Sudarshan, Indiana University
- Docker to Singularity Conversion by Peter Vaillancourt, Cornell University
- Exercise 1 - Build, Share, and Run a Container by Eric Coulter, Indiana University
- Docker MATLAB Runtime Container by Stephen Bird, Indiana University
- Containerization: Best Practices and Advanced Topics by Peter Vaillancourt
- Science Gateways by Suresh Marru, Indiana University
Supplemental materials, such as Dockerfiles and Exercise instructions, have been included as well.
If you have Questions, please contact us via
help@xsede.org with XCRI in the subject line.