Containers
Have you ever found yourself saying:
I don't want to use Hyak because I can't install whatever I want? I found this command online and I can't setup the random "program" package with
sudo apt-get install program? Hate Hyak because you somehow want to use Ubuntu?I'm trying to work on my local computer then move it to Hyak and resume without friction? How about you set up something and now want to share that exact compute environment with a collaborator (or vice-versa)?
The answer to either (or both) of those things are containers! Software containers are a way to package everything you need into a file to send around and have it work exactly the same across environments. The most popular containerization format is Docker but that does require administrative access to run natively, so for shared platforms (e.g., HPC clusters like Hyak) an alternative called Apptainer was developed. Almost every Docker container can be seamlessly converted to Apptainer so they're effectively interchangable.
What are the costs, trade offs, or downsides? You might imagine performance or that containerized applications run slower than native ones. This is not always true, in most instances they are near equivalent.
Apptainer (formerly Singularity)#
March 2022: 'Singularity' became a Linux Foundation supported project and was renamed 'Apptainer' [www].
The official Apptainer documentation [www] is the best source.
Ubuntu apt-get Example#
Let's say that you want to use git and the current version of git on Hyak is 1.8.3.1 as shown below.
Let's say you want a newer version AND you also want it running on Ubuntu for some reason. Here we'll walk you through installing the latest git binary using apt repositories for Ubuntu 16.04 [www] or "Xenial Xerus".
- Get an interactive session using some variant of the below command.
- Load the Apptainer module.
- Create a Apptainer definition file. Mine is below called
tools.defto install the latestcurlandgitbinaries from the Ubuntu repositories. Please see the Apptainer definition files reference page [www] for more advanced options.
Build a Apptainer container from its definition file. The generated SIF file is your portable container.
The
.defdefinition file should either be A) executable or B) a relative path (e.g../tools.defwhile in the same directory as the file) or an absolute path (e.g./full/path/to/tools.def).When using the
--fakerootoption, build the container image in/tmp. This avoids [a potential permission issue] with our shared storage filesystem, GPFS.
- Move the container image to the desired location and run the binary.
You'll do this through the apptainer executable to distinguish it from the git binary in the main operation system.
Notice the git version here is newer than the original we started with. Success!
Container Repositories#
If you followed the tutorial above you should be able to install anything you want but why re-create the wheel? There is a large developer community out there that maintains a majority of the most common scientific applications.
Docker Hub#
The biggest collection of Docker images is from Docker Hub [www].
Let's say Docker Hub tells you the pull command for the container you want is docker pull gcc:11.1.0-bullseye. To have Apptainer grab this Docker container and convert it to a Apptainer container you'd modify the command to be apptainer pull docker://gcc:11.1.0-bullseye.
NVIDIA GPU Cloud (NGC)#
A container registry that specializes in common GPU accelerated applications or GPU software development tools is provided by NVIDIA called the NVIDIA GPU Cloud (NGC) [www]. For example, you might want to use a PyTorch container optimized for NVIDIA GPUs as seen below.
Depending on the NGC container, it might have directions on the exact pull command for Apptainer. If it does not work be sure to prepend their pull location with docker:// since these are native Docker containers that need to be converted to Apptainer.
The example above provides a Docker pull command for PyTorch but in this case you'd modify it similarly as if you got it from Docker Hub from docker pull nvcr.io/nvidia/pytorch:21.05-py3 to apptainer pull docker://nvcr.io/nvidia/pytorch:21.05-py3.
Biocontainers.pro#
A bioinformatics focused set of Apptainer containers can be found at the Biocontainers.pro registry [www]. It is a collection of (convertible to Apptainer) Docker containers as well as native Apptainer containers.
Sylabs.io Cloud Library#
The largest collection of native Apptainer containers can be found at the Sylabs.io Cloud Container Library [www]. This would be the ideal first place to look for containers built by others since it is maintained by the creators of Apptainer and provides the native container format.
NGC API Keys#
In rare occasions a container in the NGC app store is going to require that you have an API. This is the only reason you'd need to register for a user account with NGC. Once you have an NGC account and are logged in, at the top right the pull down menu select "Setup" and there's an option to "Get API Key". Save that string of text.
You only have to register your API key once but load the ngc module (i.e., module load ngc) and run ngc config set which will prompt you for your API key. It's fine to select "ascii" as an option. Your API key will be stored under a .ngc/config file in your home directory.