The second most widely used GPU-enabled workflow on HYAK (besides machine learning) is molecular dynamics (MD) so we wanted to test one of the most popular MD codes, gromacs [source], and ensure this driver upgrade wouldn't negatively impact our researchers. I couldn't find gromacs compiled with GPU support currently in our module collection so I used it as an opportunity to create one for you all, read on!

Using gromacs#

I'll start with the end result for those of you who just want to use it but following that I'll dive into the nuts and bolts of how we created the module so you can perform additional optimizations.

This is a GPU-enabled version of gromacs so we need a GPU first (can verify with nvidia-smi).

gromacs-2020.4 module#

Once we have a GPU we use modules to load gromacs-2020.4 and all its required dependencies (e.g., CUDA11).

All packages are sub-commands of the gmx binary so you can verify the module.

Test simulation of Lysozyme#

I used a tutorial from the gromacs website here to show it runs processes on GPU(s). The tutorial runs an MD simulation on a lysozyme but that's the extent of my study there. The commands below are a summary of the tutorial with a note that the genbox subcommand is now replaced by solvate.

The final gromacs command below starts the fun, the documentation suggests it will automatically identify the GPUs available to send work to them. However, there are more explicit GPU arguments we encourage you to explore.

You can ssh into the node you're using in a separate window to have a parallel nvidia-smi command run so we can monitor the load on the GPU(s).

We can see a process occuping each GPU so it works! At least, gromacs uses GPUs...the GPUs themselves weren't stressed heavily and that requires the user to increase the number of rank processes and match that with available GPUs. You can do this by adding arguments to the gmx mdrun command but by default it did 2 ranks per GPU it detected, which is not a lot.

(Optional) Compile Notes#

You need CUDA11, GNU Compiler, and OpenBLAS library for the version I put together but I was focused on a proof-of-concept and not squeezing out every last drop of performance. There's a lot of further optimization to be done and that's left as an exercise for the reader:

1. Try the Intel compiler and see if it provides further optimization for non-GPU parts of the workflow.
2. Try other math libraries (e.g., MKL) and see if it speeds things up.
3. Add in MPI support if you want to use multiple GPUs across multiple nodes.
4. Add in modules (e.g., PLUMED).
5. Other stuff I can't think of with compile flags [here].

Download Source#

From the login node I staged a folder in the modules directory.

Grab regression tests.

Download gromacs-2020.4 [source].

Get a GPU and Code#

I used the shared build-gpu node for an interactive session but if you are affiliated with a group that has their own you can use that instead.

Once you get a session with GPU (you can run nvidia-smi to confirm you see one). Extract regression tests.

Do the same for the gromacs code and enter the directory.

Pre-requisite Modules#

Modules loaded individually for readability but you could load all modules in one command. Get a refresher on modules here.

Compile#

I created a subdirectory within the source to compile.

Use cmake to create the Makefile. Note: if you copy-and-paste the cmake command below you will have to modify the paths referenced for your environment.

With the Makefile ready you can run make -j 4 and replace 4 with however many cores you have in your session then make install. I created the module file separately so you can load it with module load gromacs/2020.4-cuda11.1 and run the single gmx binary.

tl;dr (1) decomissioned a cluster, (2) got a bunch of GPUs for maching learning, (3) launched a cluster, and (4) new and improved documentation.

2020 has definitely been an eventful year but here on Team Hyak we've been trying to make the best of a bad situation (lemons out of lemonade and such). This year saw the decomissioning of the 1^st generation Hyak cluster, ikt, and the soft launch of our 3^rd generation Hyak cluster, klone. Our partnership with the Allen School and other departments across campus has enabled an explosion in on-campus GPU capacity for the current 2^nd generation Hyak cluster, mox. This is all very exciting, machine learning is only going to get bigger. We realize whether you do your research on your laptop, Hyak, or the cloud that at the end of the day it's all just a computer and what matters is what you can actually do with it. Therefore, we are placing more emphasis on new and improved documentation (this website) and will be doing more regular research tutorials on Hyak throughout the coming year.

We hope you have weathered the adversity 2020 brought upon everyone. It has been a tough year for sure, but may your 2021 be brighter and have improvements in store. The Hyak Team has lots of efforts in the works to benefit supporting your research and they will hit full stride in the coming year. This is one improvement we can all look forward to in 2021.

Over the past year we've done a bunch of outreach to labs within the genomics community here at the UW. Given my research background it's where I feel most comfortable and this type of research is underrepresented on the cluster. Given where the field is going it's inconceivable they aren't using Hyak or HPC in general! Those lab closet clusters and under bench servers gotta go, there is a better way and we'll show you how to do some scRNA-seq today 🙂

TODO