Performance For Various MD Systems
I’m currently working on a project that requires the use of Molecular Dynamics simulations to generate raw data. It’s not really the focus of the project, but I do need some decently long simulations (in the neighborhood of 400ns) to get meaningful data.
And oh boy can it be a pain to get this data.
In spite of all the work that’s gone in to it, MD is still seems like a very finicky machine with lots of knobs and buttons and no such thing as “reasonable defaults.” As a result, it’s very easy to run into lots of problems when first trying to run your own MD simulations.
One of the problems I ran into recently was that a lot of my simulations in NAMD were running a lot slower than I would have liked (on the order of 15ns/day, or 1 month to get a result). After asking around for suggestions and doing some debugging, I managed to find a few things that helped my simulations a little, and a few programs that really sped it up.
DISCLAIMER: These are not comparative benchmarks! I don’t know nearly enough about MD to meaningful comparative benchmarks. In particular, in the comparison section, the forcefields and integrators for each program are slightly different.
QwikMD: A few rough edges #
QwikMD is a plugin for VMD which allows someone with relatively little experience (like me!) to set up and run an MD simulation with relative ease. Just load the PDB file, resolve the issues that the struture analysis tools warn you about, set a few flags, and hit the “Prepare” button to get a directory loaded with scripts and parameters that you can upload to the supercomputer. Easy!
Unfortunately, as I discovered, there are some performance issues with the defaults that QwikMD puts out. The two biggest issues, at least with explicit solvent models, are the size of the waterbox and the default simulation parameters.
NAMD Parameters #
I’ll let João Ribeiro explain what’s up with the NAMD parameters generated by QwikMD:
Please allow me to add some notes about the configuration files produced by QwikMD. QwikMD has a lot of ludic behavior in the selection of the MD parameters in the config files, namely, high frequency of energy outputs and trajectory saving frequency (dcd freq), which is useful when you are starting running simulations but has high penalties in the NAMD performance. Also, running your simulations with the option “Live view” mode (Interactive Molecular Dynamics activated) also decreases your performance substantially, as NAMD-VMD communication occurs every so often.
In summary, if you are trying to squeeze the most ns/day from your machine, please increase periods of saving frames (dcd freq) and output events (outputpressure, outputenergies and etc.) and run your simulations in the background (Live view off == IMDon off).
I don’t have numerical values for the suggestions made here, but in my experience, the changes helped a little bit–probably not by more than 10-20%.
Waterbox Size #
QwikMD, by default, will create a water box which is a cube, with edgelength equal to the diagonal length of your protein plus 15Å.
While this is large enough to definitely avoid any issues with insufficient padding when doing PME electrostatics, it’s also probably much larger than will actually be needed (the plugin writers acknowledge this themselves). We actually only need enough water so that the protein will not see a periodic image of itself when we compute the electrostatic forces on the protein. If the protein stays roughly in the same shape throughout the duration of the simulation (and that’s a major if), we only need a buffer as large as the electrostatics cutoff.
In my case, switching from the QwikMD default to a 15 angstrom padding reduced the number of atoms from 130k to 60k. This, combined with the changes in the above section, brought the performance from 13ns/day up to 43ns/day. Still not exactly what I needed, but a hell of a lot better than before.
Other MD Simulation Programs #
On the advice of one of TACC’s staff, Lei Huang, I decided to explore some other programs. Something I had noticed with NAMD was that, even with GPU acceleration, performance was still hugely dependent on the number and speed of CPU cores present.
To see if this could be fixed, I tried several of the programs that Lei had suggested: NAMD, Desmond, and OpenMM. The settings for each test were:
- [1LMB](https://www.rcsb.org/structure/1lmb)
- In an explicit water cube, padded by 15 angstroms past the side of the
protein and 0.15M NaCl
- 2fs timesteps with all bonds rigid
- PME-based electrostatics with a 12A cutoff
- Run on the GPU with 1 CPU core (except NAMD, which needed 8 CPUs)
- No internode communications
Note that there are a ton of differences that are still unaccounted for here: in particular, I’m almost certain that each simulation was using a different set of forcefields and integrators. Whether that invalidates my tests completely or not is something left to the experts–I’m just reporting what gives me the best off-the-shelf performance without worrying too much about accuracy for now.
Under these conditions, the programs gave the following numbers:
- NAMD: 42.1482 ns/day
- OpenMM: 108 ns/day
- Desmond: 170.441 ns/day
While these don’t tell us much about relative performance of these programs in a distributed environment, or for specific types of computations, in a generic “I just want to see what’s going on without turning too many dials” sense, Desmond’s performance absolutely blows everything else out of the water.
As an added bonus, Desmond puts almost all the work onto the GPU, meaning if you have a computer lying around somewhere with 4 GPUs plugged into a single motherboard (a la Maverick 2’s layout), you can very easily run multiple Desmond jobs on it without running into CPU starvation. This was a huge issue with NAMD, which could use the GPU, but demanded at least 8 CPUs per GPU to avoid having job times skyrocket.
I prefer supporting open source where I can, so if anyone knows of simple tips that would turn OpenMM or NAMD’s compute performance into something similar to Desmond’s, I’d love to hear them. However, since Desmond is currently free to academic users and leaving a 35% performance increase on the table would be a little silly, it seems that I’ll be using Desmond for my computations for the time being.