GROMACS (www.gromacs.org) is a versatile package to perform molecular dynamics, i.e. simulate the Newtonian equations of motion for systems with hundreds to millions of particles. It is primarily designed for biochemical molecules like proteins and lipids that have a lot of complicated bonded interactions, but since GROMACS is extremely fast at calculating the nonbonded interactions (that usually dominate simulations) many groups are also using it for research on non-biological systems, e.g. polymers.

GROMACS manual, downloadable in several formats.

Versions

For basic information about setting up GROMACS jobs, read the GROMACS documentation. There is a set of tutorials at http://www.gromacs.org/Documentation/Tutorials

Biowulf is a heterogenous cluster, and nodes have different #cores/node. See the chart in the user guide

Sample script for a GROMACS 4.6.1 run on Infiniband:

#!/bin/bash
# this file is Run_Gromacs
#PBS -N Gromacs
#PBS -k oe
#PBS -m be

# use the module for the Gromacs version and network that you want
module load gromacs/4.6.1

cd /data/user/my_gromacs_dir

grompp > outfile 2>&1
`which mpirun` -machinefile $PBS_NODEFILE -n $np `which mdrun_mpi` >> outfile 2>&1

The script can be submitted with the qsub command. The number of processes should be chosen to match the number of cores on the node. The cores on each type of node is listed in the second column of the 'freen' command.

NIDDK/LCP users: qsub -v np=32 -l nodes=2 -q lcp Run_Gromacs

qsub -v np=32 -l nodes=4:ib Run_Gromacs

qsub -v np=16 -l nodes=8:ipath Run_Gromacs 

qsub -v np=4 -l nodes=2 Run_Gromacs

qsub -v np=8 -l nodes=2:dc Run_Gromacs

Gromacs 4.5.5 has also been built with Plumed 1.3, a plugin for free energy calculations in molecular systems. Plumed website. Free energy calculations can be performed as a function of many order parameters with a particular focus on biological problems, using state of the art methods such as metadynamics, umbrella sampling and Jarzynski-equation based steered MD.

Sample batch script:

#!/bin/bash

#PBS -N myjob
#PBS -m be

module load gromacs/4.5.5+plumed-ib
`which mpirun` -machinefile $PBS_NODEFILE -n $np `which mdrun_mpi` >> outfile 2>&1

qsub -v np=32 -l nodes=4:ib myscript

To run on Gige or Ipath, use the appropriate module as in the chart on the top of this page. Check the number of cores per node for the node type you plan to use, and set np in the qsub command appropriately.

See the Gromacs on GPU page.

Details about running replica exchange with Gromacs are on the Gromacs website. Multiple tpr files need to be generated from multiple *.mdp files with different temperatures. Below is a sample script for generating the tpr files. (courtesy Jeetain Mittal, NIDDK)

#!/bin/csh -f 

set ff = $argv[1]
set s = $argv[2]
set proot = 2f4k
#set ff = amber03d
set i = 0

while ( $i < 40 ) 

set fileroot = "${proot}_${ff}"
set this = "trexr"

if ( $s == 1 ) then 
set mdp = "mdp/trex_ini${i}.mdp"
set gro = "unfolded.gro" 
else
set sprev = $s
@ sprev--
set mdp = "mdp/trex_cont${i}.mdp"
set gro = "data/gro/${fileroot}_${this}_s${sprev}_nd${i}.gro"
endif

# 40 rep
grompp -v -f $mdp -c $gro \
-o data/tpr/${fileroot}_${this}_nd${i}.tpr \
-p ${fileroot}_ions.top

@ i++
end

Gromacs 4.0 can run with each replica on multiple processors. It is most efficient to run each replica on a dual-core node using all the processors on that node. This requires creating a specialized list of processors with the command make-gromacs-nodefile-dc (which is in /usr/local/bin) as in the sample script below.

#!/bin/bash
# this file is Run_Gromacs_RE
#PBS -N Gromacs_RE
#PBS -k oe
#PBS -m be

# set up PATH for gige or ib nodes
export PATH=/usr/local/openmpi/bin:/usr/local/gromacs/bin:$PATH

cd /data/user/my_gromacs_dir

#create the specialized list of processors for RE
make-gromacs-nodefile-dc

/usr/local/openmpi/bin/mpirun -machinefile ~/gromacs_nodefile.$PBS_JOBID \
      -np $np /usr/local/gromacs/bin/mdrun_mpi \ 
      -multi $n -replex 2000 >> outfile 2>&1

Submit this script to the dual-core nodes with:

qsub -v np=128,n=32 -l nodes=32:dc Run_Gromacs_RE

The above command will submit the job to 32 dual-core (either o2800 or o2600) nodes. Each of the 32 replicas will run on all 4 processors of each node. The number of processors (np=128) and the number of nodes (n=32) is passed to the program via the -v flag in qsub.

It is critical to determine the appropriate number of nodes on which to run your job. As shown in the benchmarks below, different jobs scale differently. Thus, one job which scales very well could be submitted on up to 10 nodes, while another job may scale only up to 2 nodes. For some jobs, if you submit to more nodes than is optimal, your job will actually run slower.

To determine the optimal number of nodes:

• Set up a small version of your job. e.g. if your project involves a simulation of 100 ps, set up a 1ps job.
• Submit this job to 2 processors (1 node), 4processors (2 nodes), 6 processors (3 nodes) ...
• Examine the results. You want to pick the number of processors with at least 70% efficiency.

  
  100 * Time on 1 processor
  ---------------------------    = Efficiency  
  n * Time on n processors
  

• Biowulf job monitors. Click on 'List status of Running Jobs only', and then on your username in the resultant web page. This will pop up a small window showing the status of all nodes that are running your jobs, as in the picture on the right. In the ideal situation, all your nodes should be yellow (both processors used). Watch for green or red nodes. Clicking on a node will pop up a window with information about the processes running on that node.

• jobload. The jobload command will report the loads on the nodes allocated to your jobs.

  biowulf% jobload userA
  Jobs for  userA      Node    Load    
  1708520.biobos      p289     75%
                      p293     74%
                      p297     90%
                      p298     67%
              Job Average:     76%
  1952867.biobos      p308     71%
                      p363     70%
                      p369     69%
                      p377     86%
              Job Average:     74%
  
  User Average:       75%
  

• The PBS batch system will write 2 files called JOBNAME.oJOBNUM and JOBNAME.eJOBNUM in your home directory (e.g. GROMACS.o90763 and GROMACS.e90763). They contain the standard output and standard error from your job. Most problems can be figured out by examining these files.

Summary:

• Different Gromacs jobs scale differently; it is important to run your own benchmarks to determine the optimal number of nodes for your particular job.
• Gromacs scales best on the QDR Infiniband nodes. Based on the d.dppc benchmark results below, one could submit this job with reasonable efficiency to:
  • 128 QDR IB cores (8 QDR IB nodes) using 16 cores on each node.
  • 64 DDR IB cores (8 DDR IB nodes) using all 8 cores on each node.
  • 2 x2800 nodes using 12 cores on each node.

Benchmarks for Gromacs 4.5.3

Benchmarks for Gromacs 4.0.3

Benchmarks for Gromacs 3.3.3.

Benchmarks for Gromacs 3.3.1