Biowulf at the NIH
RSS Feed
Amber on Biowulf

AMBER (Assisted Model Building with Energy Refinement) is a package of molecular simulation programs. AMBER contains a large number of of modules; note that only the sander modules and pmemd are parallelized.

The following builds of Amber are available on Biowulf:

Compiler Can be used on Environment Module
(June 2014)
Intel All Biowulf computational nodes Gige: amber/14/gige
IB: amber/14/ib
Ipath: amber/14/ipath
(Aug 2012)
Intel All Biowulf computational nodes. However, sander does not scale to more than 1 IB node, so we recommend running on Ipath or Gige nodes. (see benchmarks) Gige: amber/12/gige
Ipath: amber/12/ipath
IB: amber/12/ib
GPU: amber/12/gpu
(Aug 2010)
Intel All Biowulf computational nodes. However, sander does not scale to more than 1 IB node, so we recommend running on Ipath or Gige nodes. (see benchmarks) Gige: amber/11/gige
Ipath: amber/11/ipath
IB: amber/11/ib
11 with Lennard-Jones 10_12 potentials Intel
compiled with
and AmberTools 1.5
All Biowulf computational nodes. Executable is called sander.MPI.10_12 Gige: amber/11/gige
Ipath: amber/11/ipath
IB: amber/11/ib
(May 2008)
Pathscale gige nodes Gige: amber/10/gige
Ipath: amber/10/ipath
IB: amber/10/ib

LEaP is a graphical builder of input files for AMBER modules. LEaP can be used via the Xwindows graphical interface xleap, or the terminal version tleap. To run xleap,

  1. Open an Xwindows session to Biowulf. (More information about Xwindows on Macs, Windows, and Unix desktop machines.)
  2. On Biowulf, load the module for the version you want, and then type 'xleap'.
    biowulf% module load amber/14/gige
    biowulf% xleap
    You should see the xleap window appear, in which you can type any LEaP commands.
  3. The AMBER tutorials

For basic information about setting up an Amber job, see the Amber manual and the Amber tutorials . Also see Batch Queuing System in the Biowulf user guide.

The Amber executables can run in parallel on all Biowulf computational nodes. However, benchmark runs showed that sander does not scale beyond a single IB node. Amber 12 has the best performance/scaling on the ipath nodes. (see benchmarks). Therefore we recommend that users run on the ipath or gige nodes.

Sample script for a Sander run on ipath nodes. The same script can be used for any Amber version and any network (Gige, IB or Ipath) by changing only the module that is loaded.

# This file is
#PBS -N sander
#PBS -m be
#PBS -k oe

module load amber/14/ipath

cd /data/user/amber/myproject
`which mpirun` -machinefile $PBS_NODEFILE -np $np `which sander.MPI` \
   -O -inf mdinfo -r md.rst -x md.crd -e md.en -p -o md.out -i

Submit with:
qsub -v np=16 -l nodes=8:ipath jobscript
This job would be run on 16 cores on 8 ipath nodes.

For a pmemd run, the executable sander.MPI in the last line would be replaced by pmemd.MPI with appropriate parameter changes.

Replica Exchange jobs can reasonably be run on the gige nodes since the inter-node communication are less of an issue, and the jobs will scale well.

NOTE: this refers to Amber 10 only. See the Amber 12 documentation for replica exchange with Amber 12.

First read the tutorial on Replica Exchange simulations with Amber 10. Once the topology and coordinate files are generated, the input structure minimized, and the number of replicas and temperatures determined, the equilibration and the REMD simulations can be run as batch jobs. A detailed explanation of these steps including sample input files is in the tutorial. It is possible to run each replica on multiple processors as in the example below.

Set up a batch script along the following lines:

#  this file is called amber_re.bat

export PATH=/usr/local/mpich/bin:$PATH
export AMBERHOME=/usr/local/amber11

cd /data/myname/amber/re
mpirun -machinefile $PBS_NODEFILE -np $np $AMBERHOME/exe/sander.MPI -ng 8 \
                                       -groupfile equilibrate.groupfile

Submit this job with

qsub -v np=8 -l nodes=4:o2800 amber_re.bat       (to use 8 processors on 4 o2800 nodes).


qsub -v np=16 -l nodes=8:o2800 amber_re.bat      (to use 16 processors on 8 o2800 nodes.)
                                               (Each replica will run on 2 processors)


qsub -v np=32 -l nodes=8:dc amber_re.bat       (to use 32 processors on 8 dual-core nodes)
                                               (Each replica will run on 4 processors)

As with all batch jobs, you will need to monitor the job to ensure that the load is reasonable.

In Dec 2014, the Biowulf cluster will implement walltime limits for IB jobs. (Use 'batchlim' to see the current walltime limits). Since there are a limited number of IB nodes and heavy demand, this change is intended to increase turnover of jobs on those nodes. Thus, jobs should be designed to run for a week or so, save a checkpoint file, and submit a new job starting from that checkpoint.

An example batch script is below. This script runs a single simulation, saves a copy of the output files, and then resubmits a new job starting from Amber's 'restart' file.

#PBS -N sander
#PBS -m be
#PBS -j oe

module load  amber/14/ib
module list

echo "Using $np processors, run num is $run"


# rename the restart file to the coordinate filename
mv restrt inpcrd

#run sander
`which mpirun` -machinefile $PBS_NODEFILE -np $np `which sander.MPI` -O -i mdin -c inpcrd -p prmtop -r restrt -x traj -e ene -o mdout

#keep a copy of the output from this run
 mv mdout  md_$run.out
 mv traj  md_$run.trj
 mv ene  md_$run.ene
 cp restrt  md_$run.rst

# if less than 10 runs have been performed, increase the run number and submit the next job
if (( "$run" < "10" ))
     run=`expr $run + 1`
     qsub -v np=16,run=$run -l nodes=2:ib ./Run.ib

To submit this job, copy the original input coordinate file to 'restrt' for the first run, and then submit.

cp inpcrd restrt               
qsub -v np=16,run=1 -l nodes=2:ib Run.ib

The overall objective of the MM-PBSA method and it's complementary MM-GBSA method is to calculate the free energy difference between two states which most often represent the bound and unbound state of two solvated molecules or alternatively to compare the free energy of two different solvated conformations of the same molecule. MM-PBSA tutorial.

Note: This sample script is for Amber 11. See the Amber 12 documentation for details about MMPBSA on Amber 12.

Sample batch script for a parallel MM-PBSA run. Courtesy of Tae Jin Kim (NCI).

# This file is
#PBS -m be

export AMBERHOME=/usr/local/amber11

cd /data/user/MMPBSA/

/usr/local/openmpi/bin/mpirun -machinefile $PBS_NODEFILE -np $np $AMBERHOME/bin/MMPBSA.MPI -O -i -o mmpbsa.dat -sp 1err.solvated.prmtop -cp complex.prmtop -rp receptor.prmtop -lp ligand.prmtop -y 1err_prod.mdcrd > progress.log
Submit this job with
qsub -v np=24 -l nodes=1:c24

Memory usage: The required memory for normal mode calculation is { 9*N*(3*N-1)/2 } *8 byte, where N is total atom number. (from the Amber 9 manual, p 280) Each process in a parallel MM-PBSA job loads each snapshot. Therefore the total memory used by the job will be { 9*N*(3*N-1)/2 } *8 * number_of_processes Users should adjust the number of processes and submit to an appropriate node so that the entire job fits in memory. For example, if the system has 5000 atoms, the memory required for a single process is 2.7 GB. 8 of these processes (21.6 GB total) will fit on a 24GB node, so the job should be submitted with

qsub -v np=8 -l nodes=1:g24

If you are new to Amber, you may want to work through some of the tutorials on the Amber tutorial page. Note that the entire Protein Data Bank is available on Biowulf in /pdb and is updated weekly. Some of these tutorials have been worked through by the Biowulf staff, and the input, output and batch scripts specific to Biowulf are in /usr/local/amber11/tutorials/. If you want to examine these files, you could do something like:

cd /scratch/user
tar xvzf /usr/local/amber11/tutorials/tutorial-a6.tar.gz
cd tutorial-a6
more section4.biowulf

The Biowulf staff benchmarks indicate that sander jobs should be run on a single gige node or up to 8 ipath nodes. If you submit a parallel Amber job to multiple gige nodes, more than 8 ipath nodes, or multiple IB nodes, you should run your own benchmarks to determine the appropriate number of nodes.

To determine the optimal number of nodes:

Example from the Factor IX benchmark : on the e2666 nodes the job dropped to 45% efficiency when using all cores on a single node (16 cores). Thus, it is not reasonable to run this job on more than 1 gige node. On the ipath nodes, the efficiency was 72% at 16 cores. Thus, the job could be run on 8 nodes (16 cores) at reasonable efficiency.

eric You can use the Biowulf system monitors to watch your job. Click on 'List status of Running Jobs only', and then on your username in the resultant web page. This will pop up a small window showing the status of all nodes that are running your jobs, as in the picture on the right. In the ideal situation, all your nodes should be yellow (both processors used). Watch for green or red nodes. Clicking on a node will pop up a window with information about the processes running on that node.

The PBS batch system will write 2 files called JOBNAME.oJOBNUM and JOBNAME.eJOBNUM in your home directory (e.g. sander.o90763 and sander.e90763). They contain the standard output and standard error from your job. Most problems can be figured out by examining these files.

Amber 14: Factor IX benchmark from the Amber benchmark suite. [Details] (June 2014)