NAMD on Biowulf

Features of NAMD

NAMD User Guide (also available in PDF -- 541KB)

Overview of NAMD and Molecular Dynamics (in PDF)

NAMD was developed to be compatible with existing molecular dynamics packages, especially the packages X-PLOR and CHARMM, so it will accept X-PLOR and CHARMM input files. The output files produced by NAMD are also compatible with X-PLOR and CHARMM.

Running NAMD on Biowulf

• a PSF structure file which defines the structure. This can be generated in X-PLOR, CHARMM, or by using the psfgen program in NAMD.
• a PDB coordinate file, defining the XYZ coordinates of every atom in the structure.
• a parameter file, in CHARMM or X-PLOR format.
• a NAMD configuration file which specifies the number of timesteps, temperature, and other options to the NAMD program.

Create a batch command file:

-----------------------------------------------------------------------------
Sample batch script -- /data/username/namd/namd_run
-----------------------------------------------------------------------------
#!/bin/csh
#PBS -N NAMD
#PBS -k oe
#PBS -m be
#
set path=(/usr/local/namd/ $path .)
cd $PBS_O_WORKDIR

set arch=`uname -m`
make-namd-nodelist                                     #required!!!
charmrun.$arch /usr/local/namd/namd2.$arch +p$np ++nodelist ~/namd.$PBS_JOBID bpti.inp >& bpti.out
rm ~/namd.$PBS_JOBID
-----------------------------------------------------------------------------

Submit this job with the command:

qsub -v np=16 -l nodes=8:p2800:gige /data/username/namd/namd_run

qsub -v np=16 -l nodes=8:o2800 /data/username/namd/namd_run

NAMD on Myrinet

NAMD on Infiniband

% cd ~/.ssh
% cp /usr/local/etc/ssh_config_ib config
% chmod 600 config

A version of NAMD has been built with the Pathscale compilers for Infiniband. An NAMD job with this version will require a slightly different job submission script, as below:

-----------------------------------------------------------------------------
Sample batch script -- /data/username/namd/namd_run.ib
-----------------------------------------------------------------------------
#!/bin/csh
#PBS -N myjob
#PBS -k oe
#PBS -m be
#
set path=(/usr/local/namd-ib $path .)
setenv LD_LIBRARY_PATH /usr/local/namd-ib/

cd $PBS_O_WORKDIR

charmrun /usr/local/namd-ib/namd2 +p$np myjob.conf >& myjob.log
-----------------------------------------------------------------------------

qsub -v np=16 -l nodes=8:ib namd_run.ib

Considerations:

1. Choosing the number of processors/nodes: When submitting a job with the qsub command, the number of processors (np=#) should be twice the number of nodes (-l nodes=#), so that the job uses both processors of each node. It is recommended that you use 2 processors per node unless the job requires more than half the the memory available on that node. The Benchmarks section of this page may help you to choose the appropriate nodes and number of processors.

2. Nodelist:. NAMD requires a list of nodes to run, hence the batch command file must include the 'make-namd-nodelist' utility, which is in the directory /usr/local/bin/. (This step is not required for Infiniband jobs). The utility creates a file in your home directory called namd.PBS_JOBID, which is used when running the NAMD job. This file can be deleted at the end of the job, as in the batch scripts above.

Replica Exchange

1. Set up a replica exchange configuration file along the lines of the one below. The sections that you will need to modify are in italics.

   ----------------- file fold_alanin.conf------------------------------------
   # configuration for replica exchange scripts
   # run simulation: tclsh ../replica_exchange.tcl fold_alanin.conf
   # to continue: tclsh ../replica_exchange.tcl restart_1.conf
   # view in VMD:  source fold_alanin.conf; source ../show_replicas.tcl
   # add continued:   source restart_1.conf; source ../show_replicas.tcl
   # show both:  vmd -e load_all.vmd
   
   set num_replicas 8
   set min_temp 300
   set max_temp 600
   set steps_per_run 1000
   set num_runs 10000
   # num_runs should be divisible by runs_per_frame * frames_per_restart
   set runs_per_frame 10
   set frames_per_restart 10
   set namd_config_file "alanin_base.namd"
   set output_root "output/fold_alanin" ; # directory must exist
   
   # the following used only by show_replicas.vmd
   set psf_file "alanin.psf"
   set initial_pdb_file "unfolded.pdb"
   set fit_pdb_file "alanin.pdb"
   
   set namd_bin_dir /usr/local/NAMD-2.6-Linux-amd64-TCP-icc
   set server_port 3177
   
   # NOTE:  Running namd2 through charmrun interferes with socket connections;
   # run the namd2 binary directly (in standalone mode).  MPI might work.
   
    set spawn_namd_command \
      [list spawn_namd_rsh "cd [pwd]; [file join $namd_bin_dir namd2] +netpoll" \
      [read [open $env(PBS_NODEFILE) "r"]] ]
   --------------------------------------------------------------------
   

2. Set up a batch script:

   ---------file re.bat-------------------------
   #!/bin/csh
   #PBS -N NAMD_RE
   #PBS -j oe
   #PBS -m be
   
   cd /data/user/namd/re
   tclsh /usr/local/namd/replica/replica_exchange.tcl fold_alanin.conf
   -----------------------------------------------
   

3. Check that the output directory exists, otherwise the job will die. In this example, the output directory is /data/user/namd/re.

4. Note the number of replicas in the config file. In the example above, in Step 1, there are 8 replicas. The number of nodes requested should be half of the number of replicas. Submit the job:

   qsub -l nodes=4:x86-64 re.bat
   

Benchmarks

Water sphere and water box simulations courtesy Jeff Forbes, NIAMS
In the two jobs above, note that the water sphere simulation scales well to about 36 processors. Beyond that, the efficiency falls to below 60% which is generally considered poor efficiency. The water box simulation scales to about 16 processors before the efficiency drops to below 60%. This is similar to the apoa1 example above. (We now recommend that jobs run at least at 70% efficiency)

Equilibrated model of integral membrance complex, courtesy Nara Dashdorj, LCP, NIDDK.
A total of 346,358 atoms including water, lipids, and protein with several prosthetic groups. Cutoff 12.0, fullElectFrequency 4, nonbondedFreq 2, stepspercycle 20. In these benchmarks, the job scales to about 60 processors (30 nodes) on the p2800 Xeons, and to about 50 processors (25 nodes) on the o2200 Opterons. This is typical behaviour; with higher processor speeds the communications become more of a bottleneck, and so the job does not scale as well on the faster processors. (We now recommend that jobs run at least at 70% efficiency)

Users who wish to run NAMD jobs on large numbers of processors should run their own benchmarks and verify that their particular job is at least 60% efficient on the desired number of nodes, as in the examples above.

             100 * Time on 1 processor
Efficiency = -------------------------
              n * Time on n processors

Detailed sample session:

1. Obtain the PDB file 6pti.pdb (you can copy it from /pdb/pdb/pt/pdb6pti.ent.gz and uncompress. The entire Protein Data Bank, updated nightly, is mirrored on the Helix Systems). Also obtain the appropriate topology and parameter files. These are available in /usr/local/charmm/c31b1/toppar/. A collection is available at http://www.pharmacy.umaryland.edu/faculty/amackere/force_fields.htm.
2. Run psfgen to create the PSF file.

   $ cd /data/username/namd
   $ gunzip -c /pdb/pdb/pt/pdb6pti.ent.gz > 6pti.pdb
   $ grep -v '^HETATM' 6pti.pdb > 6pti_protein.pdb
   $ grep 'HOH' 6pti.pdb > 6pti_water.pdb
   $ cat > psfgen.inp << END
   topology /usr/local/charmm/c31b1/toppar/top_all22_prot.inp
   segment BPTI {
     pdb 6pti_protein.pdb
   }
   patch DISU BPTI:5 BPTI:55
   patch DISU BPTI:14 BPTI:38
   patch DISU BPTI:30 BPTI:51
   alias atom ILE CD1 CD
   coordpdb 6pti_protein.pdb BPTI
   alias residue HOH TIP3
   segment SOLV {
     auto none
     pdb 6pti_water.pdb
   }
   alias atom HOH O OH2
   coordpdb 6pti_water.pdb SOLV
   writepsf bpti.psf
   guesscoord
   writepdb bpti.pdb
   END
   $ /usr/local/namd/psfgen.x86_64 < psfgen.inp
   

3. Create the NAMD configuration file -- the BPTI example is provided in the NAMD user guide.
4. Create the batch command file, in this case called /data/username/namd/bpti.run

   ------------------------------------------------
   #!/bin/csh
   #PBS -N NAMD
   #PBS -k oe
   #PBS -m be
   #
   set path=(/usr/local/namd/ $path .)
   cd $PBS_O_WORKDIR
   set arch=`uname -m`
   
   make-namd-nodelist
   charmrun.${arch} /usr/local/namd/namd2.${arch} +p$np ++nodelist ~/namd.$PBS_JOBID bpti.inp >& bpti.out
   rm ~/namd.$PBS_JOBID
   ------------------------------------------------
   

5. Decide on the number of processors, and submit the job with the command

   % qsub -v np=4 -l nodes=2:o2800 /data/username/namd/bpti.run