Biowulf at the NIH
RSS Feed
CHARMM on Biowulf
Quick Links
Cover scripts
    c38b2
    c37b2
    c35b5
    c32b2
Related Sites
    A tutorial
Model & script building
    CHARMMing
    CHARMM-GUI

CHARMM (Chemistry at HARvard Macromolecular Mechanics) [1]:

  • A versatile and widely used molecular simulation program with broad application to many-particle systems
  • Has been developed with a primary focus on the study of molecules of biological interest, including peptides, proteins, prosthetic groups, small molecule ligands, nucleic acids, lipids, and carbohydrates, as they occur in solution, crystals, and membrane environments
  • Provides a large suite of computational tools that encompass numerous conformational and path sampling methods, free energy estimates, molecular minimization, dynamics, and analysis techniques, and model-building capabilities
  • Useful for a much broader class of many-particle systems
  • Can be utilized with various energy functions and models, from mixed quantum mechanical-molecular mechanical force fields, to all-atom classical potentials with explicit solvent and various boundary conditions, to implicit solvent and membrane models
Major new features:
  • Large performance improvements via both an addition of a domain decomposition scheme (DOMDEC) [2], and an explicit interface to OpenMM
  • New features include replica exchange support (REPDSTR), and routines allowing the exchange of coordinates, energies, and forces with an external program (MSCALE)
  • Fully converted to Fortran95 with some Fortran2003 extensions; greatly improved memory management, now size configurable at run time
[1] CHARMM: The Biomolecular simulation Program J. Comp. Chem. 30, 1545-1615 (2009).
[2] New faster CHARMM molecular dynamics engine J. Comp. Chem. 35, 406-413 (2014).

CHARMM on Biowulf is built and documented by Rick Venable, NHLBI.

In order provide support for multiple executable types using a range of parallel communication methods and hardware via a common interface, cover scripts have been made available in /usr/local/charmm/bin on Biowulf, each named for the CHARMM release version they support. The recommended setup is to use

module load CHARMM

to modify the command search path for your shell (bash or csh). Besides the cover scripts, a tool for extracting data from CHARMM output log files, getprop.csh, is also available. Descriptions of the cover script commands, syntax listings, and usage examples are given in this section. General usage notes for running CHARMM via the PBS queue and using the X11 graphics are given below, after the cover script descriptions.

Latest CHARMM release: c38b2

Recommended version for all applications; includes DOMDEC, and many other new features. (The OpenMM interface is not currently available, but could be easily added if there is sufficient interest.) Typing 'c38b2 -help' on Biowulf gives the following syntax and notes listing:

GenuineIntel x86_64 16 ifort ntl
                                Syntax:

c38b2 -h, -help      # this listing

        SINGLE PROCESSOR; no required args
c38b2 [qchem] [chm-args] < file.inp >& file.out

Notes:
1) Omit "< file.inp >& file.out" for interactive use (e.g. graphics)
2) Square brackets [] indicate optional arguments
3) chm-args   optional CHARMM args; change array sizes, set variables
      e.g.  c38b2 -chsize 150000 RUN=3 < bld-model.inp >& model3.out
4) qchem keyword enables interface to Q-Chem, must be 1st arg; see qchem.doc
5) env var CHM_FORT can be set to ifort or pgi to override default choice


        PARALLEL; minimum 3 args required
c38b2 [opt]  mpi   Nproc  file.inp  [chm-args] >& file.out  # within node
c38b2 [opt]  ib    Nproc  file.inp  [chm-args] >& file.out  # Infiniband
c38b2 [opt]  ipath Nproc  file.inp  [chm-args] >& file.out  # Infinipath

Notes:
1) Parallel arguments are positional, i.e. order dependent
2) mpi       within node; Nproc <= the number of cores in the node
3) ib        select run-time libs for Infiniband based Intel nodes
4) ipath     select run-time libs for InfiniPath based AMD nodes
5) Nproc     integer number of processes, a multiple of 2
6) file.inp  the input file name is *REQUIRED* as the 3rd argument
7) [opt]  optional keyword to override of default COLFFT PME method
          *must* be first; keyword is one of async, domdec, or qchem
          async :: asynchronous PME, needed for REPDSTR, MSCALE, P21 symmetry
          domdec :: use high performance MD code; see domdec.doc
          qchem  :: enables interface to Q-Chem for QM/MM; see qchem.doc
8) env var CHM_FORT can be set to ifort or pgi to override default choice

             http://biowulf.nih.gov/apps/charmm/
Note that typing 'c38b2' with no arguments will start up the program in foreground, making some default assumptions, some of which are reported in the first line printed (see above), along with some hardware and OS information. For parallel usage, 3 arguments are required, in this order: (1) the parallel interface type, (2) the number of processes to start, and (3) the input filename.

Examples:

c38b2 MDL:2 < minmodel.inp >& minmodel.out           # single proc min, arg @MDL
c38b2 mpi 8 minmodel.inp MDL:2 >& minmodel.out       # 8 proc minimization

c38b2 ipath 16 dyn.inp >& dyn.out                    # ipath
c38b2 ib 32 dyn.inp >& dyn.out                       # Infiniband
c38b2 async ib 32 dyn.inp >& dyn.out                 # P21 symmetry, REPDSTR
c38b2 async ib 32 dyn.inp -chsize 450000 >& dyn.out  # 450000 atom limit

c38b2 domdec ib 64 dyn.inp >& dyn.out                # DOMDEC, w. OpenMPI

Recent CHARMM release: c37b2

Previous stable 'beta' version of CHARMM; includes DOMDEC, and many other new features. Typing 'c37b2 -help' on Biowulf gives the following syntax and notes listing:
GenuineIntel x86_64 16 ifort ntl
                                Syntax:

c37b2 -h, -help      # this listing

        SINGLE PROCESSOR; no required args
c37b2  [chm-args] < file.inp >& file.out

Notes:
1) Omit "< file.inp >& file.out" for interactive use (e.g. graphics)
2) Square brackets [] indicate optional arguments
3) chm-args   optional CHARMM args; change array sizes, set variables
      e.g.  c37b2 -chsize 150000 RUN=3 < bld-model.inp >& model3.out
4) env var CHM_FORT can be set to ifort or pgi to override default choice

        PARALLEL; minimum 3 args required
c37b2 [opt]  mpi   Nproc  file.inp  [chm-args] >& file.out  # within node
c37b2 [opt]  ib    Nproc  file.inp  [chm-args] >& file.out  # Infiniband
c37b2 [opt]  ipath Nproc  file.inp  [chm-args] >& file.out  # Infinipath

Notes:
1) Parallel arguments are positional, i.e. order dependent
2) mpi       within node; Nproc <= the number of cores in the node
3) ib        select run-time libs for Infiniband based Intel nodes
4) ipath     select run-time libs for InfiniPath based AMD nodes
5) Nproc     integer number of processes, a multiple of 2
6) file.inp  the input file name is *REQUIRED* as the 3rd argument

7) [opt]  optional keyword to override of default COLFFT PME method
          *must* be first; keyword is either  async  or  domdec
          async :: asynchronous PME, needed for REPDSTR, MSCALE, P21 symmetry
          domdec :: use high performance MD code; see domdec.doc
8) env var CHM_FORT can be set to ifort or pgi to override default choice

           http://biowulf.nih.gov/apps/charmm/


Legacy CHARMM release: c35b5

The c35b5 cover script supports the August 2010 release of CHARMM; new features include the addition of the MCMA peptide sampling method, and capability to use the CHARMM General Force Field (CGenFF). Typing 'c35b5 -help' on Biowulf gives the following syntax and notes listing:

GenuineIntel x86_64 16 ifort ntl xxl
                                Syntax:

c35b5 -h, -help      # this listing

        SINGLE PROCESSOR; no required args
c35b5  [chm-args] < file.inp >& file.out

Notes:
1) Omit "< file.inp >& file.out" for interactive use (e.g. graphics)
2) Square brackets [] indicate optional arguments
3) chm-args  CHARMM script @ vars, in the form N:27 or RUN=15, etc.


        PARALLEL; minimum 3 args required
c35b5  mpi   Nproc  file.inp  [chm-args] >& file.out  # within node
c35b5  ib    Nproc  file.inp  [chm-args] >& file.out  # Infiniband
c35b5  ipath Nproc  file.inp  [chm-args] >& file.out  # Infinipath

Notes:
1) Parallel arguments are positional, i.e. order dependent
2) mpi       within node; Nproc <= the number of cores in the node
3) ib        select run-time libs for Infiniband based Intel nodes
4) ipath     select run-time libs for InfiniPath based AMD nodes
5) Nproc     integer number of processes, usually 2, 4, 8, 16, 32, or 64
6) file.inp  the input file name is *REQUIRED* as the 3rd argument

         http://biowulf.nih.gov/apps/charmm
Note that typing 'c35b5' with no arguments will start up the program in foreground, making some default assumptions, some of which are reported in the first line printed (see above), along with some hardware and OS information. For parallel usage, 3 arguments are required, in this order: (1) the parallel interface type, (2) the number of processes to start, and (3) the input filename.


c35b5 usage examples

The following series of examples illustrates the usage; first, some foreground or direct Unix shell usage (use 'qsub -V -I -l nodes=1' for an interactive node):

c35b5
Start default single proc version in foreground, using standard I/O for interactive use, e.g. exploratory analysis or graphics; the use of stream files for setup and other often used operations is highly recommended. The procedure to run the X11 graphics is given below.

c35b5 FST:2 LST:21 < file.inp >& file.out &
Run a single-threaded (non-parallel) calc in a Unix shell; the final '&' char will place the job in the background. Note the argument passing; @FST will be '2' when the script is run.

c35b5 mpi 4 file.inp >& file.out &
Run an OpenMPI parallel job in a shell on Biowulf dual core (dc) machines

c35b5 ib 16 file.inp >& file.out
Run a 16 processor job on the InfininBand (ib) parallel transport; the above command would be in a batch script (more below) submitted via:
qsub -l nodes=2:ib batch_script.csh

c35b5 ipath 16 file.inp >& file.out
Run a 16 processor job on the InfininPath (ipath) parallel transport. Analogous to InfiniBand, the above command would be in a batch script submitted via:
qsub -l nodes=8:ipath batch_script.csh

Legacy support for c32b2

Support for the legacy c32b2 release has been maintained specifically to read trajectory files created with c32b2 on the initial InfiniPath installation on Biowulf. Because of limitations in that particular CHARMM release related to the PathScale compiler, the 64-bit executable built with the PathScale compiler is currently needed to read those files (other workarounds may be possible). The files can be converted to a more widely readable form via:
  • c32b2 path N:22 < fmt.inp > fmt.out
    create a platform and compiler independent text-based file via DYNA FORMAT (see dynamc.doc)
  • c32b2 N:22 < unfmt.inp > unfmt.out
    convert transfer format back to a binary data file via DYNA UNFORMAT
The CHARMM input for these scripts is relatively straightforward; the following fragments assume that the standard startup tasks such as reading parameters and molecular information for the simulated system, including periodic boundary conditions (if used) has already been performed. First, an outline for fmt.inp:
! open files; var N passed as command line parameter
open unit 10 read file name dyn@N.trj
open unit 11 write card name dyn@N.zfmt
dyna format firstu 10 nunit 1 output 11
This produces the special transfer format file, which is text-based and has been used for many years to exchange data between machines with very different ways of representing floating point numbers in binary form. The second step converts the transfer format back to a binary form that is usually different from the original, via unfmt.inp:
! open files; var N passed as command line parameter
open unit 2 read card name dyn@N.zfmt
open unit 12 write file name ndyn@N.trj
dyna unformat firstu 2 nunit 1 output 12

Typing 'c32b2 -help' on Biowulf gives the following syntax and notes listing:
                                Syntax:
        SINGLE PROCESSOR; no required args
c32b2 [i386] [path] [chm-args] < file.inp >& file.out

c32b2 -h, -help      # this listing

                                Notes:
0) Arguments are positional, i.e. order dependent
1) Square brackets [] indicate optional arguments
2) Omit "< file.inp >& file.out" for interactive use (e.g. graphics)
3) i386   MUST be first arg; forces i386 (32-bit) for PathScale
4) path   force PathScale (default 64-bit); default compiler is pgi (32-bit only)
5) chm-args  CHARMM script @ vars, in the form N:27 or RUN=15, etc.

         http://biowulf.nih.gov/apps/charmm

Simple PBS batch script example

A minimal batch_script.csh is outlined below, useful for most simple uses of CHARMM; (for extended simulations, see the daisy chaining description below):
#!/bin/csh
#PBS -N Minimiz
#PBS -j oe
cd $PBS_O_WORKDIR
c38b2 ...

Using the X11 graphics

The CHARMM X11 graphics commands are avaiable in the single-threaded (non-parallel) executables; a pre-requisite is to login to biowulf using the ssh command with X11 forwarding enabled. From some systems, the -Y option to ssh is needed to setup use of X11, e.g.

ssh -Y biowulf

The next step is to request an "interactive batch" node, via

qsub -l nodes=1 -I -V

Finally, start CHARMM via simply

c38b2

Given the command line nature of the graphics (all typing, no mouse), it's useful to have a few CHARMM stream files that do all the system setup, e.g. read in RTF, PARAM, PSF and COOR file(s). Then one can simply type

* a title
*
stream init-prot.str
graphx
draw sele .not. type H* end
center
scale 0.5
... (etc.)

Graphics features include trajectory file viewing, and output of coordinate data as a file of POV-Ray objects.


Syntax highlighted editing of CHARMM input files

For users of emacs and vim text editors, there are configuration files available in the Script Archive of the CHARMM forums which support color based highlighting when editing files with .inp and .str extensions.

Locations of executable programs and support files

Academic CHARMM executables with different features and sizes are available via /usr/local/charmm, as well as the doc, toppar, test, and support subdirs for each of the supported releases. The c38b2/toppar subdir includes the recent C36 update of all CHARMM parameter sets, with revised parameters, esp. for proteins, lipids, and CGenFF, as well as being recast in FLEX format. The FLEX format allows combining parameter sets directly, w/o the need to combine the files.

Here is a more detailed usage example for c38b2 illustrating the use of paired scripts to keep a simulation running for an extended period w/o intervention. Several points need to made about these scripts:

  • It assumes the use of generic output file names, e.g. dyn.trj dyn.out
  • The initiating script (which calls 'qsub') may be renamed to break the chain gracefully at the end of a run
  • Almost all job failures will break the chain; one known (rare) exception, which writes a bad restart file, and causes the next job to fail and break the chain
  • Killing a running job (via 'qdel') will also break the chain
  • A sequence number in last.seqno which is greater than or equal to the one in next.seqno will gracefully break the chain

Samples and variations of the csh and CHARMM scripts discussed below can also be found
in the /usr/local/charmm/scripts subdir.

First, there's the initiating script, used to start the process, which we'll call biowulf.csh:


#!/bin/csh 
qsub -l nodes=4:ib dyn.csh

More important (and more complicated) is the dyn.csh script, which runs CHARMM, checks the output files to test for a successful run, renames each output file by adding a sequence number, and submits the next run to the queue. For a failed run, the output file is renamed to dyn.err.TS, where TS is a timestamp. This script has been extensively tested and used on many systems with appropriate modifcations, including LoBoS, Galaxy, and Biowulf.

#! /bin/csh -f
#PBS -N RunDyn
#PBS -j oe -r n
# ASSUMPTION (1): input files are named   dynXX.inp   where XX is optional
#  useful for startup phase; for example, to start using dynstrt.inp,
#    dyn.csh strt
#  would be used; for production dynamics runs, simply dyn.csh
# ASSUMPTION (2): output files are named    dyn.res  dyn.trj  dyn.out
# ASSUMPTION (3): previous restart file read as  dyn.rea
#

cd $PBS_O_WORKDIR

set chm "c38b2 ib 32"
if ( -e next.seqno ) then
 $chm < dyn.inp >& dyn.out
else
 $chm < dynstrt.inp >& dyn.out
endif

set okay = true
# TEST FOR EXISTENCE, THEN NONZERO LENGTH OF OUTPUT FILES
if ( -e dyn.res && -e dyn.trj ) then
 @ res = `wc dyn.res | awk '{print $1}'`
 @ tsz = `ls -s dyn.trj | awk '{print $1}'`
 @ nrm = `grep ' NORMAL TERMINATION ' dyn.out | wc -l`
 if ( $res > 100 && $tsz > 0 && $nrm == 1 ) then
# SUCCESSFUL RUN; COPY RESTART FILE
  cp dyn.res dyn.rea
# DETERMINE RUN NUMBER
  if ( -e next.seqno ) then
   @ i = `cat next.seqno` 
  else
   @ i = 1
  endif
# ENABLE/DISABLE ANALYSIS EVERY 2ND RUN
#  if ( 0 == $i % 2 ) qsub -l nodes=1 chk.csh
# NUMBER THE OUTPUT FILES; CHANGE EXTENSIONS TO SUIT APPLICATION
  foreach fil ( out res trj )
   mv dyn.$fil dyn$i.$fil
  end
  gzip dyn$i.out dyn$i.res
# CONDITIONAL END CHECK
  if ( -e last.seqno ) then
    @ l = `cat last.seqno`
    if ( $i == $l ) goto endrun
  endif
# SUBMIT THE NEXT JOB
  ./biowulf.csh
  endrun:
  @ i += 1
  echo $i > next.seqno
 else
# ZERO LENGTH FILE(S)
  set okay = false
 endif
else
# FILE DOESN'T EXIST
 set okay = false
endif

# TEST FOR CHARMM RUN FAILED; CREATE .ERR FILE WITH TIMESTAMP
if ( $okay == true ) then
else
 set ts = `date +%m%d.%H%M`
 mv dyn.out dyn.err.$ts
 exit(201)
endif

To complete the setup, the CHARMM input files need to use the generic names; first, there is dynstrt.inp used only for the first run:

* start NPAT melittin in dopc bilayer sim at high hydration
*
bomlev -1

stream inim2p1.str

! SETUP SHAKE
shake fast bonh param

open unit 41 write file name dyn.trj
open unit 51 write card name dyn.res

prnlev 3 NODE 0

dyna cpt start nstep 500000 timestep 0.001  -
    pcons pint pref 1.0 pmzz 2500. pmxx 0.0 pmyy 0.0 -
    hoover reft 303.15 tmass 20000. -
    inbfrq -1 atom vatom cutnb 15.0 ctofnb 12. cdie eps 1. -
    ctonnb 10. vswitch cutim 15.0 imgfrq -1 wmin 1.0 bycb -
    ewald pmew fftx 48 ffty 48 fftz 64  kappa .34 spline order 6 -
    iprfrq 5000 ihtfrq 0 ieqfrq 0 ntrfrq 1000 -
    iuncrd 41 iunrea -1 iunwri 51 kunit -1 -
    nprint 1000 nsavc 1000 nsavv 0 ihbfrq 0 ilbfrq 0 -
    firstt 303.15 finalt 303.15 teminc 10.0 tstruct 303.15 tbath 303.15 -
    iasors 1 iasvel 1 iscvel 0 ichecw 0 twindh 5.0 twindl -5.0

stop

Finally, there is the truly generic dyn.inp file, which is run repeatedly until the desired simulation time duration is achieved:

* melittin in dopc lipid bilayer NPAT at high hydration
*
bomlev -1

stream inim2p1.str

! SETUP SHAKE
shake fast bonh param

open unit 31 read  card name dyn.rea
open unit 41 write file name dyn.trj
open unit 51 write card name dyn.res
prnlev 3 node 0
dyna cpt restart nstep 500000 timestep 0.001  -
    pcons pint pref 1.0 pmzz 2500.0 pmxx 0.0 pmyy 0.0 -
    inbfrq -1 atom vatom cutnb 15. ctofnb 12. cdie eps 1. -
    ctonnb 10. vswitch cutim 15.0 imgfrq -1 wmin 1.0 -
    ewald pmew fftx 48 ffty 48 fftz 64  kappa .34 spline order 6 -
    hoover reft 303.15 tmass 20000. -
    iprfrq 5000 ihtfrq 0 ieqfrq 0 ntrfrq 1000 -
    iuncrd 41 iunrea 31 iunwri 51 kunit -1 -
    nprint 100 nsavc 500 nsavv 0 ihbfrq 0 ilbfrq 0 -
    firstt 303.15 finalt 303.15 teminc 10.0 tstruct 303.15 -
    iasors 1 iasvel 1 iscvel 0 ichecw 0 twindh 5.0 twindl -5.0

stop

A key thing to note is that dyn.rea is always the restart file created from the preceding run (see dyn.csh). An added feature is the use of gzip to compress output and restart files, saving considerable disk space. Compressed files may be browsed easily via e.g.

zcat dyn2.out.gz | less


This document prepared by Rick Venable, NHLBI/BBC Lab of Computational Biology

March 2014 (Rv)