DMOL on Biowulf

Status
About
Hardware
Applications
Batch queues
Disk storage

MPI
Performance
New Users
User Guide
Documentation
Research
Photos

DMOL on Biowulf

DMol³ is a Density Functional Theory (DFT) quantum mechanical code that allows users to study problems in chemistry and materials science with high accuracy and reliability. It is developed by Accelrys and can be used to simulate processes in gas phase, solvent, surface and solid environments.

DMOL v3.2 is the default. Users can choose to run v 3.1 for comparison or to complete ongoing projects by following the instructions further down this page.

[Setting up your environment] [Submitting a parallel job]

Setting up your environment

DMOL requires some libraries and environment variables, and may fail if they are not found. Users should therefore modify their /home/user/.cshrc or /home/user/.bashrc files to add the following lines:

For bash shell users:

eval `/usr/local/msi/MaterialsStudio/Licensing/Setup/lic_setup.sh -s sh`
eval `/usr/local/msi/MaterialsStudio/share/bin/ms_setup.sh  -s sh`
DMOL3_DATA=/usr/local/msi/MaterialsStudio/Data/Resources/Quantum/DMol3; export DMOL3_DATA
DMOL_TMP=/scratch; export DMOL_TMP

For csh or tcsh users:

eval `/usr/local/msi/MaterialsStudio/Licensing/Setup/lic_setup.sh -s csh`
eval `/usr/local/msi/MaterialsStudio/share/bin/ms_setup.sh -s csh`
setenv DMOL3_DATA /usr/local/msi/MaterialsStudio/Data/Resources/Quantum/DMol3
setenv DMOL_TMP /scratch

These commands will run the default version of DMOL, v 3.2 Users who wish to specifically run v 3.1 should replace MaterialsStudio with MaterialsStudio3.1 in the lines above.

Submitting a parallel job on the command line.

This example assumes that the user's environment has been set up as above, and that the required DMOL input files have been previously created from Cerius, Visualizer, or another program.

Create a simple batch command file along the following lines:

#!/bin/bash # This file is dmol_run.sh # #PBS -N DMOL_water #PBS -m be #PBS -k oe cd /data/username/dmol /usr/local/msi/MaterialsStudio/DMol3/bin/RunDMol3.sh -np $np water

(replace MaterialsStudio with MaterialsStudio3.1 or MaterialsStudio3.2 to run a specific version, if desired.)

This command file would be submitted with:

qsub -v np=4 -l nodes=2,dmol=4 dmol_run.sh

This job will use 4 processors (np = 4) on 2 nodes (-l nodes=2). In our experience DMOL runs do not scale to more than 4 processors (2 nodes). Users who run jobs on more than 2 nodes should justify this decision by running their own benchmarks. (We would be very interested in seeing the results of DMOL jobs that scale better than 4p; please email staff@helix.nih.gov).

Note: DMOL resources

The PBS Batch system on the Biowulf cluster keeps track of the DMOL licenses via the DMOL "resources". Jobs requiring a DMOL license will be started or remain in the queue depending on the availability of DMOL licenses. All DMOL jobs must therefore specify the appropriate resource when the job is submitted. For example,

qsub -v np=4 -l nodes=2,dmol=4 my_dmol_script

specifies that the job requires 4 dmol licenses, one for each processor. If a job appears to be queued for a long time when there are nodes available, you can check its status via:

qstat -f job_number

The output from this qstat command should show the dmol resource required.

[susanc@biobos test]$ qstat -f 64402.biobos
Job Id: 64402.biobos
    Job_Name = DMOL_CHA
    [...]
    Resource_List.dmol = 4
    Resource_List.ncpus = 2
    [...]
    comment = Not Running: Waiting for dmol resource

The 'comment' line near the end of the qstat output will explain why the job is queued. In this case the batch system is waiting for DMOL licenses to become available before starting the job.

Submitting jobs to the Altix

firebolt.nih.gov, the Altix, is considered a "fat node" on the Biowulf cluster, and jobs are submitted via the PBS v7.1. To specify that a DMOL job should run on the Altix, use altix as the resource, and specify the number of processors with ncpus=#.

qsub -v np=6  -l nodes=1:altix:ncpus=6,dmol=6,mem=6gb /data/username/large/dmol.run.altix

np=6 is an environment variable passed to your batch script.
ncpus=6 tells PBS how many Altix processors to allocate to this job.
dmol=6 tells PBS how many 'dmol' resources (= DMOL licenses) are required by this job.
mem=6gb tells PBS that 6GB of memory is required for the job.

This document is available as http://biowulf.nih.gov/apps/dmol.html
Biowulf home page | Helix Systems | NIH