When do you need to use Biowulf?

• Long jobs (e.g. molecular dynamics)
• Many jobs (e.g. bioinformatics with 50+ sequences)
• Large-memory jobs (e.g. Gaussian requiring 8GB memory)
• Parallel jobs (e.g. NAMD on 16 processors)

What jobs are unsuitable for Biowulf?

• "Serial" jobs.
• Small numbers of short jobs.
• Interactive jobs

Accounts & Passwords

Connecting to Biowulf

Storage & Backups

Snapshots. (see http://helix.nih.gov/new_users/backups.html)

Running Programs on Biowulf

#!/bin/tcsh
#
# this file is myjob.sh
#
#PBS -N MyJob
#PBS -m be
#PBS -k oe
#
myprog -a 100 < /data/me/mydata

qsub -l nodes=1 myjob.sh

Notes:

Monitoring Batch Jobs

#!/bin/bash
#
# this file is myjob.sh
#
#PBS -N MyJob
#PBS -m be
#PBS -k oe
#
myprog -a 100 < infile1 > outfile1 &
myprog -a 200 < infile2 > outfile2 &
wait

Notes:

Interactive Batch

qsub -I -l nodes=1

Installed Applications

Demo: easyblast

Biowulf Hardware Configuration (2100 nodes, 6300 processors)

Fileservers		Network Switches
NetApp FAS960c (2 Filers) NetApp 3050c (4 Filers) NetApp R200 NearStore DDN Gridscaler (8 fileservers)		Foundry MG-8 (core) 10 GbE & GbE Foundry EdgeSwitch FES448 GbE Foundry FastIron 100Base-T & GbE Myricom Clos-128 Myrinet 2000 Voltaire ISR 9288 Infiniband Voltaire ISR 2012 Infiniband

Node configurations

Large Memory Nodes

Jobs on the Biowulf Cluster

small numbers of serial jobs (~1-4)

swarms of serial jobs

#!/bin/bash
#
# this file is myjob.sh
#
#PBS -N MyJob
#PBS -m be
#PBS -k oe
#
myprog -a 100 < infile1 > outfile1 &
myprog -a 200 < infile2 > outfile2 &
wait

Using swarm

#
# this file is cmdfile
#
myprog -param a < infile-a > outfile-a
myprog -param b < infile-b > outfile-b
myprog -param c < infile-c > outfile-c
myprog -param d < infile-d > outfile-d
myprog -param e < infile-e > outfile-e
myprog -param f < infile-f > outfile-f
myprog -param g < infile-g > outfile-g

swarm -f cmdfile

32 lines (processes) => 16 jobs
2000 lines (processes) => 1000 jobs

# bundle option
#
swarm -b 20 -f cmdfile

swarmdel

parallel jobs

large memory jobs (>4 GB)

See also Firebolt web page (http://biowulf.nih.gov/firebolt.html) for running jobs on the Altix.

Node Selection and Allocation

PBS Node Properties

Notes:

Single-core vs. Dual/Quad-core nodes

Examples

qsub -l nodes=1 myjob
qsub -l nodes=1:x86-64 my64bitjob
qsub -l nodes=8:o2800:gige -v np=16 namdjob
qsub -l nodes=16:ipath -v np=32 bignamd.sh
qsub -l nodes=16:ipath -v np=128 biggernamd.sh
qsub -l nodes=1:m4096 bigjob.bat
swarm -l nodes=1:m4096 -f bigjobs
qsub -l nodes=1,mem=384gb verybigmem.bat

Limits

$ batchlim
             Max CPUs    Max CPUs
             Per User   Available
            ---------- -----------
norm        300        n/a        
nist2       32         n/a        
nist1       64         n/a        
ib          256        n/a        
norm3       40         100        
ipath       128        n/a        
altix       8          31         

            Max Mem    Max Mem
            Per User   on System
            ---------- -----------
altix       48gb       91 gb
DL-785      512gb      512gb

System Software

Note: CentOS is a clone of RHEL

Compiling Code

qsub -I -l nodes=1:x86-64

Compilers and set-up scripts available on Biowulf:

Debuggers

• GNU Debugger (gdb)
• Valgrind (valgrind)
• Intel Debugger (idb)
• Portland Group Debugger (pgdbg)
• Pathscale Debugger (pathdb)

Compiling MPI parallel programs

PATH=<MPI Home>/bin:$PATH 

qsub -l nodes=1:ipath -I
qsub -l nodes=1:ib -I

MPI Home PATHs:

MPI compiler wrappers:

Sample MPI program:

#include <stdio.h>
#include "mpi.h"
int
main(int argc, char **argv)
{
  int myrank, n_processes, srcrank, destrank;
  char mbuf[512], name[40];
  MPI_Status mstat;
  MPI_Init(&argc, &argv);
  MPI_Comm_rank(MPI_COMM_WORLD, &myrank);
  MPI_Comm_size(MPI_COMM_WORLD, &n_processes);
  if (myrank != 0) {
    gethostname(name,39);
    sprintf(mbuf, "Hello, from process %d on node %s!", myrank, name);
    destrank = 0;
    MPI_Send(mbuf, strlen(mbuf)+1, MPI_CHAR, destrank, 90, MPI_COMM_WORLD);
  } else {
    for (srcrank = 1; srcrank < n_processes; srcrank++) {
      MPI_Recv(mbuf, 512, MPI_CHAR, srcrank, 90, MPI_COMM_WORLD, &mstat);
      printf("From process %d: %s\n", srcrank, mbuf);
    }
  }
  MPI_Finalize();
}

% PATH=/usr/local/mpich/bin:$PATH
% mpicc -o hello_mpi hello_mpi.c

Running MPI Programs (MPICH2)

#!/bin/bash
# This file is hello-mpich2.bat
#
#PBS -N Hello
PATH=/usr/local/mpich2/bin:$PATH; export PATH
mpdboot -f $PBS_NODEFILE -n `cat $PBS_NODEFILE | wc -l`
mpiexec -n $np /home/steve/hello/hello-mpich2
mpdallexit

qsub -v np=16 -l nodes=8 hello-mpich2.sh

Running MPI Programs (MPICH1)

#!/bin/bash
# This file is hello-mpich.bat
#
#PBS -N MyJob
#PBS -m be
#PBS -k oe
PATH=/usr/local/mpich/bin:$PATH; export PATH
mpirun -machinefile $PBS_NODEFILE -np $np hello_mpi

qsub -v np=8 -l nodes=4:myr2k hello-myr.sh
qsub -v np=16 -l nodes=8:ib hello-ib.sh

Scientific Libraries

• FFTW
• Intel Math Kernel Library (MKL)
• AMD Core Math Library (ACML)
• Intel Integrated Performance Primitives (IPP)
• GNU Scientific Library (GSL)