NIH Helix Systems
Firebolt User Information

About Firebolt

Firebolt is an SGI Altix 350 with 32 Itanium 2 processors and 96GB of memory, using an SGI NUMAlink interconnect and runs under RedHat Enterprise Linux 3.

It is managed as a "fat node" of the NIH Biowulf Cluster (see Using Firebolt, below).

All Biowulf users automatically obtain accounts on Firebolt, and as such all account details such as passwords, home and data directories, shells, mail, etc., are handled through Biowulf, the Biowulf cluster head node. Firebolt users should therefore be familiar with the Biowulf user guide.

Due to the unique characteristics of Firebolt, its use should be limited to applications that require large amounts of memory (>4GB). All other applications should be run on the Biowulf cluster. Please contact staff@helix.nih.gov to discuss whether your jobs are suitable for Firebolt.

The number of processors per user is limited, and will be adjusted depending on usage and load.

Note that Firebolt has a different architecture than the Opterons and Xeons in the Biowulf cluster. All homegrown code will therefore need to be recompiled for Firebolt.

Please send comments or problem reports to staff@biowulf.nih.gov.

Disk Storage

Using Firebolt

Node and CPU allocation

Firebolt is integrated into the Biowulf cluster as a 'fat node'. Note that unlike the regular Biowulf nodes, Firebolt may be shared by several jobs using a few processors each.

Examples of batch commands for Firebolt jobs

Submit a job	qsub -l nodes=1:altix:ncpus=4,mem=4gb my_script
Delete a job	qdel jobnumber
Check job status	qstat -u username qstat -f jobnumber

The number of processors required must be specified when submitting the job. Unlike the other nodes on the Biowulf cluster, multiple independent jobs belonging to different users may run simultaneously on the Altix. To avoid oversubscribing the system, it is critical to specify the number of processors required for the job. For this purpose, the resource flag "ncpus=#" is preferable to the old "ppn=#". When a job is allocated with "ncpus=#", the following PBS environment variable is set:

NCPUS=#

----------------------------------------------------------------------
#!/bin/csh
#PBS -N myjob
#
cd /data/user/dmol
/usr/local/msi/MaterialsStudio/DMol3/bin/RunDMol3.sh -np $NCPUS myjob
----------------------------------------------------------------------

Jobs must also specify the memory required. Since several large-memory jobs may be running on Firebolt simultaneously, this will ensure that the batch system can track the memory usage correctly, and that jobs will wait in the queue until the required memory is available. Memory is specified using the mem=###mb or mem=###gb resource. If no memory is specified, PBS will reject the job.. If the job uses more memory than is requested, it will be terminated by the batch system.

qsub -l nodes=1:altix:ncpus=4,mem=5gb PBS_script

qsub -I -l nodes=1:altix,mem=6500mb

Note that jobs will be terminated by the batch system if they use more resources (cpus, memory) than were requested at the time of submission. Jobs that require less than 4GB of memory should be submitted to the regular Biowulf nodes.

Monitoring your jobs

Jobs submitted to Firebolt through PBS are visible using qstat:

[joe@biowulf mydir]$ qstat -u joe biobos:16001: Req'd Req'd Elap Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time --------------- -------- -------- ---------- ------ --- --- ------ ----- - ----- 326.biobos joe altix DMOL_large 11540 1 4 4gb -- R 00:00

Available cpus and memory on Firebolt can be viewed with the freen command:

[joe@biowulf mydir]$ freen m1024 m2048 m4096 Total ------------- GeneralPool ------------- o2800 / 10/91 26/240 36/331 o2200 / 86/234 21/43 107/277 o2000 / 34/38 / 34/38 p2800 52/80 164/202 45/62 261/344 p1800 0/34 0/13 / 0/47 p1400 0/4 / / 0/4 ------------- Myrinet ------------- o2200 / 37/75 / 37/75 o2000 37/47 / / 37/47 p2800 6/38 / / 6/38 ------------- Infiniband ------------- o2800 / / 10/97 10/97 ------------- Reserved ------------- o2800 / / 5/15 5/15 o2600 / / 82/276 82/276 o2200 / / 0/14 0/14 --------------- Altix ---------------- Available: 21 processors, 69.1 GB memory

In the example above, there are 21 processors and 69.1 GB of memory available.

The current per-user limits on the Altix (and all Biowulf batch queues) can be seen with the 'batchlim' command. e.g.

[jane@biowulf ~]$ batchlim Max CPUs Max CPUs Per User Available ---------- ----------- norm 160 n/a nist2 32 n/a nist1 64 n/a ib2 48 138 groope 16 16 ib 128 164 norm3 32 100 centos 160 n/a altix 8 31 altix mem 48gb 91.6gb

Programming Tools & Libraries

In addition, Firebolt also has the latest Intel compilers, including C (icc), C++ (icpc) and Fortran (ifort -ldl).

Documentation for the Intel compilers is online at http://biowulf.nih.gov/doc/intel/cc80/ccompindex.htm and http://biowulf.nih.gov/doc/intel//fc80/fcompindex.htm.

Compiling on Firebolt: Since Firebolt has a different architecture from the other Biowulf nodes, programs will need to be compiled specifically for Firebolt. To compile a program, allocate an interactive processor on Firebolt as follows:

qsub -I -l nodes=1:altix

Before using the Intel compilers you must source a file which will set up environment variables:

% source /usr/local/intel/intel_fc_80/bin/ifortvars.sh (for fortran with bash shell)
% source /usr/local/intel/intel_fc_80/bin/ifortvars.csh (for fortran with csh)
% source /usr/local/intel/intel_cc_80/bin/iccvars.sh (for c/c++ with bash shell)
% source /usr/local/intel/intel_cc_80/bin/iccvars.csh (for c/c++ with csh)

NOTE: Fortran programs will not compile and link unless the option -ldl is added to the link line. This is due to the static linkage of libdl and the fact that libdl itself changed in SGI ProPack 3 Service Pack 1 when the glibc package was updated to glibc-2.3.2-95.20.

Applications

Gaussian (note that $GAUSS_SCRDIR = /gaussian by default)
Here is a plot of 2-cpu (%nproc = 2) benchmark jobs (6/2/06) on various systems as a comparison to firebolt:

DMOL

ClustalW

R