Changes to swarm on Biowulf

Effective Tuesday September 13, 2011 a number of significant changes will be made to the swarm command:

Given the heterogeneity of the Biowulf cluster, these changes are being made to (1) let users focus on the requirements of their programs instead of the hardware characteristics of the nodes (2) reduce the burden on users of having to calculate the correct number of processes and threads for each type of node.

The previous version of swarm will continue to be available as oswarm for 60 days. After this time, only the new version will be available, so please start using it as soon as possible.

Syntax Changes

The simplest case use of swarm will be:

swarm --file <swarm control file>

This command will create batch jobs which place one process (i.e., one line in the swarm command file) on each processor core of a node. As long as your programs are single-threaded and do not require more than 1 GB memory per program instance you can use the simplest case syntax.

swarm will now accept three new switches:

switch	short form	description	default value	special values
--gb-per-process	-g	GB memory per process (single line in the swarm control file)	1
--threads-per-process	-t	number of threads in a multi-threaded program	1	auto
--resources	-R	application resources

swarm will continue to support qsub switches.

Examples:

Single-threaded programs that require < 1GB memory

If a single instance of your program (i.e. one line in the swarm command file) uses less than 1 GB of memory, and is single-threaded, you can use the simplest form of the new swarm command:

swarm --file <swarm control file>

You can also use the short form of the --file switch:

swarm -f <swarm control file>

This command will create batch jobs which place one process (i.e., one line in the swarm command file) on each processor core of a node.

Single-threaded programs that require > 1 GB memory

If a single instance of your program (i.e. one line in the swarm command file) is single-threaded and requires more than 1 GB memory, you will need to specify the 'gb-per-process' switch to swarm.

e.g. if one instance of your program requires up to 3 GB of memory:

swarm -g 3 -f <swarm control file> 

This command will create batch jobs such that each process is guaranteed 3 GB of memory. Swarm will ensure that the batch jobs are submitted to appropriate nodes. You do not need to be concerned about the number of cores or the actual memory of each node.

Multithreaded programs where you specify the number of threads

An example of such a program is cufflinks. In this case, you should choose the number of threads for each instance, define it appropriately in the swarm command file (e.g. the "-p" option for cufflinks), and then use the '-t #' option when submitting the swarm job.

In the example below, each instance of the program will consist of 4 threads and requires 4 GB per instance.

swarm -t 4 -g 4 -f <swarm control file>

Multi-threaded programs that 'auto-thread'

Such programs check how many cores are on the node and will create as many threads as there are cores. An example is novoalign. You should use the '-t auto' option to swarm.

In the example below, the programs auto-thread and require less than 1 GB of memory per instance.

swarm -t auto -f <swarm control file> 

Swarm jobs that require specific resources

An example is a swarm of Matlab jobs that each requires a Matlab license resource. You should use the swarm '-R' option to specify the resource required. e.g.

swarm -R matlab=1 -f <swarm control file> 

Memory Usage Reporting

------------------------- PBS time and memory report ------------------------
3875594.biobos elapsed time: 236 seconds
3875594.biobos maximum memory: 2.15 GB
-----------------------------------------------------------------------------

In the example above, the swarm job used 2.15 GB of memory. When submitting future swarms like this one, you should specify '-g 3', and swarm will ensure that at least 3 GB of memory is available to the job.

Bash Shell Syntax

export PATH=/data/user/myprogs:$PATH

Users who have old swarm command files that contain csh syntax should add '--usecsh' to their swarm command.

Additional Documentation