FSL on the Biowulf Linux CLuster

		Search:

FSL on Biowulf

FSL is a comprehensive library of image analysis and statistical tools for FMRI, MRI and DTI brain imaging data. FSL is written mainly by members of the Analysis Group, FMRIB, Oxford, UK. FSL website.

On the Biowulf cluster, the latest version of FSL is installed in /usr/local/fsl. We usually keep 2 previous versions. You can set up the environment for any version using the modules commands as in the example below.

[user@biowulf]$ module avail fsl

--------------------- /usr/local/Modules/3.2.9/modulefiles ------------------
fsl/4.1.8        fsl/4.1.9        fsl/5.0(default)

[user@biowulf]$ module load fsl/5.0

[user@biowulf]$ module list
Currently Loaded Modulefiles:
  1) fsl/5.0

It is best to put the 'module load fsl/version' command into your .bashrc or .cshrc file, so that it will be automatically set up for batch jobs. Users who want to set up personal environment variables for FSL should also put those in a .bashrc or .cshrc file.

32-bit vs 64-bit

The Biowulf head node and all computational nodes are 64-bit as of 27 July 2010. Thus, only a 64-bit version of FSL is currently maintained on the cluster. Both /usr/local/fsl and /usr/local/fsl-64 point to the 64-bit version of FSL.

Submitting a swarm of FSL jobs

FSL is typically used on Biowulf to process many images. This is most easily done via the swarm utility. Below is a sample swarm command file which runs mcflirt and bedfunc in succession on each image.

Example FSL runs courtesy of Adam Thomas, NIMH

# this file is called myswarmfile
mcflirt -in /data/user/fmri1 -out mcf1 -mats -plots -refvol 90 -rmsrel -rmsabs; betfunc mcf1 bet1
mcflirt -in /data/user/fmri2 -out mcf2 -mats -plots -refvol 90 -rmsrel -rmsabs; betfunc mcf2 bet2
mcflirt -in /data/user/fmri3 -out mcf3 -mats -plots -refvol 90 -rmsrel -rmsabs; betfunc mcf3 bet3
...

If each process (a single line in the command file above) requires less than 1 GB RAM, submit with

swarm -f myswarmfile

If each process requires more than 1 GB RAM, submit with

swarm -g # -f myswarmfile

where '#' is the number of Gigabytes required by a single process.

Parallel FSL programs

All parallelization in FSL v4.0 and up is done via the fsl_sub command that is built into several tools. fsl_sub has been modified to use the Biowulf swarm utility.

The following programs in FSL can use parallelization: FEAT, MELODIC, TBSS, BEDPOST, FSL-VBM, POSSUM. See the FSL website for more information.

Sample session running bedpost (User input in bold)

[susanc@biowulf ~]$ cd mydir
[susanc@biowulf mydir]$ bedpostx sampledataset
subjectdir is /data/susanc/bedpost/sampledataset
Making bedpostx directory structure
Queuing preprocessing stages
Queuing parallel processing stage
0 slices processed
Queuing post processing stage
/usr/local/fsl-4.1.0/bin/bedpostx: line 210: 27598 Terminated ${subjdir}.bedpostX/monitor

[susanc@biowulf ~]

Note: the line with 'Terminated' is not a problem and can be safely ignored.

The 'bedpostx' command above submits a set of jobs to first run the preprocessing commands, then process the single slices simultaneously, and then run the post-processing commands.

The jobs can be monitored using the 'qstat' and 'jobload' commands, and the user monitor. Typically, you would see the pre-processing stage run immediately while the bedpost processing steps are in 'H' (hold) state. A few minutes later all the bedpost single-slice processing should run simultaneously, as in the example below.

[susanc@p2 mydir]$ qstat -u susanc

biobos: 
                                                            Req'd  Req'd   Elap
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----
1580363.biobos  susanc   norm     STDIN       27081   1   1    --    --  R 00:09
1580463.biobos  susanc   norm     bpx_prepro   3137   1   1    --    --  R 00:00
1580464.biobos  susanc   norm     swarm1n106    --    1   1    --    --  H   --
1580465.biobos  susanc   norm     swarm2n106    --    1   1    --    --  H   --
1580466.biobos  susanc   norm     swarm3n106    --    1   1    --    --  H   --
1580467.biobos  susanc   norm     swarm4n106    --    1   1    --    --  H   --
1580468.biobos  susanc   norm     swarm5n106    --    1   1    --    --  H   --
1580469.biobos  susanc   norm     swarm6n106    --    1   1    --    --  H   --
1580470.biobos  susanc   norm     swarm7n106    --    1   1    --    --  H   --
1580471.biobos  susanc   norm     swarm8n106    --    1   1    --    --  H   --
1580472.biobos  susanc   norm     swarm9n106    --    1   1    --    --  H   --
1580473.biobos  susanc   norm     swarm10n10    --    1   1    --    --  H   --
1580475.biobos  susanc   norm     swarm11n10    --    1   1    --    --  H   --
1580476.biobos  susanc   norm     swarm12n10    --    1   1    --    --  H   --
1580477.biobos  susanc   norm     swarm13n10    --    1   1    --    --  H   --
1580478.biobos  susanc   norm     swarm14n10    --    1   1    --    --  H   --
1580479.biobos  susanc   norm     swarm15n10    --    1   1    --    --  H   --
1580480.biobos  susanc   norm     swarm16n10    --    1   1    --    --  H   --
1580481.biobos  susanc   norm     swarm17n10    --    1   1    --    --  H   --
1580482.biobos  susanc   norm     swarm18n10    --    1   1    --    --  H   --
1580483.biobos  susanc   norm     bpx_postpr    --    1   1    --    --  H   -- 

[,.. after 2 mins...]
[susanc@p4 mydir]$ qstat -u susanc

biobos: 
                                                            Req'd  Req'd   Elap
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----
1580363.biobos  susanc   norm     STDIN       23498   1   1    --    --  R 01:05
1580464.biobos  susanc   norm     swarm1n106  19394   1   1    --    --  R 00:00
1580465.biobos  susanc   norm     swarm2n106  25610   1   1    --    --  R 00:00
1580466.biobos  susanc   norm     swarm3n106  22687   1   1    --    --  R 00:00
1580467.biobos  susanc   norm     swarm4n106  30442   1   1    --    --  R 00:00
1580468.biobos  susanc   norm     swarm5n106   9794   1   1    --    --  R 00:00
1580469.biobos  susanc   norm     swarm6n106  30983   1   1    --    --  R 00:00
1580470.biobos  susanc   norm     swarm7n106  12429   1   1    --    --  R 00:00
1580471.biobos  susanc   norm     swarm8n106  12006   1   1    --    --  R 00:00
1580472.biobos  susanc   norm     swarm9n106   6919   1   1    --    --  R 00:00
1580473.biobos  susanc   norm     swarm10n10  17030   1   1    --    --  R 00:00
1580475.biobos  susanc   norm     swarm11n10  24613   1   1    --    --  R 00:00
1580476.biobos  susanc   norm     swarm12n10  24682   1   1    --    --  R 00:00
1580477.biobos  susanc   norm     swarm13n10  14467   1   1    --    --  R 00:00
1580478.biobos  susanc   norm     swarm14n10  18595   1   1    --    --  R 00:00
1580479.biobos  susanc   norm     swarm15n10  20954   1   1    --    --  R 00:00
1580480.biobos  susanc   norm     swarm16n10  18845   1   1    --    --  R 00:00
1580481.biobos  susanc   norm     swarm17n10  17404   1   1    --    --  R 00:00
1580482.biobos  susanc   norm     swarm18n10  25006   1   1    --    --  R 00:00
1580483.biobos  susanc   norm     bpx_postpr    --    1   1    --    --  H   -- 

[susanc@p4 bedpost]$ jobload susanc
Jobs for  susanc    Node    Load    
1580363.biobos      p4       25%
1580464.biobos      p1483    96%
1580465.biobos      p1512    96%
1580466.biobos      p1513    96%
1580467.biobos      p1515    96%
1580468.biobos      p1517    95%
1580469.biobos      p1519    96%
1580470.biobos      p1520    96%
1580471.biobos      p1525    96%
1580472.biobos      p1535    96%
1580473.biobos      p1536    97%
1580475.biobos      p1538    96%
1580476.biobos      p1539    97%
1580477.biobos      p1540    97%
1580478.biobos      p1543    95%
1580479.biobos      p1544    95%
1580480.biobos      p1545    97%
1580481.biobos      p1546    97%
1580482.biobos      p1548    49%

User Average:       90%

Using the FSL GUI

To use the FSL GUI, you need to set up an X-windows connection to the Biowulf head node (biowulf.nih.gov). Check that the Xwindows is working correctly by typing 'xclock' at the Biowulf prompt. Then allocate an interactive node as described below:

biowulf% qsub -I -V -l nodes=1
qsub: waiting for job 1977038.biobos to start
qsub: job 1977038.biobos ready

[susanc@p3 ~]$fsl

You should now see the FSL GUI appear on your desktop as above. Once you are finished using the GUI, please exit your interactive session by typing 'quit'.

You can also start up the GUI for an individual FSL program by typing the name of the program with an uppercase first character. The programs available with GUIs are

ApplyXFM
Bet
ConcatXFM
Fast
Fdt
Feat
Featquery
Flirt
Fmrib_prepare_fieldmap
fsl
fsl_gui
Glm
InvertXFM
Make_flobs
Melodic
Nudge
Possum
Renderhighres
Renderstats
Runtcl
Susan

Documentation

Documentation for individual FSL tools at the FSL website in Oxford.
FSL FAQ
FSL support and training.