Biowulf at the NIH
RSS Feed
FreeSurfer on Biowulf

FreeSurfer is a set of automated tools for reconstruction of the brain's cortical surface from structural MRI data, and overlay of functional MRI data onto the reconstructed surface. It was developed at the Martinos Center for Biological Imaging at Harvard. FreeSurfer website.

FreeSurfer is not a parallel program. The only advantage of running FreeSurfer on Biowulf would be to process different subjects simultaneously. Serial processing via FreeSurfer is best done on Helix.

FreeSurfer on Biowulf is a 64-bit program. This means that it will not run on the Biowulf head node. The GUI-based FreeSurfer programs should be run by allocating an interactive node (as described below) or on Helix.

FreeSurfer environment:
The easiest way to see what versions are available is by using the module commands as in the example below. Alternatively, the link /usr/local/freesurfer will always point to the latest version.

biowulf% module avail freesurfer

------------------ /usr/local/Modules/3.2.9/modulefiles -------------------
freesurfer/4.5 freesurfer/5.0 freesurfer/5.1 freesurfer/5.3

biowulf% module load freesurfer/5.3

biowulf% module list
Currently Loaded Modulefiles:
  1) freesurfer/5.3

The freesurfer initialization scripts need to be run in addition to loading the module. If you expect to be running Freesurfer frequently, it's probably best to add these lines to your ~/.bashrc. In the following example, the user has chosen to use Freesurfer 5.3.


# User specific aliases and functions

module load freesurfer/5.3; source $FREESURFER_HOME/

The command 'module load freesurfer/5.3' defines which version of freesurfer you wish to use, and the 'source ...' command runs the appropriate Freesurfer initialization script.

Note that the 'module load freesurfer' command will print out some information to standard error, which may appear in the standard error file from your job. If this is annoying, then use

module load freesurfer/5.3 > /dev/null 2>&1
in your ~/.bashrc file or batch script file.

Running a swarm of FreeSurfer jobs

The following example demonstrates a recon-all job run via the swarm command on Biowulf. (Thanks to Nikhil Sharma at NINDS for this example.)

In the working directory for this project (e.g. /data/user/freesurfer), create a directory called subjects, and create a subdirectory for each subject. See the Freesurfer documentation for a detailed explanation of the directories and files required and created during the recon-all job.

In the working directory, create a swarm command file, e.g. freesurfer.swarm, containing the commands you wish to run on each subject.

cd /data/user/freesurfer/; recon-all -subject hv1 -all
cd /data/user/freesurfer/; recon-all -subject hv2 -all
cd /data/user/freesurfer/; recon-all -subject hv3 -all
with one line for each subject (hv1, hv2, hv3 etc.).

Submit this swarm set with the command:

swarm -f freesurfer.swarm

If each freesurfer job requires more than 1 GB of memory, use

swarm -g # -f freesurfer.swarm
where '#' is the number of Gigabytes of memory required by a single freesurfer process. See the swarm documentation for details.

Running a single Freesurfer program

To run a single Freesurfer program, you will need to create a batch script and submit this job to the Biowulf batch system. Note that you will have a single process running on a node, and therefore wasting one or more processors on that node. This is fine for a small number of jobs, but for larger numbers of jobs you should use the swarm command as described above.

It may be simpler to run directly on Helix if you have only one Freesurfer process. Log on to Helix and type the freesurfer command directly on the Helix command line.

Create a batch script along the following lines:

#  this script is fs.bat

export FREESURFER_HOME=/usr/local/freesurfer   

cd /data/user/mydir
recon-all -subject hv1 -all

Submit this job to the batch system with

biowulf% qsub -l nodes=1 fs.bat 

Running a FreeSurfer GUI program

FreeSurfer GUI programs can easily be run on Helix. Open an Xwindows session to Helix, set up the FreeSurfer environment, and enter a FreeSurfer GUI command.

Alternatively, allocate an interactive node on Biowulf, making sure to export your graphics environment with the '-V' flag, and enter a Freesurfer GUI command. Sample session (user input in bold):

[user@biowulf ~]$ qsub -I -V -l nodes=1 (this commandallocates a 64-bit node by default)
qsub: waiting for job 2113609.biobos to start
qsub: job 2113609.biobos ready

[user@p4 ~]$ export FREESURFER_HOME=/usr/local/freesurfer
[user@p4 ~]$ source /usr/local/freesurfer/
-------- freesurfer-Linux-centos4_x86_64-stable-pub-v4.2.0 --------
Setting up environment for FreeSurfer/FS-FAST (and FSL)
FREESURFER_HOME   /usr/local/freesurfer
FSFAST_HOME       /usr/local/freesurfer/fsfast
SUBJECTS_DIR      /usr/local/freesurfer/subjects
MNI_DIR           /usr/local/freesurfer/mni
FSL_DIR           /usr/local/fsl

[user@p4 ~]$ tkmedit bert orig.mgz
Setting subject to bert
Reading 0 control points...
Reading 0 control points...
Reading /usr/local/freesurfer/lib/tcl/tkm_common.tcl
Reading /usr/local/freesurfer/lib/tcl/tkm_wrappers.tcl
Reading /usr/local/freesurfer/lib/tcl/fsgdfPlot.tcl
Reading /usr/local/freesurfer/lib/tcl/tkUtils.tcl

freesurfer example
[user@p4 ~] exit qsub: job 2113609.biobos completed
You should see the GUI windows appear on your desktop as above. Be sure to type 'exit' when you have finished, so that the node is de-allocated.