Biowulf at the NIH
RSS Feed
FreeSurfer on Biowulf

FreeSurfer is a set of automated tools for reconstruction of the brain's cortical surface from structural MRI data, and overlay of functional MRI data onto the reconstructed surface. It was developed at the Martinos Center for Biological Imaging at Harvard. FreeSurfer website.

FreeSurfer is not a parallel program. The only advantage of running FreeSurfer on Biowulf would be to process different subjects simultaneously. Serial processing via FreeSurfer is best done on Helix.

FreeSurfer on Biowulf is a 64-bit program. This means that it will not run on the Biowulf head node. The GUI-based FreeSurfer programs should be run by allocating an interactive node (as described below) or on Helix.

FreeSurfer environment:
The following commands are required to set up the Freesurfer environment. (Available versions can also been seen and set upe using modules)

bash shell:
export FREESURFER_HOME=/usr/local/apps/freesurfer/freesurfer-5.1
source $FREESURFER_HOME//SetUpFreeSurfer.sh

csh/tcsh shell:
setenv FREESURFER_HOME /usr/local/apps/freesurfer/freesurfer-5.1
source $FREESURFER_HOME//SetUpFreeSurfer.csh

Running a swarm of FreeSurfer jobs

The following example demonstrates a recon-all job run via the swarm command on Biowulf. (Thanks to Nikhil Sharma at NINDS for this example.)

In the working directory for this project (e.g. /data/user/freesurfer), create a directory called subjects, and create a subdirectory for each subject. See the Freesurfer documentation for a detailed explanation of the directories and files required and created during the recon-all job.

In the working directory, create a swarm command file, e.g. freesurfer.swarm, containing the commands you wish to run on each subject.

cd /data/user/freesurfer/; recon-all -subject hv1 -all
cd /data/user/freesurfer/; recon-all -subject hv2 -all
cd /data/user/freesurfer/; recon-all -subject hv3 -all
[...]
with one line for each subject (hv1, hv2, hv3 etc.).

Submit this swarm set with the command:

swarm -f freesurfer.swarm

If each freesurfer job requires more than 1 GB of memory, use

swarm -g # -f freesurfer.swarm
where '#' is the number of Gigabytes of memory required by a single freesurfer process. See the swarm documentation for details.

Running a single Freesurfer program

To run a single Freesurfer program, you will need to create a batch script and submit this job to the Biowulf batch system. Note that you will have a single process running on a node, and therefore wasting one or more processors on that node. This is fine for a small number of jobs, but for larger numbers of jobs you should use the swarm command as described above.

It may be simpler to run directly on Helix if you have only one Freesurfer process. Log on to Helix and type the freesurfer command directly on the Helix command line.

Create a batch script along the following lines:

#!/bin/bash
#  this script is fs.bat

export FREESURFER_HOME=/usr/local/freesurfer   
source $FREESURFER_HOME//SetUpFreeSurfer.sh

cd /data/user/mydir
recon-all -subject hv1 -all

Submit this job to the batch system with

biowulf% qsub -l nodes=1 fs.bat 

Running a FreeSurfer GUI program

FreeSurfer GUI programs can easily be run on Helix. Open an Xwindows session to Helix, set up the FreeSurfer environment, and enter a FreeSurfer GUI command.

Alternatively, allocate an interactive node on Biowulf, making sure to export your graphics environment with the '-V' flag, and enter a Freesurfer GUI command. Sample session (user input in bold):

[user@biowulf ~]$ qsub -I -V -l nodes=1 (this commandallocates a 64-bit node by default)
qsub: waiting for job 2113609.biobos to start
qsub: job 2113609.biobos ready

[user@p4 ~]$ export FREESURFER_HOME=/usr/local/freesurfer
[user@p4 ~]$ source /usr/local/freesurfer/SetUpFreeSurfer.sh
-------- freesurfer-Linux-centos4_x86_64-stable-pub-v4.2.0 --------
Setting up environment for FreeSurfer/FS-FAST (and FSL)
FREESURFER_HOME   /usr/local/freesurfer
FSFAST_HOME       /usr/local/freesurfer/fsfast
FSF_OUTPUT_FORMAT nii
SUBJECTS_DIR      /usr/local/freesurfer/subjects
MNI_DIR           /usr/local/freesurfer/mni
FSL_DIR           /usr/local/fsl

[user@p4 ~]$ tkmedit bert orig.mgz
Setting subject to bert
Reading 0 control points...
Reading 0 control points...
Reading /usr/local/freesurfer/lib/tcl/tkm_common.tcl
Reading /usr/local/freesurfer/lib/tcl/tkm_wrappers.tcl
Reading /usr/local/freesurfer/lib/tcl/fsgdfPlot.tcl
Reading /usr/local/freesurfer/lib/tcl/tkUtils.tcl

freesurfer example
[user@p4 ~] exit qsub: job 2113609.biobos completed
You should see the GUI windows appear on your desktop as above. Be sure to type 'exit' when you have finished, so that the node is de-allocated.