FreeSurfer is a set of automated tools for reconstruction of the brain's cortical surface from structural MRI data, and overlay of functional MRI data onto the reconstructed surface. It was developed at the Martinos Center for Biological Imaging at Harvard. FreeSurfer website.
FreeSurfer is not a parallel program. The only advantage of running FreeSurfer on Biowulf would be to process different subjects simultaneously. Serial processing via FreeSurfer is best done on Helix.
FreeSurfer on Biowulf is a 64-bit program. This means that it will not run on the Biowulf head node. The GUI-based FreeSurfer programs should be run by allocating an interactive node (as described below) or on Helix.
The easiest way to see what versions are available is by using the module commands as in the example below. Alternatively, the link /usr/local/freesurfer will always point to the latest version.
biowulf% module avail freesurfer ------------------ /usr/local/Modules/3.2.9/modulefiles ------------------- freesurfer/4.5 freesurfer/5.0 freesurfer/5.1 freesurfer/5.3 biowulf% module load freesurfer/5.3 biowulf% module list Currently Loaded Modulefiles: 1) freesurfer/5.3
The freesurfer initialization scripts need to be run in addition to loading the module. If you expect to be running Freesurfer frequently, it's probably best to add these lines to your ~/.bashrc. In the following example, the user has chosen to use Freesurfer 5.3.
#.bashrc # User specific aliases and functions [...] module load freesurfer/5.3; source $FREESURFER_HOME/SetUpFreeSurfer.sh
The command 'module load freesurfer/5.3' defines which version of freesurfer you wish to use, and the 'source ...' command runs the appropriate Freesurfer initialization script.
Note that the 'module load freesurfer' command will print out some information to standard error, which may appear in the standard error file from your job. If this is annoying, then use
module load freesurfer/5.3 > /dev/null 2>&1 source $FREESURFER_HOME/SetUpFreeSurfer.shin your ~/.bashrc file or batch script file.
The following example demonstrates a recon-all job run via the swarm command on Biowulf. (Thanks to Nikhil Sharma at NINDS for this example.)
In the working directory for this project (e.g. /data/user/freesurfer), create a directory called subjects, and create a subdirectory for each subject. See the Freesurfer documentation for a detailed explanation of the directories and files required and created during the recon-all job.
In the working directory, create a swarm command file, e.g. freesurfer.swarm, containing the commands you wish to run on each subject.
cd /data/user/freesurfer/; recon-all -subject hv1 -all cd /data/user/freesurfer/; recon-all -subject hv2 -all cd /data/user/freesurfer/; recon-all -subject hv3 -all [...]
Submit this swarm set with the command:
swarm -f freesurfer.swarm
If each freesurfer job requires more than 1 GB of memory, use
swarm -g # -f freesurfer.swarm
To run a single Freesurfer program, you will need to create a batch script and submit this job to the Biowulf batch system. Note that you will have a single process running on a node, and therefore wasting one or more processors on that node. This is fine for a small number of jobs, but for larger numbers of jobs you should use the swarm command as described above.
It may be simpler to run directly on Helix if you have only one Freesurfer process. Log on to Helix and type the freesurfer command directly on the Helix command line.
Create a batch script along the following lines:
#!/bin/bash # this script is fs.bat export FREESURFER_HOME=/usr/local/freesurfer source $FREESURFER_HOME//SetUpFreeSurfer.sh cd /data/user/mydir recon-all -subject hv1 -all
Submit this job to the batch system with
biowulf% qsub -l nodes=1 fs.bat
FreeSurfer GUI programs can easily be run on Helix. Open an Xwindows session to Helix, set up the FreeSurfer environment, and enter a FreeSurfer GUI command.
Alternatively, allocate an interactive node on Biowulf, making sure to export your graphics environment with the '-V' flag, and enter a Freesurfer GUI command. Sample session (user input in bold):
[user@biowulf ~]$ qsub -I -V -l nodes=1 (this commandallocates a 64-bit node by default) qsub: waiting for job 2113609.biobos to start qsub: job 2113609.biobos ready [user@p4 ~]$ export FREESURFER_HOME=/usr/local/freesurfer [user@p4 ~]$ source /usr/local/freesurfer/SetUpFreeSurfer.sh -------- freesurfer-Linux-centos4_x86_64-stable-pub-v4.2.0 -------- Setting up environment for FreeSurfer/FS-FAST (and FSL) FREESURFER_HOME /usr/local/freesurfer FSFAST_HOME /usr/local/freesurfer/fsfast FSF_OUTPUT_FORMAT nii SUBJECTS_DIR /usr/local/freesurfer/subjects MNI_DIR /usr/local/freesurfer/mni FSL_DIR /usr/local/fsl [user@p4 ~]$ tkmedit bert orig.mgz Setting subject to bert Reading 0 control points... Reading 0 control points... Reading /usr/local/freesurfer/lib/tcl/tkm_common.tcl Reading /usr/local/freesurfer/lib/tcl/tkm_wrappers.tcl Reading /usr/local/freesurfer/lib/tcl/fsgdfPlot.tcl Reading /usr/local/freesurfer/lib/tcl/tkUtils.tcl
[user@p4 ~] exit qsub: job 2113609.biobos completed