FreeSurfer is a set of automated tools for reconstruction of the brain's cortical surface from structural MRI data, and overlay of functional MRI data onto the reconstructed surface. It was developed at the Martinos Center for Biological Imaging at Harvard. FreeSurfer website.
FreeSurfer is not a parallel program. The only advantage of running FreeSurfer on Biowulf would be to process different subjects simultaneously. Serial processing via FreeSurfer is best done on Helix.
FreeSurfer on Biowulf is a 64-bit program. This means that it will not run on the Biowulf head node. The GUI-based FreeSurfer programs should be run by allocating an interactive node (as described below) or on Helix.
The following commands are required to set up the Freesurfer environment. (Available versions can also been seen and set upe using modules)
bash shell: export FREESURFER_HOME=/usr/local/apps/freesurfer/freesurfer-5.1 source $FREESURFER_HOME//SetUpFreeSurfer.sh csh/tcsh shell: setenv FREESURFER_HOME /usr/local/apps/freesurfer/freesurfer-5.1 source $FREESURFER_HOME//SetUpFreeSurfer.csh
The following example demonstrates a recon-all job run via the swarm command on Biowulf. (Thanks to Nikhil Sharma at NINDS for this example.)
In the working directory for this project (e.g. /data/user/freesurfer), create a directory called subjects, and create a subdirectory for each subject. See the Freesurfer documentation for a detailed explanation of the directories and files required and created during the recon-all job.
In the working directory, create a swarm command file, e.g. freesurfer.swarm, containing the commands you wish to run on each subject.
cd /data/user/freesurfer/; recon-all -subject hv1 -all cd /data/user/freesurfer/; recon-all -subject hv2 -all cd /data/user/freesurfer/; recon-all -subject hv3 -all [...]
Submit this swarm set with the command:
swarm -f freesurfer.swarm
If each freesurfer job requires more than 1 GB of memory, use
swarm -g # -f freesurfer.swarm
To run a single Freesurfer program, you will need to create a batch script and submit this job to the Biowulf batch system. Note that you will have a single process running on a node, and therefore wasting one or more processors on that node. This is fine for a small number of jobs, but for larger numbers of jobs you should use the swarm command as described above.
It may be simpler to run directly on Helix if you have only one Freesurfer process. Log on to Helix and type the freesurfer command directly on the Helix command line.
Create a batch script along the following lines:
#!/bin/bash # this script is fs.bat export FREESURFER_HOME=/usr/local/freesurfer source $FREESURFER_HOME//SetUpFreeSurfer.sh cd /data/user/mydir recon-all -subject hv1 -all
Submit this job to the batch system with
biowulf% qsub -l nodes=1 fs.bat
FreeSurfer GUI programs can easily be run on Helix. Open an Xwindows session to Helix, set up the FreeSurfer environment, and enter a FreeSurfer GUI command.
Alternatively, allocate an interactive node on Biowulf, making sure to export your graphics environment with the '-V' flag, and enter a Freesurfer GUI command. Sample session (user input in bold):
[user@biowulf ~]$ qsub -I -V -l nodes=1 (this commandallocates a 64-bit node by default) qsub: waiting for job 2113609.biobos to start qsub: job 2113609.biobos ready [user@p4 ~]$ export FREESURFER_HOME=/usr/local/freesurfer [user@p4 ~]$ source /usr/local/freesurfer/SetUpFreeSurfer.sh -------- freesurfer-Linux-centos4_x86_64-stable-pub-v4.2.0 -------- Setting up environment for FreeSurfer/FS-FAST (and FSL) FREESURFER_HOME /usr/local/freesurfer FSFAST_HOME /usr/local/freesurfer/fsfast FSF_OUTPUT_FORMAT nii SUBJECTS_DIR /usr/local/freesurfer/subjects MNI_DIR /usr/local/freesurfer/mni FSL_DIR /usr/local/fsl [user@p4 ~]$ tkmedit bert orig.mgz Setting subject to bert Reading 0 control points... Reading 0 control points... Reading /usr/local/freesurfer/lib/tcl/tkm_common.tcl Reading /usr/local/freesurfer/lib/tcl/tkm_wrappers.tcl Reading /usr/local/freesurfer/lib/tcl/fsgdfPlot.tcl Reading /usr/local/freesurfer/lib/tcl/tkUtils.tcl
[user@p4 ~] exit qsub: job 2113609.biobos completed