Biowulf at the NIH
RSS Feed
Fastx-Toolkit on Biowulf

Fastx-Toolkit is a collection of command line tools for Short-Reads FASTA/FASTQ files preprocessing.

How to Use

There are multiple versions of Fasxt-Toolkit available. An easy way of selecting the version is to use modules. To see the modules available, type

module avail fastxtoolkit

To select a module, type

module load fastxtoolkit/[ver]

where [ver] is the version of choice. This command can be added to your ~/.bashrc file for permanency.

Commands

These are the current commands that are available:

fasta_clipping_histogram.pl                     fastx_barcode_splitter.pl
fasta_formatter                                 fastx_clipper
fasta_nucleotide_changer                        fastx_collapser
fastq_masker                                    fastx_nucleotide_distribution_graph.sh
fastq_quality_boxplot_graph.sh                  fastx_nucleotide_distribution_line_graph.sh
fastq_quality_converter                         fastx_quality_stats
fastq_quality_filter                            fastx_renamer
fastq_quality_trimmer                           fastx_reverse_complement
fastq_to_fasta                                  fastx_trimmer
fastx_artifacts_filter                          fastx_uncollapser
Submitting a single batch job

1. Create a script file. The file will contain the lines similar to the lines below. Modify the path of program location before running.

#!/bin/bash
# This file is YourOwnFileName
#
#PBS -N yourownfilename
#PBS -m be
#PBS -k oe

export PATH=/usr/local/fastx_toolkit/:$PATH
cd /data/user/somewhereWithInputFile
fastx_toolkit command 1
fastx_toolkit command 2
....
....

2. Submit the script using the 'qsub' command on Biowulf, e.g. Note, user is recommend to run benchmarks to determine what kind of node is suitable for his/her jobs.

[user@biowulf]$ qsub -l nodes=1 /data/username/theScriptFileAbove

Useful commands:

freen: see http://biowulf.nih.gov/user_guide.html#freen

qstat: search for 'qstat' on http://biowulf.nih.gov/user_guide.html for it's usage.

jobload: search for 'jobload' on http://biowulf.nih.gov/user_guide.html for it's usage.

 

Submitting a swarm of jobs

Using the 'swarm' utility, one can submit many jobs to the cluster to run concurrently.

Set up a swarm command file (eg /data/username/cmdfile). Here is a sample file:

export PATH=/usr/local/fastx_toolkit/:$PATH ; cd /data/user/run1/; fastx_toolkit command 1; fastx_toolkit command 2

export PATH=/usr/local/fastx_toolkit/:$PATH ; cd /data/user/run2/; fastx_toolkit command 1; fastx_toolkit command 2

........

........

export PATH=/usr/local/fastx_toolkit/:$PATH ; cd /data/user/run10/; fastx_toolkit command 1; fastx_toolkit command 2

 

There are one flag of swarm that's required '-f' and two other flags of swarm user most possibly needs to specify when submit a swarm job: '-t' and '-g'.

-f: the swarm command file name above (required)
-t: number of processors per node to use for each line of the commands in the swarm file above.(optional)
-g: GB of memory needed for each line of the commands in the swarm file above.(optional)

By default, each line of the commands above will be executed on '1' processor core of a node and uses 1GB of memory. If this is not what you want, you will need to specify '-t' and '-g' flags when you submit the job on biowulf.

Say if each line of the commands above also will need to use 10gb of memory instead of the default 1gb of memory, make sure swarm understands this by including '-g 10' flag:

[user@biowulf]$ swarm -g 10 -f cmdfile

For more information regarding running swarm, see swarm.html

 

Running an interactive job

User may need to run jobs interactively sometimes. Such jobs should not be run on the Biowulf login node. Instead allocate an interactive node as described below, and run the interactive job there.

[user@biowulf] $ qsub -I -l nodes=1
qsub: waiting for job 2236960.biobos to start
qsub: job 2236960.biobos ready

[user@p4]$ cd /data/user/myruns
[user@p4]$ export PATH=/usr/local/fastx_toolkit:$PATH/
[user@p4]$ cd /data/userID/fastx_toolkit/run1
[user@p4]$ fastx_toolkit command 1
[user@p4]$ fastx_toolkit command 2
[user@p4]$ ...........
[user@p4] exit
qsub: job 2236960.biobos completed
[user@biowulf]$

User may add property of node in the qsub command to request specific interactive node. For example, if you need a node with 24gb of memory to run job interactively, do this:

[user@biowulf]$ qsub -I -l nodes=1:g24:c16

 

Documentation

http://hannonlab.cshl.edu/fastx_toolkit/commandline.html