Fastx-Toolkit is a collection of command line tools for Short-Reads FASTA/FASTQ files preprocessing.
There are multiple versions of Fasxt-Toolkit available. An easy way of selecting the version is to use modules. To see the modules available, type
module avail fastxtoolkit
To select a module, type
module load fastxtoolkit/[ver]
where [ver] is the version of choice. This command can be added to your ~/.bashrc file for permanency.
These are the current commands that are available:
fasta_clipping_histogram.pl fastx_barcode_splitter.pl fasta_formatter fastx_clipper fasta_nucleotide_changer fastx_collapser fastq_masker fastx_nucleotide_distribution_graph.sh fastq_quality_boxplot_graph.sh fastx_nucleotide_distribution_line_graph.sh fastq_quality_converter fastx_quality_stats fastq_quality_filter fastx_renamer fastq_quality_trimmer fastx_reverse_complement fastq_to_fasta fastx_trimmer fastx_artifacts_filter fastx_uncollapser
1. Create a script file. The file will contain the lines similar to the lines below. Modify the path of program location before running.
#!/bin/bash # This file is YourOwnFileName # #PBS -N yourownfilename #PBS -m be #PBS -k oe export PATH=/usr/local/fastx_toolkit/:$PATH cd /data/user/somewhereWithInputFile fastx_toolkit command 1 fastx_toolkit command 2 .... ....
2. Submit the script using the 'qsub' command on Biowulf, e.g. Note, user is recommend to run benchmarks to determine what kind of node is suitable for his/her jobs.
freen: see http://biowulf.nih.gov/user_guide.html#freen
qstat: search for 'qstat' on http://biowulf.nih.gov/user_guide.html for it's usage.
jobload: search for 'jobload' on http://biowulf.nih.gov/user_guide.html for it's usage.
Using the 'swarm' utility, one can submit many jobs to the cluster to run concurrently.
Set up a swarm command file (eg /data/username/cmdfile). Here is a sample file:
export PATH=/usr/local/fastx_toolkit/:$PATH ; cd /data/user/run1/; fastx_toolkit command 1; fastx_toolkit command 2
export PATH=/usr/local/fastx_toolkit/:$PATH ; cd /data/user/run2/; fastx_toolkit command 1; fastx_toolkit command 2
export PATH=/usr/local/fastx_toolkit/:$PATH ; cd /data/user/run10/; fastx_toolkit command 1; fastx_toolkit command 2
There are one flag of swarm that's required '-f' and two other flags of swarm user most possibly needs to specify when submit a swarm job: '-t' and '-g'.
-f: the swarm command file name above (required)
-t: number of processors per node to use for each line of the commands in the swarm file above.(optional)
-g: GB of memory needed for each line of the commands in the swarm file above.(optional)
By default, each line of the commands above will be executed on '1' processor core of a node and uses 1GB of memory. If this is not what you want, you will need to specify '-t' and '-g' flags when you submit the job on biowulf.
Say if each line of the commands above also will need to use 10gb of memory instead of the default 1gb of memory, make sure swarm understands this by including '-g 10' flag:
[user@biowulf]$ swarm -g 10 -f cmdfile
For more information regarding running swarm, see swarm.html
User may need to run jobs interactively sometimes. Such jobs should not be run on the Biowulf login node. Instead allocate an interactive node as described below, and run the interactive job there.
qsub: waiting for job 2236960.biobos to start
qsub: job 2236960.biobos ready
[user@p4]$ cd /data/user/myruns
[user@p4]$ export PATH=/usr/local/fastx_toolkit:$PATH/
[user@p4]$ cd /data/userID/fastx_toolkit/run1
[user@p4]$ fastx_toolkit command 1
[user@p4]$ fastx_toolkit command 2
qsub: job 2236960.biobos completed
User may add property of node in the qsub command to request specific interactive node. For example, if you need a node with 24gb of memory to run job interactively, do this: