Biowulf at the NIH
RSS Feed
Parallel Computing with Matlab on Biowulf
The Matlab Distributed Computing Toolbox and Distributed Computing Engine enables you to develop distributed and parallel MATLAB applications and execute them on the cluster. We have 10 Parallel Computing toolbox licenses and 16 Distributed Computing engines. The licenses expire on June 1, 2012.

There is now a trial installation of Distributed Matlab on the Biowulf cluster. We will consider purchasing this product if there is sufficient interest from the Biowulf user community. The number of licenses will depend on demonstrated interest from users. Therefore, if you expect to use Distributed Matlab on the Biowulf cluster, please let us know by sending email to staff@helix.nih.gov.

Parallel Distributed Matlab includes functions based on the Message Passing Interface (MPI) standard that supports explicit communication, enabling you to develop parallel applications in MATLAB. The products also support parallel for loops and global array semantics via distributed arrays for creating parallel applications without explicit message passing.


Creating a Distributed Array job

See the Matlab documentation on distributed arrays.

  1. Create a a distributed array Matlab script.

    Sample Distributed Matlab script (replace the italicized text with your own appropriate directory path)

    ------------------ file dce_rand.m ---------------------------------
    abc = findResource('scheduler','type','generic');
    abc.DataLocation = '/data/user/mydir';
    abc.ClusterMatlabRoot = matlabroot;
    abc.HasSharedFilesystem = true;
    abc.ClusterOsType = 'unix';
    abc.SubmitFcn = @distributedSubmitFnc;
    abc.GetJobStateFcn = @getJobStateFcn;
    abc.DestroyJobFcn = @destroyJobFcn;
    get(abc)
    a=abc.createJob;
    a.createTask(@rand,1,{4,4});
    a.createTask(@rand,1,{4,4});
    a.createTask(@rand,1,{4,4});
    a.createTask(@rand,1,{4,4});
    a.submit;
    waitForState(a,'finished');
    results=getAllOutputArguments(a);
    results{1:4}
    exit
    ---------------------------------------------------------------
    

  2. Create a batch script. Sample script
    ---------------------------- file matlabdce.bat----------------
    #!/bin/tcsh
    #
    #PBS -N Matlab
    #PBS -m be 
    #PBS -k oe
     
    date
    cd /data/user/mydir
    matlab -nodisplay -r dce_rand
    ----------------------------------------------------------------
    
    Note that the Matlab script is called 'dce_rand.m', but the '.m' extension should be left out when using the '-r filename' switch in the Matlab command line.
    The alternate command-line syntax 'matlab -nodisplay < dce_rand.m' is not recommended, as it may cause type-ahead buffer warnings.

  3. Submit the batch script:
    qsub -l nodes=1 matlabdce.bat
    

Testing a Distributed Matlab job

If the main Matlab job is not compute-intensive, it could be run interactively on the Biowulf head node or an interactive batch session. Sample session: (user input in bold)
[user@biowulf ~]$  matlab -nodisplay -r dce_rand

                              < M A T L A B >
                  Copyright 1984-2005 The MathWorks, Inc.
                   Version 7.1.0.183 (R14) Service Pack 3
                              August 02, 2005

  To get started, type one of these: helpwin, helpdesk, or demo.
  For product information, visit www.mathworks.com.
 
>> >> >> >> >> >>                    Type: 'generic'
           DataLocation: '/home/user/'
    HasSharedFilesystem: 1
                   Jobs: [1x1 distcomp.simplejob]
      ClusterMatlabRoot: matlabroot
     MatlabCommandToRun: [1x58 char]
              SubmitFcn: @pbsSubmitFunc
          Configuration: ''

>> >> >> >> >> >> Submitting job
126748[].biobos

>> >> >> 
ans =

    0.9501    0.8913    0.8214    0.9218
    0.2311    0.7621    0.4447    0.7382
    0.6068    0.4565    0.6154    0.1763
    0.4860    0.0185    0.7919    0.4057


ans =

    0.9501    0.8913    0.8214    0.9218
    0.2311    0.7621    0.4447    0.7382
    0.6068    0.4565    0.6154    0.1763
    0.4860    0.0185    0.7919    0.4057


ans =

    0.9501    0.8913    0.8214    0.9218
    0.2311    0.7621    0.4447    0.7382
    0.6068    0.4565    0.6154    0.1763
    0.4860    0.0185    0.7919    0.4057


ans =

    0.9501    0.8913    0.8214    0.9218
    0.2311    0.7621    0.4447    0.7382
    0.6068    0.4565    0.6154    0.1763
    0.4860    0.0185    0.7919    0.4057

>> [user@biowulf ~]$

Monitoring a Distributed Array job

A distributed array job submits a single PBS job array. This job can be monitored with the qstat command.
[susanc@biowulf ~]$ qstat -u susanc

biobos: 
                                                            Req'd  Req'd   Elap
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----
204183[].biobos susanc   norm     matlab_Job    --    1   1    --    --  B   -- 
The state 'B' indicates that the job is an array job, with at least one subjob in Run state. To see the status of all the subjobs in this job, use 'qstat -t'.
[susanc@biowulf ~]$ qstat -t -u susanc

biobos: 
                                                            Req'd  Req'd   Elap
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----
204183[1].biobo susanc   norm     matlab_Job    --    1   1    --    --  R   00:01 
204183[2].biobo susanc   norm     matlab_Job    --    1   1    --    --  R   00:01
204183[3].biobo susanc   norm     matlab_Job    --    1   1    --    --  X   -- 
204183[4].biobo susanc   norm     matlab_Job    --    1   1    --    --  X   -- 
204183[5].biobo susanc   norm     matlab_Job    --    1   1    --    --  X   -- 
204183[6].biobo susanc   norm     matlab_Job    --    1   1    --    --  R   00:01 
204183[7].biobo susanc   norm     matlab_Job    --    1   1    --    --  R   00:01 
204183[8].biobo susanc   norm     matlab_Job    --    1   1    --    --  Q   -- 
204183[9].biobo susanc   norm     matlab_Job    --    1   1    --    --  Q   -- 
204183[10].biob susanc   norm     matlab_Job    --    1   1    --    --  Q   -- 
In this job array, 4 subjobs are running ('R' state), 3 have exited ('X'), and 3 are still queued ('Q').


A Parallel (MPI) Job

  1. Sample Matlab script (replace the italicized text with your own appropriate directory path).
    ---------------------file dce_mpi.m----------------------------------------
    pjob = findResource('scheduler', 'type','generic')
    pjob.ClusterMatlabRoot = matlabroot
    pjob.DataLocation = '/data/user/mydir'
    pjob.HasSharedFilesystem = true;
    pjob.ClusterOsType = 'unix';
    pjob.ParallelSubmitFcn = @parallelSubmitFcn;
    pjob.GetJobStateFcn = @getJobStateFcn;
    pjob.DestroyJobFcn = @destroyJobFcn;
    MaxNumWorkers = 7
    j2 = pjob.createParallelJob('Max', MaxNumWorkers )
    j2.createTask(@rand,1,{4,4})
    j2.submit 
    j2.waitForState
    results = j2.getAllOutputArguments
    results{1:7}
    quit
    ---------------------------------------------------------------------------
    

  2. Run this job on the command line:
    [user@biowulf ~] matlab -nodisplay -r dce_mpi
    
    This job has 7 workers. It will therefore be submitted to 4 nodes by the Matlab program. When the workers have completed the tasks, the results from the 7 workers will be displayed on the terminal window.

  3. Or create a batch script and submit it to the batch system. Sample batch script:
    ---------------------------- file matlabdce.bat----------------
    #!/bin/tcsh
    #
    #PBS -N Matlab
    #PBS -m be 
    #PBS -k oe
     
    date
    cd /data/user/mydir
    matlab -nodisplay -r dce_mpi
    ----------------------------------------------------------------
    
    Submit with:
    qsub -l nodes=1 matlabdce.bat
    
    The main Distributed Matlab job will run on one node, which will then submit the Matlab MPI job to 4 nodes.


Documentation

Matlab Distributed Computing Engine
Distributed Computing Toolbox
Run Matlab in Batch mode on Unix Machines
Matlab's Search Results on Batch
Batch Queuing System on Biowulf
Matlab Help
Matlab FAQ
The Mathworks Homepage


This document is available as http://biowulf.nih.gov/apps/matlab.html
Biowulf home page | Helix Systems | NIH