More About Parallel Environments

---

Last Modified: 07/26 8:29

Parallel Environments are a feature of GridEngine that allows it to interface with various parallel libraries such as MPI, PVM, or Charm++. They allow the scheduler to properly dispatch, distribute, manage, and account for parallel processes. In this section, we will see what parallel libraries are supported on Circe and how to use them.

Available Environments

The following is a table of available parallel environments:

Name	Library	Description
ompi*	OpenMPI 1.2.3	Interfaces with OpenMPI which provides support for every available interconnect (though you can only choose one interconnect at a time). Supports both 32 and 64-bit applications.
mpich-mx*	MPICH-MX-1.2.7p1	Interfaces with the Myricom-supplied MPICH library for Myrinet. This is supported for legacy applications which have not been ported to OpenMPI. 64-bit applications only.
mpich-p4	MPICH-1.2.7p1	Interfaces with the standard MPICH libraries with the TCP/IP driver. This is supported for legacy applications which have not been ported to OpenMPI. 32-bit applications only.

Using Parallel Environments

In your job script, you will need to add the following line:

#$ -pe <pe_name> <#procs>

where <pe_name> specifies the name of the parallel environment from the above list, and <#procs> specifies the desired number or range of processors. Some examples are provided below:

#!/bin/bash
#$ -N my_parallel_job
#$ -cwd
#$ -j y
#$ -o output.$JOB_ID
#$ -pe ompi* 4-8
#$ -l i_ib=true
#$ -l h_rt=08:00:00
#$ -l mem_request=3.5G,h_data=3.5G

sge_mpirun my_application

• Script uses OpenMPI environment
• Request at least 4 processors but no more than 8
• Make sure we use InfiniBand, set our runtime, and define our memory requirements. See Using Complexes

#!/bin/bash
#$ -N my_parallel_job
#$ -cwd
#$ -j y
#$ -o output.$JOB_ID
#$ -pe mpich-mx* 16
#$ -l i_mx=true
#$ -l h_rt=12:00:00
#$ -l mem_request=2.0G,h_data=2.0G

sge_mpirun my_application

• Script uses MPICH-MX environment
• Requires 16 processors
• Requests only nodes which provide Myrinet. See Using Complexes

For more information, contact support at  support@rc.usf.edu