GridEngine User's Guide
Last Modified: 07/02 10:11
This document will provide information on submitting jobs, removing jobs, checking job status, and interacting with the scheduler.
Key Commands
- qsub [job_script] -- Submit jobs. Refer to man qsub to display detailed explanations for each available option. These options can be added to the command line or to your submit script.
- qstat -- Display the status of your jobs. The man page for qstat will provide detailed explanations for each available option. Useful options include -u <user_name> to filter a single user and -g t to see how your parallel process has been distributed. To see jobs for all users, use -u \*. To find out why your job has not started, use -j <job_id>.
- qdel [job_id] -- Delete/stop jobs. Again, the man page will provide further information. To stop all of your jobs, issue qdel -u [your_user_name]. To force the deletion of your job, issue qdel -f [job_id].
- qs -- Display a status summary of the entire grid. This displays total, used, and available slots per queue as well as the aggregate load average.
Submitting Jobs
You will need to create a submit script to run your job on the cluster. The submit script will specify what type of hardware you wish to run on, what priority your job has, what do do with console output, etc. We will look at SGE submit scripts for Serial and Parallel jobs so that you may have a good understanding of how they work and what it is they do.
Submission for Serial Jobs (Single Processor)
GridEngine uses pre-processed shell scripts to submit jobs. SGE provides predefined variables to help integrate your process with the scheduler and the job dispatcher. It is likely that you will need to pass options to GridEngine to retieive statistical information, set job specifications, redirect your I/O, change your working directory, and possibly notify you of job failure or completion. To set these options, you will need to pass arguments to qsub or you can embed these options in your submit file.
All submit scripts will start of with a definition of your shell, e.g. #!/bin/bash, #!/bin/csh, etc.
A simple job for SGE would look like this
date
It is a simple script that calls date on whatever machine SGE decides to run your job on. Let's have a look at another submit file that does the same thing:
#$ -cwd #$ -j y #$ -N get_date #$ -l h_rt=00:30:00 #$ -o output.$JOB_ID date
With this file, we have specified our output file, the name of the job, and that SGE should execute this job from within the same directory as it was submitted.
An overview of the options (following the character sequence "#$") is as follows:
- -N (name): Set job name for queue
- -cwd: Set directory to current working directory
- -j y: Join standard output and error streams together
- -l h_rt=00:30:00: Tell the scheduler that this process should only run for 30 minutes. Note: This request will become mandatory in the near future so that the scheduler will know when jobs will finish.
These options should be sufficient for the most basic serial jobs.
Submission for Parallel Jobs (Multi-Processor)
Because many of the applications available for science and engineering are highly resource intensive, many of them have been "parallelized" in order to run on multiple pieces of hardware to divide the work load. Most applications have standardized upon the MPI or MPI2 specification. Since many of you will want to run your applications in parallel in order to take advantage of performance gains, you'll need to know how to create a job script for submitting such an applications. GridEngine provides "Parallel Environments" for interfacing your job with an MPI library and distributing it across a cluster.
Rather than explain everything all at once, here is a submit script for a parallel job:
#$ -N parallel-job #$ -cwd #$ -pe ompi* 4 #$ -l i_ib=true,h_rt=08:00:00 #$ -j y #$ -o output.$JOB_ID sge_mpirun parallel-executable
Most of the submit directives (remember, those lines starting with "#$" ?) should already be familiar to you. But you should also notice that we've added a few new directives:
- -pe ompi* 4: This specifies the Parallel Environment you wish to run your job in and the number of slots you would like to allocate (4) for your job. A slot on Circe is generally defined as a processor. Multiple slots can be allocated on a single node, depending on the number of processors that node has. The parallel environment may vary depending on the hardware resources you are assigned when your job is submitted or which MPI library you built your application with. ompi* is considered the default parallel environment. It includes several sub-environments. For a complete run-down of the different parallel environments, please see About Parallel Environments
- -R y: Not shown in the script is the -R y option. This asks the scheduler to reserve resources as they become available until there are enough to satisfy your job. This is useful to prevent job starvation while trying to run a large parallel job. On Circe, it is useful to specify this option for parallel jobs requesting 64 slots or more.
- -V: Not shown in the script is the -V directive. This allows you to export your current environment variables to the submit script.
- -o: This can either be a file path or a directory path. Your standard output will either be written to the file specified or to a file in the path named (job_name).o(job_id).
- -e: -e does the same thing as -o except that it redirects your standard error stream to some file or a file in a path.
- -l i_ib=true: This tells the scheduler that you only want your job to run on resources where i_ib is true. i_ib is a complex (More Information) that tells you which queues are connected via InfiniBand. Any job that requests i_ib=true will run only on hosts which have this capability.
Interactive jobs
Interactive jobs can be run via the qlogin command. Please do not use rlogin, rsh, or ssh to access the nodes. If there are no free nodes available, SGE will notify you that the system is currently full and your interactive session will not be started. If you need immediate access to stop one of your processes from a running SGE job please contact us.
qlogin uses many of the same options that are available to submit scripts. However, you will really only be concerned with ensuring that your job lands on a piece of hardware that is compatible with your job. If your application was compiled to run on an x86_64 processor, the following will ensure that you don't wind up on an incompatible machine:
[user@host ~]$ qlogin -l arch=lx24-amd64
In this case, we are using a complex to specify details about the hosts we wish to utilize. In this case, we want only machines that are of the amd64 variety. For more information on using complexes to specify resource options, please see Using Complexes
Available Environment Variables
The following environment variables are defined by GridEngine at run time. They may be referenced in your submit scripts to add functionality to your code:
- $JOB_ID: The job number assigned by the scheduler to your job
- $JOBDIR: The directory your job is currently running in
- $USER: The username of the person currently running the job
- $TMPDIR: The directory containing informational files about the job e.g. the machines allocated to run the job, etc.
- $SGE_O_WORKDIR: Similar to $JOBDIR
- $HOME: User's home directory on execution machine
- $JOB_NAME: The job name specified by -N option
- $HOSTNAME: Name of execution host
- $TASK_ID: You can submit array jobs. This would be the TASK number in the array job of the current task
- $SGE_CELL: SGE cell that the job is currently running in
- $SGE_O_SHELL: Invoking shell of the current SGE job
- $QUEUE: Current running queue
Complete qsub and Submit Script Options
[-a date_time] request a job start time [-ac context_list] add context variable(s) [-A account_string] account string in accounting record [-b y[es]|n[o]] handle command as binary [-c ckpt_selector] define type of checkpointing for job [-ckpt ckpt-name] request checkpoint method [-clear] skip previous definitions for job [-cwd] use current working directory [-C directive_prefix] define command prefix for job script [-dc simple_context_list] remove context variable(s) [-dl date_time] request a deadline initiation time [-e path_list] specify standard error stream path(s) [-h] place user hold on job [-hard] consider following requests "hard" [-help] print this help [-hold_jid job_identifier_list] define jobnet interdependencies [-i file_list] specify standard input stream file(s) [-j y[es]|n[o]] merge stdout and stderr stream of job [-js job_share] share tree or functional job share [-l resource_list] request the given resources [-m mail_options] define mail notification events [-masterq wc_queue_list] bind master task to queue(s) [-notify] notify job before killing/suspending it [-now y[es]|n[o]] start job immediately or not at all [-M mail_list] notify these e-mail addresses [-N name] specify job name [-o path_list] specify standard output stream path(s) [-P project_name] set job's project [-p priority] define job's relative priority [-pe pe-name slot_range] request slot range for parallel jobs [-q wc_queue_list] bind job to queue(s) [-R y[es]|n[o]] reservation desired [-r y[es]|n[o]] define job as (not) restartable [-sc context_list] set job context (replaces old context) [-shell y[es]|n[o]] start command with or without wrapping <loginshell> -c [-soft] consider following requests as soft [-sync y[es]|n[o]] wait for job to end and return exit code [-S path_list] command interpreter to be used [-t task_id_range] create a job-array with these tasks [-terse] tersed output, print only the job-id [-v variable_list] export these environment variables [-verify] do not submit just verify [-V] export all environment variables [-w e|w|n|v] verify mode (error|warning|none|just verify) for jobs [-@ file]
Storing GridEngine Options
Commonly used options in submit scripts or calls to qstat can be stored in dot files within your home directory in order to prevent you from needlessly using them repeatedly.
qstat Options
Options for the qstat command may be stored in the ~/.sge_qstat file, in your home directory. This allows you to add default options to the qstat command. An example of this follows:
~/.sge_qstat
-u * -g t
Adding these options to the ~/.sge_qstat file ensures that the default call to qstat will show jobs for all users and provide a view of the process distribution throughout the cluster. You can add any of the options defined in the qstat man pages to this file to make them defaults for your account.
qsub and Submit Script Options
Even more useful is the ~/.sge_request file which stores default options for submit scripts and calls to qsub. This is extremely useful for creating a consistent environment for your jobs. In the following file, we have defined defaults for the treatment of STDOUT, STDERR, the name of the output files, to use the current working directory, and what shell I want. These defaults will be used by all of your submit scripts unless overridden in the submit script itself. Below is an example of an ~/.sge_request file and an example submit script using those defaults:
~/.sge_request
-cwd -S /bin/bash -j y -o output.$JOB_NAME.$JOB_ID
And now a job script. Notice the lack of the above options as they are assumed from the ~/.sge_request file:
#$ -N job_name #$ -pe ompi* 12-16 #$ -l h_rt=08:00:00,mem_free=2.0G,i_ib=true sge_mpirun my_parallel_job
External Resources
Additional information on job submission with Sun GridEngine is available at
