FAQ: Sun Grid Engine (SGE)
- What is Sun Grid Engine (SGE)?
- How do I submit a job?
- What is the option for maximum walltime?
- Do I need to run any special commands in my job script?
- Why do I have problems while I use the ksh or bash shells?
- Where is an example of SGE submit script?
- How do I monitor a job?
- How do I see the status of my job, or other jobs, and what does it mean?
- How do I delete a job?
- How do I see the job's output files?
- How do I temporarily exceed my maximum resource settings?
- Can I still use OpenPBS?
- What parallel environments are available on each system?
- Output from qstat is makes no sense, what is up?
- Why won't my job start running?
- Why is there no qtail, or how I see the jobs output files?
- What are .pe and .po files?
Q: What is Sun Grid Engine (SGE)?
A: Sun's Grid Engine (SGE) is a resource management system that is used on Jet to control jobs running on the system. All jobs that run on Jet are submitted as scripts to the SGE server.
A: To submit a job, you write a script that performs the operations necessary to complete the job.The job is then submitted with qsub to the batch system. When the requested resources are available, the job is run. Unless the output of the script is redirected explicitly, the system will capture the standard out and standard error output and write it to a file. The file will be written in the user's home account, or the directory specified in the batch script.
The status of the job can been displayed by running sgestat.
Typical syntax for qsub is:
qsub myscript.sge
where myscript.sge is he name of your batch script.
Below is a list of the main options that most users will require. These can be specified on the command line, but are usually inserted into the batch script. In a batch script, the commands are listed after the delimeter '#$'. You do not have to specify the options both in the script and on the command line. Options on the command line will override those found in the batch script.
-A account name- Specify the account under which to run the job (REQUIRED)
-N name- The name of the job
-r y,n- Should this job be re-runnable (default y)
-pe type num- Request num amount of type nodes. In general the comp type of node should be used.
-cwd- place the output files (.e,.o) in the working directory. The default is to place them in the users home directory.
-S shell path- specify the shell to use when running job script
Q: What is the option for maximum walltime?
A: -l h_rt - hr:min:sec - Maximum walltime for this job (The maximum is 8 hours)
Q: Do I need to run any special commands in my job script?
A: To run mpi jobs on iJet, it is necessary to source the following script to correctly setup the environment for mpi.
For csh and tcsh
source /usr/local/bin/setup-mpi.csh.
For bash and ksh
source /usr/local/bin/setup-mpi.sh. To run mpi jobs on eJet, you don't need to source any scripts.
Q: Why do I have problems while I use the ksh or bash shells?
A: The SGE system will use /bin/csh to execute the shell scripts regardless of the shell specified at the top. If you are using an alternate shell, use the -S to specify the shell.
sig: #$ -S /bin/bash
Q: Where is an example of SGE submit script?
A: The following script:
#!/bin/csh -f
#
# Set the name of the job.
#$ -N hpl
#
# Make sure that the .e and .o file arrive in the working directory
#$ -cwd
#
# Set the account to jetmgmt
#$ -A jetmgmt
#
# Ask for 4 cpus of type comp(ijet), ecomp(eJet)
# $ -pe comp 4
#
# My code is re-runnable
#$ -r y
#
# The max walltime for this job is 31 minutes
#$ -l h_rt=00:31:00
echo "STARTING THE JOB"
source /usr/local/bin/setup-mpi.csh
echo -n "NODEFILE:"
cat $SGE_NODEFILE | awk '{printf("%s", $1);}END{printf("\n");}'
cd /home/username/sge/test/hpl
/home/username/bin/mpirun -np 4 ./xhpl############## end of simple.hpl
A: The command sgestat will show the status of the queue. The -why option will show you why a job is not running. (sgestat -why). You can use the command qtop to get information about your processes:
qtop jobid
Q: How do I see the status of my job, or other jobs, and what does it mean?
A: To see a list of the jobs running on the system, run the command sgestat. This will display the jobs running on the system, and waiting to run on the system.
Job-ID Jobname Username Account State Cpus Queue Time Time
-----------------------------------------------------------------------------
1235589 migrate turner of_pams r 20 qcomp5 23:11 24:00
1235665 rucRunMom retroruc naos-ruc r 2 qcomp1 23:29 100:00
1235887 gitm19990111 ridley of_swr r 32 qcomp1 04:16 06:00
1235894 gitm19990607 ridley of_swr r 32 qcomp5 04:00 06:00
1235901 mm5_rg shaw nhra Eqw 0 qcomp1 --:-- 00:20
1236020 gitm20000111 ridley of_swr R 32 qcomp3 00:01 06:00
1237895 retro_wrf rt-aq ap-fc qw 36 comp --:-- 06:00
Job-ID is the job number. Jobname is the name specified by the user, or the name of the batch script. The State of the job has several different commands.
r - Job is running
dr - Job is scheduled for deletion
qw - Job is queued and waiting
E - Job is in error, if this happens please notify the help system
R - Job has been rerun, usually due to a detected system error
A: qdel Job-ID. If for some reason the job sits in "dr" for a long period of time, please submit a help call. hpcshelp.fsl@noaa.gov
Q: How do I see the job's output files?
A: SGE will write the .e and .o files directly to the working directory. Since NFS is used, the files will already be in your directory as the code is executed instead of being delivered at the end of the process. Therefore, the .e and .o files will be updated continuously in the working directory while the job runs.
If you do not use the -cwd option, the files will be placed in the users home directory. If the -cwd option is used, the file will be place in the directory from which the job was submitted.
Q: How do I temporarily exceed my maximum resource settings?
A: If you need to exceed the maximum resource settings for your account, whether it is the number of cpus allowed, total jobs allowed, or wallclock time, you can run jobs in the overflow queue. To access the overflow queue, submit your job with priority zero. Do this by adding the following option to your batch script:
#$ -p 0
If you require the use of more resources for a longer duration, you should submit a request to have your account changed. Keep in mind that the overflow queue can be downsized or even eliminated at any time. This will depend on the load of the system.
A: No, the SGE and PBS systems are mutually exclusive. However, FSL has built in support for PBS syntax in the SGE scripts. You cannot mix syntax in your batch scripts, but if you submit a script with PBS syntax to SGE it will run.
Q: What parallel environments are available on each system?
A: On the Intel Jet system, you should submit jobs to the 'comp' parallel environment (pe). If you need extra ram for a single processor job on the Intel system, submit to the 'io' pe. ALSO, you must specify '#$ -l io' as an SGE option in your script. The IO nodes are not available if you are using PBS syntax.
Q: Output from qstat is makes no sense, what is up?
A: SGE was originally not designed for systems like our cluster. Therefore, it tells you how all nodes are doing. This isn't very useful for large mpi jobs. Use the program sgestat for a more user - friendly output.
Q: Why won't my job start running?
A: If your job hasn't started to run after 5 minutes, there are several possible reasons including the system being full or you are exceeding the maximum resources of your account. The first step to determine the problem is to run:
sgestat -why
This should tell you why.
Q: Why is there no qtail, or how I see the job's output files?
A: SGE will write the .e and .o files directly to the working directory. Since NFS is used, the files will already be in your directory as the code is executed instead of being delivered at the end of the process. Therefore, the .e and .o files will be updated continuously in the working directory while the job runs.
If you do not use the -cwd option, the files will be placed in the users home directory. If the -cwd option is used, the file will be place in the directory from which the job was submitted.
You watched for the existence of the .e and .o files for job completion you should use the wait_job script.
Syntax:
wait_job [jobid]
The script will wait for the job [jobid] to complete.
Q: What are .pe and .po files?
A: These files are created by to the parallel jobs that are submitted. We are not entirely sure what importance they have. In 30000 jobs we have had no failures, so we don't know how these files could be relevant. Until we understand them, we are not going to do anything to prevent their creation.