I Queue template script syntax
I.1 Queue template tags
Each of the tags that will be replaced is listed in the following table, and an example of the text that will be generated when Arm Forge submits your job is given for each.
Note
It is often sufficient to simply use AUTO_LAUNCH_TAG. See section A.3.1 The template script for an example.
Tag |
Description |
After Submission Example |
AUTO_LAUNCH_TAG |
This tag expands to the entire replacement for your 'mpirun' command line. |
ddt-mpirun -np 4 myexample.bin |
DDTPATH_TAG |
The path to the Arm Forge installation |
/opt/arm/forge |
WORKING_DIRECTORY_TAG |
The working directory Arm Forge was launched in |
/users/ned |
NUM_PROCS_TAG |
Total number of processes |
16 |
NUM_PROCS_PLUS_ONE_TAG |
Total number of processes + 1 |
17 |
NUM_NODES_TAG |
Number of compute nodes |
8 |
NUM_NODES_PLUS_ONE_TAG |
Number of compute nodes + 1 |
9 |
PROCS_PER_NODE_TAG |
Processes per node |
2 |
PROCS_PER_NODE_PLUS_ONE_ TAG |
Processes per node + 1 |
3 |
NUM_THREADS_TAG |
Number of OpenMP threads per node (empty if OpenMP if "off") |
4 |
OMP_NUM_THREADS_TAG |
Number of OpenMP threads per node (empty if OpenMP is "off") |
4 |
MPIRUN_TAG |
mpirun binary (can vary with MPI implementation) |
/usr/bin/mpirun |
AUTO_MPI_ARGUMENTS_TAG |
Required command line flags for mpirun (can vary with MPI implementation) |
-np 4 |
EXTRA_MPI_ARGUMENTS_TAG |
Additional mpirun arguments specified in the Run window |
-partition DEBUG |
PROGRAM_TAG |
Target path and filename |
/users/ned/a.out |
PROGRAM_ARGUMENTS_TAG |
Arguments to target program |
-myarg myval |
INPUT_FILE_TAG |
The stdin file specified in the Run window |
/users/ned/input.dat |
Additionally, any environment variables in the GUI environment ending in _TAG are replaced throughout the script by the value of those variables.
I.2 Defining new tags
As well as the pre-defined tags listed in the table above you can also define new tags in your template script whose values can be specified in the GUI.
Tag definitions have the following format:
EXAMPLE_TAG: { key1=value1, key2=value2, ... }
Where key1, key2, are attribute names and value1, value2, are the corresponding values.
The tag will be replaced wherever it occurs with the value specified in the GUI, for example:
#PBS -option EXAMPLE_TAG
The following attributes are supported:
Attribute |
Purpose |
Example |
type |
text: General text input. |
type=text |
label |
The label for the user interface widget. |
label="Account" |
default |
Default value for this tag |
default="interactive" |
text type | ||
mask |
Input mask: |
mask="09:09:09" |
options type | ||
options |
Options to use, separated by the - character |
options="not_shared|shared" |
check type | ||
checked |
Value of a check tag if checked. |
checked="enabled" |
unchecked |
Value of a check tag if unchecked. |
unchecked="enabled" |
integer and number types | ||
min |
Minimum value. |
min="0" |
max |
Maximum value. |
max="100" |
step |
Amount to step by when the up or down arrows are clicked. |
step="1" |
decimals |
Number of decimal places. |
decimals="2" |
suffix |
Display only suffix (will not be included in tag value). |
suffix="s" |
prefix |
Display only prefix (will not be included in tag value). |
prefix="$" |
file type | ||
mode |
open-file: an existing file. |
mode="open-file" |
caption |
Window caption for file chooser. |
caption="Select File" |
dir |
Initial directory for file chooser. |
dir="/work/output" |
filter |
Restrict the files displayed in the file chooser to a certain file pattern. |
filter="Text files (*.txt)" |
Examples
# JOB_TYPE_TAG: {type=select,options=parallel| \
serial,label="Job Type",default=parallel}
# WALL_CLOCK_LIMIT_TAG: {type=text,label="Wall Clock Limit", \
default="00:30:00",mask="09:09:09"}
# NODE_USAGE_TAG: {type=select,options=not_shared| \
shared,label="Node Usage",default=not_shared}
# ACCOUNT_TAG: {type=text,label="Account",global}
See the template files in {installation-directory} /templates for more examples.
To specify values for these tags click the Edit Template Variables button on the Job Submission Options page (see Figure 121 Integration with queuing systems shown previously) or the Run window. You will see a window similar to the one below:
Figure 122: Queue Parameters Window
|
The values you specify are substituted for the corresponding tags in the template file when you run a job.
I.3 Specifying default options
A queue template file may specify defaults for the options on the Job Submission page so that when a user selects the template file these options are automatically filled in.
Name |
Job Submission Setting |
Example |
submit |
Submit command Note: the command may include tags. |
qsub -n NUM_NODES_TAG -t WALL_CLOCK_LIMIT_TAG --mode script -A PROJECT_TAG |
display |
Display command The output from this command is shown while waiting for a job to start. |
qstat |
job regexp |
Job regexp |
(\d+) |
cancel |
Cancel command |
qdel JOB_ID_TAG |
submit scalar |
Also submit scalar jobs through the queue |
yes |
show num_procs |
Number of processes: Specify in Run window |
yes |
show num_nodes |
Number of nodes: Specify in Run Window |
yes |
show procs_per_node |
Processes per node: Specify in Run window |
yes |
procs_per_node |
Processes per node: Fixed |
16 |
Example
# submit: qsub -n NUM_NODES_TAG -t WALL_CLOCK_LIMIT_TAG \
--mode script -A PROJECT_TAG
# display: qstat
# job regexp: (\d+)
# cancel: qdel JOB_ID_TAG
I.4 Launching
Usually, your queue script will end in a line that starts mpirun with your target executable.
In a template file, this needs to be modified to run a command that will also launch the Arm Forge backend agents.
Some methods to do this are mentioned in this section.
I.4.1 Using AUTO_LAUNCH_TAG
This is the easiest method, and caters for the majority of cases. Simply replace your mpirun command line with AUTO_LAUNCH_TAG. Arm Forge will replace this with a command appropriate for your configuration (one command on a single line).
For example an mpirun line that looks like this:
mpirun -np 16 program_name myarg1 myarg2
Becomes:
AUTO_LAUNCH_TAG
AUTO_LAUNCH_TAG is roughly equivalent to:
DDT_MPIRUN_TAG DDT_DEBUGGER_ARGUMENTS_TAG \
MPI_ARGUMENTS_TAG PROGRAM_TAG ARGS_TAG
A typical expansion is:
/opt/arm/forge/bin/ddt-mpirun --ddthost login1,192.168.0.191 \
--ddtport 4242 --ddtsession 1 \
--ddtsessionfile /home/user/.allinea/session/login1-1 \
--ddtshareddirectory /home/user --np 64 \
--npernode 4 myprogram arg1 arg2 arg3
I.4.2 Using ddt-mpirun
If you need more control than is available using AUTO_LAUNCH_TAG, Arm Forge also provides a drop-in mpirun replacement that can be used to launch your job.
Note
this is only suitable for use in a queue template file when Arm Forge is submitting to the queue itself.
You should replace mpirun with DDTPATH_TAG/bin/ddt-_mpirun.
For example, if your script currently has the line:
mpirun -np 16 program_name myarg1 myarg2
Then (for illustration only) the equivalent that Arm Forge needs to use would be:
DDTPATH_TAG/bin/ddt-mpirun -np 16 program_name myarg1 myarg2
For a template script you use tags in place of the program name, arguments and so on, so they can be specified in the GUI rather than editing the queue script each time:
DDTPATH_TAG/bin/ddt-mpirun -np NUM_PROCS_TAG \
EXTRA_MPI_ARGUMENTS_TAG DDTPATH_TAG/bin/ddt-debugger \
DDT_DEBUGGER_ARGUMENTS_TAG PROGRAM_TAG PROGRAM_ARGUMENTS_TAG
See I.1 Queue template tags for more information on template tags.
I.4.3 MPICH 1 based MPI
If AUTO_LAUNCH_TAG or ddt-mpirun are not suitable, you can also use the following method for MPICH 1 based MPIs.
If your mpirun command line looks like:
mpirun -np 16 program_name myarg1 myarg2
You need to export the TOTALVIEW environment variable, and add the -tv parameter to mpirun.
For example:
export TOTALVIEW=DDTPATH_TAG/bin/ddt-debugger-mps
MPIRUN_TAG -np NUM_PROCS_TAG \
-tv PROGRAM_TAG PROGRAM_ARGUMENTS_TAG
I.4.4 Scalar programs
If AUTO_LAUNCH_TAG is not suitable, you can also use the following method to launch scalar jobs with your template script:
DDTPATH_TAG/bin/ddt-client DDT_DEBUGGER_ARGUMENTS_TAG \
PROGRAM_TAG PROGRAM_ARGUMENTS_TAG
I.5 Using PROCS_PER_NODE_TAG
Some queue systems allow you to specify the number of processes, others require you to select the number of nodes and the number of processes per node.
The software caters for both of these but it is important to know whether your template file and queue system expect to be told the number of processes (NUM_PROCS_TAG) or the number of nodes and processes per node (NUM_NODES_TAG and PROCS_PER_NODE_TAG).
If these terms seem strange, see sample.qtf for an explanation of the queue template system.
I.6 Job ID regular expression
The Regexp for job id regular expression is matched on the output from your submit command. The first bracketed expression in the regular expression is used as the job ID. The elements listed in the table are in addition to the conventional quantifiers, range and exclusion operators.
Element | Matches |
C | A character represents itself |
\t | A tab |
. | Any character |
\d | Any digit |
\D | Any non-digit |
\s | White space |
\S | Non-white space |
\w | Letters or numbers (a word character) |
\W | Non-word character |
For example, your submit program might return the output job id j1128 has been submitted. One possible regular expression for retrieving the job ID is id\s(.+)\shas.
If you would normally remove the job from the queue by typing job_remove j1128 then you should enter job_removeJOB_ID_TAG as the cancel command.
I.7 Arm IPMI Energy Agent
The Arm IPMI Energy Agent allows Arm MAP and Arm Performance Reports to measure the total energy consumed by the compute nodes in a job.
Note
Measuring energy with IPMI Energy agent requires Arm Forge Professional.
The IPMI Energy Agent is available to download from our website: IPMI Energy Agent.
I.7.1 Requirements
- The compute nodes must support IPMI.
- The compute nodes must have an IPMI exposed power sensor.
- The compute nodes must have an OpenIPMI compatible kernel module installed, such as ipmi_devintf.
- The compute nodes must have the corresponding device node in /dev, for example /dev/ipmi0.
- The compute nodes must run a supported operating system.
- The IPMI Energy Agent must be run as root.
To list the names of possible IPMI power sensors on a compute node use the following command:
ipmitool sdr | grep 'Watts'