You copied the Doc URL to your clipboard.

I Queue template script syntax

I.1 Queue template tags

Each of the tags that will be replaced is listed in the following table, and an example of the text that will be generated when Arm Forge submits your job is given for each.

Note

It is often sufficient to simply use AUTO_LAUNCH_TAG. See section A.3.1 The template script for an example.

Tag

Description

After Submission Example

AUTO_LAUNCH_TAG

This tag expands to the entire replacement for your 'mpirun' command line.

ddt-mpirun -np 4 myexample.bin

DDTPATH_TAG

The path to the Arm Forge installation

/opt/arm/forge

WORKING_DIRECTORY_TAG

The working directory Arm Forge was launched in

/users/ned

NUM_PROCS_TAG

Total number of processes

16

NUM_PROCS_PLUS_ONE_TAG

Total number of processes + 1

17

NUM_NODES_TAG

Number of compute nodes

8

NUM_NODES_PLUS_ONE_TAG

Number of compute nodes + 1

9

PROCS_PER_NODE_TAG

Processes per node

2

PROCS_PER_NODE_PLUS_ONE_ TAG

Processes per node + 1

3

NUM_THREADS_TAG

Number of OpenMP threads per node (empty if OpenMP if "off")

4

OMP_NUM_THREADS_TAG

Number of OpenMP threads per node (empty if OpenMP is "off")

4

MPIRUN_TAG

mpirun binary (can vary with MPI implementation)

/usr/bin/mpirun

AUTO_MPI_ARGUMENTS_TAG

Required command line flags for mpirun (can vary with MPI implementation)

-np 4

EXTRA_MPI_ARGUMENTS_TAG

Additional mpirun arguments specified in the Run window

-partition DEBUG

PROGRAM_TAG

Target path and filename

/users/ned/a.out

PROGRAM_ARGUMENTS_TAG

Arguments to target program

-myarg myval

INPUT_FILE_TAG

The stdin file specified in the Run window

/users/ned/input.dat

Additionally, any environment variables in the GUI environment ending in _TAG are replaced throughout the script by the value of those variables.

I.2 Defining new tags

As well as the pre-defined tags listed in the table above you can also define new tags in your template script whose values can be specified in the GUI.

Tag definitions have the following format:




   EXAMPLE_TAG: { key1=value1, key2=value2, ... }

Where key1, key2, are attribute names and value1, value2, are the corresponding values.

The tag will be replaced wherever it occurs with the value specified in the GUI, for example:


  #PBS -option EXAMPLE_TAG

The following attributes are supported:

Attribute

Purpose

Example

type

text: General text input.
select: Select from two or more options.
check: Boolean.
file: File name.
number: Real number.
integer: Integer number.

type=text

label

The label for the user interface widget.

label="Account"

default

Default value for this tag

default="interactive"

text type

mask

Input mask:
0: ASCII digit permitted but not required.
9: ASCII digit required. 0-9.
N: ASCII alphanumeric character required. A-Z, a-z, 0-9.
n: ASCII alphanumeric character permitted but not required.

mask="09:09:09"

options type

options

Options to use, separated by the - character

options="not_shared|shared"

check type

checked

Value of a check tag if checked.

checked="enabled"

unchecked

Value of a check tag if unchecked.

unchecked="enabled"

integer and number types

min

Minimum value.

min="0"

max

Maximum value.

max="100"

step

Amount to step by when the up or down arrows are clicked.

step="1"

decimals

Number of decimal places.

decimals="2"

suffix

Display only suffix (will not be included in tag value).

suffix="s"

prefix

Display only prefix (will not be included in tag value).

prefix="$"

file type

mode

open-file: an existing file.
save-file: a new or existing file.
existing-directory: an existing directory.
open-files: one or more existing files, separated by spaces.

mode="open-file"

caption

Window caption for file chooser.

caption="Select File"

dir

Initial directory for file chooser.

dir="/work/output"

filter

Restrict the files displayed in the file chooser to a certain file pattern.

filter="Text files (*.txt)"

Examples


   # JOB_TYPE_TAG: {type=select,options=parallel| \ 
serial,label="Job␣Type",default=parallel}

# WALL_CLOCK_LIMIT_TAG: {type=text,label="Wall Clock Limit", \
default="00:30:00",mask="09:09:09"}

# NODE_USAGE_TAG: {type=select,options=not_shared| \
shared,label="Node␣Usage",default=not_shared}

# ACCOUNT_TAG: {type=text,label="Account",global}

See the template files in {installation-directory} /templates for more examples.

To specify values for these tags click the Edit Template Variables button on the Job Submission Options page (see Figure 121 Integration with queuing systems shown previously) or the Run window. You will see a window similar to the one below:

PIC

Figure 122: Queue Parameters Window

The values you specify are substituted for the corresponding tags in the template file when you run a job.

I.3 Specifying default options

A queue template file may specify defaults for the options on the Job Submission page so that when a user selects the template file these options are automatically filled in.

Name

Job Submission Setting

Example

submit

Submit command Note: the command may include tags.

qsub -n NUM_NODES_TAG -t WALL_CLOCK_LIMIT_TAG --mode script -A PROJECT_TAG

display

Display command The output from this command is shown while waiting for a job to start.

qstat

job regexp

Job regexp

(\d+)

cancel

Cancel command

qdel JOB_ID_TAG

submit scalar

Also submit scalar jobs through the queue

yes

show num_procs

Number of processes: Specify in Run window

yes

show num_nodes

Number of nodes: Specify in Run Window

yes

show procs_per_node

Processes per node: Specify in Run window

yes

procs_per_node

Processes per node: Fixed

16

Example

   # submit: qsub -n NUM_NODES_TAG -t WALL_CLOCK_LIMIT_TAG \ 
--mode script -A PROJECT_TAG
# display: qstat
# job regexp: (\d+)
# cancel: qdel JOB_ID_TAG

I.4 Launching

Usually, your queue script will end in a line that starts mpirun with your target executable.

In a template file, this needs to be modified to run a command that will also launch the Arm Forge backend agents.

Some methods to do this are mentioned in this section.

I.4.1 Using AUTO_LAUNCH_TAG

This is the easiest method, and caters for the majority of cases. Simply replace your mpirun command line with AUTO_LAUNCH_TAG. Arm Forge will replace this with a command appropriate for your configuration (one command on a single line).

For example an mpirun line that looks like this:


   mpirun -np 16 program_name myarg1 myarg2

Becomes:


   AUTO_LAUNCH_TAG
AUTO_LAUNCH_TAG is roughly equivalent to:

   DDT_MPIRUN_TAG DDT_DEBUGGER_ARGUMENTS_TAG \ 
MPI_ARGUMENTS_TAG PROGRAM_TAG ARGS_TAG

A typical expansion is:

  /opt/arm/forge/bin/ddt-mpirun --ddthost login1,192.168.0.191 \ 
--ddtport 4242 --ddtsession 1 \
--ddtsessionfile /home/user/.allinea/session/login1-1 \
--ddtshareddirectory /home/user --np 64 \
--npernode 4 myprogram arg1 arg2 arg3

I.4.2 Using ddt-mpirun

If you need more control than is available using AUTO_LAUNCH_TAG, Arm Forge also provides a drop-in mpirun replacement that can be used to launch your job.

Note

this is only suitable for use in a queue template file when Arm Forge is submitting to the queue itself.

You should replace mpirun with DDTPATH_TAG/bin/ddt-mpirun.

For example, if your script currently has the line:


   mpirun -np 16 program_name myarg1 myarg2

Then (for illustration only) the equivalent that Arm Forge needs to use would be:

   DDTPATH_TAG/bin/ddt-mpirun -np 16 program_name myarg1 myarg2

For a template script you use tags in place of the program name, arguments and so on, so they can be specified in the GUI rather than editing the queue script each time:


   DDTPATH_TAG/bin/ddt-mpirun -np NUM_PROCS_TAG \ 
EXTRA_MPI_ARGUMENTS_TAG DDTPATH_TAG/bin/ddt-debugger \
DDT_DEBUGGER_ARGUMENTS_TAG PROGRAM_TAG PROGRAM_ARGUMENTS_TAG

See I.1 Queue template tags for more information on template tags.

I.4.3 MPICH 1 based MPI

If AUTO_LAUNCH_TAG or ddt-mpirun are not suitable, you can also use the following method for MPICH 1 based MPIs.

If your mpirun command line looks like:


   mpirun -np 16 program_name myarg1 myarg2

You need to export the TOTALVIEW environment variable, and add the -tv parameter to mpirun.

For example:


   export TOTALVIEW=DDTPATH_TAG/bin/ddt-debugger-mps 
MPIRUN_TAG -np NUM_PROCS_TAG \
-tv PROGRAM_TAG PROGRAM_ARGUMENTS_TAG

I.4.4 Scalar programs

If AUTO_LAUNCH_TAG is not suitable, you can also use the following method to launch scalar jobs with your template script:


   DDTPATH_TAG/bin/ddt-client DDT_DEBUGGER_ARGUMENTS_TAG \ 
PROGRAM_TAG PROGRAM_ARGUMENTS_TAG

I.5 Using PROCS_PER_NODE_TAG

Some queue systems allow you to specify the number of processes, others require you to select the number of nodes and the number of processes per node.

The software caters for both of these but it is important to know whether your template file and queue system expect to be told the number of processes (NUM_PROCS_TAG) or the number of nodes and processes per node (NUM_NODES_TAG and PROCS_PER_NODE_TAG).

If these terms seem strange, see sample.qtf for an explanation of the queue template system.

I.6 Job ID regular expression

The Regexp for job id regular expression is matched on the output from your submit command. The first bracketed expression in the regular expression is used as the job ID. The elements listed in the table are in addition to the conventional quantifiers, range and exclusion operators.

Element Matches
C A character represents itself
\t A tab
. Any character
\d Any digit
\D Any non-digit
\s White space
\S Non-white space
\w Letters or numbers (a word character)
\W Non-word character

For example, your submit program might return the output job id j1128 has been submitted. One possible regular expression for retrieving the job ID is id\s(.+)\shas.

If you would normally remove the job from the queue by typing job_remove j1128 then you should enter job_removeJOB_ID_TAG as the cancel command.

I.7 Arm IPMI Energy Agent

The Arm IPMI Energy Agent allows Arm MAP and Arm Performance Reports to measure the total energy consumed by the compute nodes in a job.

Note

Measuring energy with IPMI Energy agent requires Arm Forge Professional.

The IPMI Energy Agent is available to download from our website: IPMI Energy Agent.

I.7.1 Requirements

  • The compute nodes must support IPMI.
  • The compute nodes must have an IPMI exposed power sensor.
  • The compute nodes must have an OpenIPMI compatible kernel module installed, such as ipmi_devintf.
  • The compute nodes must have the corresponding device node in /dev, for example /dev/ipmi0.
  • The compute nodes must run a supported operating system.
  • The IPMI Energy Agent must be run as root.

To list the names of possible IPMI power sensors on a compute node use the following command:


    ipmitool sdr | grep 'Watts'
Was this page helpful? Yes No