Arm Forge User Guide

I Queue template script syntax

I.1 Queue template tags

Each of the tags that will be replaced is listed in the following table, and an example of the text that will be generated when Arm Forge submits your job is given for each.

Additionally, any environment variables in the GUI environment ending in _TAG are replaced throughout the script by the value of those variables.

I.2 Defining new tags

As well as the pre-defined tags listed in the table above you can also define new tags in your template script whose values can be specified in the GUI.

Tag definitions have the following format:

   EXAMPLE_TAG: { key1=value1, key2=value2, ... }

Where key1, key2, are attribute names and value1, value2, are the corresponding values.

The tag will be replaced wherever it occurs with the value specified in the GUI, for example:

  #PBS -option EXAMPLE_TAG

The following attributes are supported:

Examples

   # JOB_TYPE_TAG: {type=select,options=parallel| \ 
      serial,label="Job Type",default=parallel} 
 
   # WALL_CLOCK_LIMIT_TAG: {type=text,label="Wall Clock Limit", \ 
      default="00:30:00",mask="09:09:09"} 
 


   # NODE_USAGE_TAG: {type=select,options=not_shared| \ 
      shared,label="Node Usage",default=not_shared} 
 
   # ACCOUNT_TAG: {type=text,label="Account",global}

See the template files in {installation-directory} /templates for more examples.

To specify values for these tags click the Edit Template Variables button on the Job Submission Options page (see Figure 121 Integration with queuing systems shown previously) or the Run window. You will see a window similar to the one below:

The values you specify are substituted for the corresponding tags in the template file when you run a job.

I.3 Specifying default options

A queue template file may specify defaults for the options on the Job Submission page so that when a user selects the template file these options are automatically filled in.

Example

   # submit: qsub -n NUM_NODES_TAG -t WALL_CLOCK_LIMIT_TAG \ 
      --mode script -A PROJECT_TAG 
   # display: qstat 
   # job regexp: (\d+) 
   # cancel: qdel JOB_ID_TAG
                                                                                       
                                                                                       

I.4 Launching

Usually, your queue script will end in a line that starts mpirun with your target executable.

In a template file, this needs to be modified to run a command that will also launch the Arm Forge backend agents.

Some methods to do this are mentioned in this section.

I.4.1 Using AUTO_LAUNCH_TAG

This is the easiest method, and caters for the majority of cases. Simply replace your mpirun command line with AUTO_LAUNCH_TAG. Arm Forge will replace this with a command appropriate for your configuration (one command on a single line).

For example an mpirun line that looks like this:

   mpirun -np 16 program_name myarg1 myarg2

Becomes:

   AUTO_LAUNCH_TAG

   DDT_MPIRUN_TAG DDT_DEBUGGER_ARGUMENTS_TAG \ 
   MPI_ARGUMENTS_TAG PROGRAM_TAG ARGS_TAG

A typical expansion is:

  /opt/arm/forge/bin/ddt-mpirun --ddthost login1,192.168.0.191 \ 
  --ddtport 4242 --ddtsession 1 \ 
  --ddtsessionfile /home/user/.allinea/session/login1-1 \ 
  --ddtshareddirectory /home/user --np 64 \ 
  --npernode 4 myprogram arg1 arg2 arg3

I.4.2 Using ddt-mpirun

If you need more control than is available using AUTO_LAUNCH_TAG, Arm Forge also provides a drop-in mpirun replacement that can be used to launch your job.

You should replace mpirun with DDTPATH_TAG/bin/ddt-_mpirun.

For example, if your script currently has the line:

   mpirun -np 16 program_name myarg1 myarg2

Then (for illustration only) the equivalent that Arm Forge needs to use would be:

   DDTPATH_TAG/bin/ddt-mpirun -np 16 program_name myarg1 myarg2

For a template script you use tags in place of the program name, arguments and so on, so they can be specified in the GUI rather than editing the queue script each time:

   DDTPATH_TAG/bin/ddt-mpirun -np NUM_PROCS_TAG \ 
   EXTRA_MPI_ARGUMENTS_TAG DDTPATH_TAG/bin/ddt-debugger \ 
   DDT_DEBUGGER_ARGUMENTS_TAG PROGRAM_TAG PROGRAM_ARGUMENTS_TAG

See I.1 Queue template tags for more information on template tags.

I.4.3 MPICH 1 based MPI

If AUTO_LAUNCH_TAG or ddt-mpirun are not suitable, you can also use the following method for MPICH 1 based MPIs.

If your mpirun command line looks like:

   mpirun -np 16 program_name myarg1 myarg2

You need to export the TOTALVIEW environment variable, and add the -tv parameter to mpirun.

For example:

   export TOTALVIEW=DDTPATH_TAG/bin/ddt-debugger-mps 
   MPIRUN_TAG -np NUM_PROCS_TAG \ 
   -tv PROGRAM_TAG PROGRAM_ARGUMENTS_TAG

I.4.4 Scalar programs

If AUTO_LAUNCH_TAG is not suitable, you can also use the following method to launch scalar jobs with your template script:

   DDTPATH_TAG/bin/ddt-client DDT_DEBUGGER_ARGUMENTS_TAG \ 
   PROGRAM_TAG PROGRAM_ARGUMENTS_TAG

I.5 Using PROCS_PER_NODE_TAG

Some queue systems allow you to specify the number of processes, others require you to select the number of nodes and the number of processes per node.

The software caters for both of these but it is important to know whether your template file and queue system expect to be told the number of processes (NUM_PROCS_TAG) or the number of nodes and processes per node (NUM_NODES_TAG and PROCS_PER_NODE_TAG).

If these terms seem strange, see sample.qtf for an explanation of the queue template system.

I.6 Job ID regular expression

The Regexp for job id regular expression is matched on the output from your submit command. The first bracketed expression in the regular expression is used as the job ID. The elements listed in the table are in addition to the conventional quantifiers, range and exclusion operators.

For example, your submit program might return the output job id j1128 has been submitted. One possible regular expression for retrieving the job ID is id\s(.+)\shas.

If you would normally remove the job from the queue by typing job_remove j1128 then you should enter job_removeJOB_ID_TAG as the cancel command.

I.7 Arm IPMI Energy Agent

The Arm IPMI Energy Agent allows Arm MAP and Arm Performance Reports to measure the total energy consumed by the compute nodes in a job.

The IPMI Energy Agent is available to download from our website: IPMI Energy Agent.

I.7.1 Requirements

• The compute nodes must support IPMI.
• The compute nodes must have an IPMI exposed power sensor.
• The compute nodes must have an OpenIPMI compatible kernel module installed, such as ipmi_devintf.
• The compute nodes must have the corresponding device node in /dev, for example /dev/ipmi0.
• The compute nodes must run a supported operating system.
• The IPMI Energy Agent must be run as root.

To list the names of possible IPMI power sensors on a compute node use the following command:

    ipmitool sdr | grep 'Watts'