# -*- text -*-
#
# Copyright (c) 2022 Nanook Consulting.  All rights reserved.
# $COPYRIGHT$
#
# Additional copyrights may follow
#
# $HEADER$
#
#  DISPLAY
#
[display]
The "display" command line directive must be accompanied by a comma-delimited
list of case-insensitive options indicating what information about the job
and/or allocation is to be displayed. The full directive need not be provided -
only enough characters are required to uniquely identify the directive. For
example, ALL is sufficient to represent the ALLOCATION directive - while MAP
can not be used to represent MAP-DEVEL (though MAP-D would suffice).

Supported values include:

-   ALLOCATION displays the detected hosts and slot assignments for this job

-   BINDINGS displays the resulting bindings applied to processes in this job

-   MAP displays the resulting locations assigned to processes in this job

-   MAP-DEVEL displays a more detailed report on the locations assigned to
    processes in this job that includes local and node ranks, assigned
    bindings, and other data

-   TOPO=LIST displays the topology of each node in the comma-delimited list
    that is allocated to the job

No qualifiers are defined for this directive.
#
#  OUTPUT
#
[output]
The "output" command line directive must be accompanied by a comma-delimited
list of case-insensitive options that control how output is generated. The
full directive need not be provided - only enough characters are required
to uniquely identify the directive. For example, MERGE is sufficient to
represent the MERGE-STDERR-TO-STDOUT directive - while TAG can not be used
to represent TAG-DETAILED (though TAG-D would suffice).

Supported values include:

-   TAG marks each output line with the [job,rank]<stream>: of the process that
    generated it

-   TAG-DETAILED marks each output line with a detailed annotation containing
    [namespace,rank][hostname:pid]<stream>: of the process that generated it

-   TAG-FULLNAME marks each output line with the [namespace,rank]<stream>: of
    the process that generated it

-   TIMESTAMP prefixes each output line with a [datetime]<stream>: stamp. Note
    that the timestamp will be the time when the line is output by the DVM and
    not the time when the source output it

-   XML provides all output in a pseudo-XML format

-   MERGE-STDERR-TO-STDOUT merges stderr into stdout

-   DIR=DIRNAME redirects output from application processes into DIRNAME/job/rank/std[out,err,diag]. The provided name will be
    converted to an absolute path

-   FILE=FILENAME redirects output from application processes into
    filename.rank. The provided name will be converted to an absolute
    path

Supported qualifiers include NOCOPY (do not copy the output to the
stdout/err streams), and RAW (do not buffer the output into complete
lines, but instead output it as it is received).
#
#  DVM-HOSTFILE
#
[dvm-hostfile]
PRRTE supports several levels of user-specified host lists based on an established
precedence order. Users can specify a default hostfile that contains a list of
nodes to be used by the DVM. Only one default hostfile can be provided for a given
DVM. In addition, users can specify a hostfile that contains a list of nodes to be
used for a DVM, or can provide a comma-delimited list of nodes to be used for that
DVM via the --host command line option.

The precedence order applied to these various options depends to some extent on
the local environment. The following table illustrates how host and hostfile directives
work together to define the set of hosts upon which a DVM will execute in the absence
of a resource manager (RM):

 default
 hostfile      host        hostfile       Result
----------    ------      ----------      -----------------------------------------
 unset        unset          unset        The DVM will consist solely of the local
                                          host where the DVM was started.
 unset         set           unset        Host option defines resource list for the DVM
 unset        unset           set         Hostfile defines resource list for the DVM
 unset         set            set         Hostfile defines resource list for the DVM,
                                          then host filters the list to define the final
                                          set of nodes to be used by the DVM
  set         unset          unset        Default hostfile defines resource list for the DVM
  set          set           unset        Default hostfile defines resource list for the DVM,
                                          then host filters the list to define the final
                                          set of nodes to be used by the DVM
  set          set            set         Default hostfile defines resource list for the DVM,
                                          then hostfile filters the list, and then host filters
                                          the list to define the final set of nodes to be
                                          used by the DVM

This changes somewhat in the presence of an RM as that entity specifies the
initial allocation of nodes. In this case, the default hostfile, hostfile and host
directives are all used to filter the RM's specification so that a user can utilize different
portions of the allocation for different DVMs. This is done according to the same precedence
order as in the prior table, with the RM providing the initial pool of nodes.

#include#help-schizo-cli#hostfile-syntax
#
#  HOSTFILE-SYNTAX
#
[hostfile-syntax]
Hostfile syntax consists of a node name on each line, optionally including a designated number
of "slots":

node01  slots=10
node13  slots=5
...

Blank lines and lines beginning with a '#' are ignored.  A "slot" is the PRRTE term for an
allocatable unit where we can launch a process. In the absence of the slot parameter, PRRTE
will assign either the number of slots to be the number of CPUs detected on the node or the
RM assigned value if operating in the presence of an RM. Note that in the latter case, the
user-specified number of slots is capped by the RM-assigned value.
#
#  LAUNCHER-HOSTFILE
#
[launcher-hostfile]
PRRTE supports several levels of user-specified hostfiles based on an established
precedence order. Users can specify a hostfile that contains a list of nodes to be
used for the job, or can provide a comma-delimited list of nodes to be used for that
job via the --host command line option.

The precedence order applied to these various options depends to some extent on
the local environment. The following table illustrates how host and hostfile directives
work together to define the set of hosts upon which a DVM will execute the job in the
absence of a resource manager (RM):

 host        hostfile       Result
------      ----------      -----------------------------------------
unset          unset        The DVM will utilize all its available resources
                            when mapping the job.
 set           unset        Host option defines resource list for the job
unset           set         Hostfile defines resource list for the job
 set            set         Hostfile defines resource list for the job,
                            then host filters the list to define the final
                            set of nodes to be used for the job

#include#help-schizo-cli#hostfile-syntax

#include#help-schizo-cli#relative-indexing

#
#  RELATIVE-INDEXING
#
[relative-indexing]
**************************
    Relative Indexing
**************************

Hostfile and --host specifications can also be made using relative indexing. This allows a
user to stipulate which hosts are to be used for a given app_context without specifying the
particular host name, but rather its relative position in the allocation.

This can probably best be understood through consideration of a few examples. Consider the case
where a DVM is comprised of a set of nodes named "foo1, foo2, foo3, foo4". The user wants the
first app_context to have exclusive use of the first two nodes, and a second app_context
to use the last two nodes. Of course, the user could printout the allocation to find the names
of the nodes allocated to them and then use --host to specify this layout, but this is cumbersome
and would require hand-manipulation for every invocation.

A simpler method is to utilize PRRTE's relative indexing capability to specify the desired
layout. In this case, a command line containing:

--host +n1,+n2 ./app1 : --host +n3,+n4 ./app2

would provide the desired pattern. The "+" syntax indicates that the information is being
provided as a relative index into the existing allocation. Two methods of relative indexing
are supported:

+n#    A relative index into the allocation referencing the "#" node. PRRTE will substitute
       the "#" node in the allocation

+e[:#] A request for "#" empty nodes - i.e., PRRTE is to substitute this reference with
       nodes that have not yet been used by any other app_context. If the ":#" is not
       provided, PRRTE will substitute the reference with all empty nodes. Note that PRRTE
       does track the empty nodes that have been assigned in this manner, so multiple
       uses of this option will result in assignment of unique nodes up to the limit of the
       available empty nodes. Requests for more empty nodes than are available will generate
       an error.

Relative indexing can be combined with absolute naming of hosts in any arbitrary manner,
and can be used in hostfiles as well as with the --host command line option. In addition,
any slot specification provided in hostfiles will be respected - thus, a user can specify
that only a certain number of slots from a relative indexed host are to be used for a
given app_context.

Another example may help illustrate this point. Consider the case where the user has a
hostfile containing:

dummy1 slots=4
dummy2 slots=4
dummy3 slots=4
dummy4 slots=4
dummy5 slots=4

This may, for example, be a hostfile that describes a set of commonly-used resources that
the user wishes to execute applications against. For this particular application, the user
plans to map byslot, and wants the first two ranks to be on the second node of any allocation,
the next ranks to land on an empty node, have one rank specifically on dummy4, the next rank
to be on the second node of the allocation again, and finally any remaining ranks to be on
whatever empty nodes are left. To accomplish this, the user provides a hostfile of:

+n2 slots=2
+e:1
dummy4 slots=1
+n2
+e

The user can now use this information in combination with PRRTE's sequential mapper to
obtain their specific layout:

<launcher> --hostfile dummyhosts --hostfile mylayout --prtemca rmaps seq ./my_app

which will result in:

rank0 being mapped to dummy3
rank1 to dummy1 as the first empty node
rank2 to dummy4
rank3 to dummy3
rank4 to dummy2 and rank5 to dummy5 as the last remaining unused nodes

Note that the sequential mapper ignores the number of slots arguments as it only
maps one rank at a time to each node in the list.

If the default round-robin mapper had been used, then the mapping would have resulted in:

ranks 0 and 1 being mapped to dummy3 since two slots were specified
ranks 2-5 on dummy1 as the first empty node, which has four slots
rank6 on dummy4 since the hostfile specifies only a single slot from that node is to be used
ranks 7 and 8 on dummy3 since only two slots remain available
ranks 9-12 on dummy2 since it is the next available empty node and has four slots
ranks 13-16 on dummy5 since it is the last remaining unused node and has four slots

Thus, the use of relative indexing can allow for complex mappings to be ported across
allocations, including those obtained from automated resource managers, without the need
for manual manipulation of scripts and/or command lines.
#
#  DASH-HOST
#
[dash-host]
Host syntax consists of a comma-delimited list of node names, each entry optionally containing
a ":N" extension indicating the number of slots to assign to that entry:

--host node01:5,node02

In the absence of the slot extension, one slot will be assigned to the node. Duplicate entries
are aggregated and the number of slots assigned to that node are summed together.

Note: A "slot" is the PRRTE term for an allocatable unit where we can launch a process. Thus,
the number of slots equates to the maximum number of processes PRRTE may start on that node
without oversubscribing it.
#
#  SESSION-DIR
#
[session-dir]
PRRTE establishes a "session directory" to serve as a top-level location for temporary
files used by both the local PRRTE daemon and its child processes.
By default, the session directory is set to

  <system-tmpdir>/prte.<nodename>.<uid>

This can be altered by setting the "prte_top_session_dir" MCA parameter (either in the environment, on the
cmd line, or in a default parameter file). The session directory name can further be altered to include
the PID of the daemon process, if desired:

  <system-tmpdir>/prte.<nodename>.<pid>.<uid>

by setting the "prte_add_pid_to_session_dirname" MCA parameter.
#
#  TMPDIR
#
[tmpdir]
PRRTE establishes a session directory tree to server as an isolated location for storage
of temporary files used by both local PRRTE daemons and their child processes. This is
done to enable quick and easy cleanup in the event that PRRTE is unable to fully cleanup
after itself. The root of the session directory defaults to the system temporary directory
as defined in the environment using (in precedence order) the envars TMPDIR, TEMP, and
finally TMP. In the absence of any of those variables, PRRTE will default to the "/tmp"
location.
#
#  TMPDIR-TOOL
#
[tmpdir-tool]
PRRTE establishes a session directory tree to server as an isolated location for storage
of temporary files used by both local PRRTE daemons and their child processes. This is
done to enable quick and easy cleanup in the event that PRRTE is unable to fully cleanup
after itself.

In the case of tools, the rendezvous files containing connection information for a target
server are located in the session directory tree. Thus, it may be necessary to point the
tool at the location where those files can be found if that location is other than the
expected default.

The root of the session directory defaults to the system temporary directory
as defined in the environment using (in precedence order) the envars TMPDIR, TEMP, and
finally TMP. In the absence of any of those variables, PRRTE will default to the "/tmp"
location.
#
#  PRTEMCA
#
[prtemca]
Pass a PRRTE MCA parameter

Syntax: --prtemca <arg0> <arg1>
where arg0 is the parameter name and arg1 is the parameter value
#
#  PMIXMCA
#
[pmixmca]
Pass a PMIx MCA parameter

Syntax: --pmixmca <arg0> <arg1>
where arg0 is the parameter name and arg1 is the parameter value
#
#  TUNE
#
[tune]
Comma-delimited list of one or more files containing PRRTE and PMIx MCA params
for tuning DVM and/or application operations. Parameters in the file will be
treated as _generic_ parameters and subject to the translation rules/uncertainties.
 See "--help mca" for more information.

Syntax in the file is:

param = value

with one parameter and its associated value per line. Empty lines and lines beginning
with the '#' character are ignored.
#
#  DEBUG-DAEMONS
#
[debug-daemons]
Debug daemon output enabled. This is a somewhat limited stream of information normally
used to simply confirm that the daemons started. Includes leaving the output streams
open.
#
#  DEBUG-DAEMONS-FILE
[debug-daemons-file]
Debug daemon output is enabled and all output from the daemons is redirected into
files named

   output-prted-<daemon-nspace>-<nodename>.log

to avoid conflict on shared file systems. The files are located in the top-level
session directory assigned to the DVM.

#include#help-schizo-cli#session-dir
#
#  LEAVE-SESSION-ATTACHED
#
[leave-session-attached]
Do not discard stdout/stderr of remote PRRTE daemons. The primary use for this option
is to ensure that the daemon output streams (i.e., stdout and stderr) remain open after
launch, thus allowing the user to see any daemon-generated error messages. Otherwise,
the daemon will "daemonize" itself upon launch, thereby closing its output streams.
#
#  PREFIX
#
[prefix]
Prefix to be used to look for PRRTE executables. PRRTE automatically sets the prefix
for remote daemons if it was either configured with the --enable-prte-prefix-by-default
option OR prte itself was executed with an absolute path to the prte cmd. This option
overrides those settings, if present, and forces use of the provided path.
#
#  NO-PREFIX
#
[noprefix]
Disable automatic --prefix behavior. PRRTE automatically sets the prefix for remote
daemons if it was either configured with the --enable-prte-prefix-by-default option
OR prte itself was executed with an absolute path to the prte cmd. This option disables
that behavior.
#
#  FORWARD-SIGNALS
#
[forward-signals]
Comma-delimited list of additional signals (names or integers) to forward to application
processes ["none" => forward nothing]. Signals provided by default include SIGTSTP,
SIGUSR1, SIGUSR2, SIGABRT, SIGALRM, and SIGCONT
#
#  ALLOW-RUN-AS-ROOT
#
[allow-run-as-root]
Allow execution as root (STRONGLY DISCOURAGED).

Running as root exposes the user to potentially catastrophic file system corruption
and damage - e.g., if the user accidentally points the root of the session directory
to a system required point, this directory and all underlying elements will be
deleted upon job completion, thereby rendering the system inoperable.

It is recognized that some environments (e.g., containers) may require operation as
root, and that the user accepts the risks in those scenarios. Accordingly, one can
override PRRTE's run-as-root protection by providing one of the following:

  * the "--allow-run-as-root" cmd line directive
  * adding BOTH of the following environmental parameters
    * PRTE_ALLOW_RUN_AS_ROOT=1
    * PRTE_ALLOW_RUN_AS_ROOT_CONFIRM=1

Again, we recommend this only be done if absolutely necessary.
#
#  X
#
[x]
Export an environment variable, optionally specifying a value. For example:

  * "-x foo" exports the environment variable foo and takes its value from the
    current environment
  * "-x foo=bar" exports the environment variable name foo and sets its value
    to "bar" in the started processes
  * "-x foo*" exports all current environmental variables starting with "foo"
#
#  STREAM-BUFFERING
#
[stream-buffering]
Adjust buffering for stdout/stderr
    [0 => unbuffered]
    [1 => line buffered]
    [2 => fully buffered]
#
#  PERSONALITY
#
[personality]
Specify the personality to be used. This governs selection of the plugin
responsible for defining and parsing the command line, harvesting and
forwarding environmental variables, and providing library-dependent
support to the launched processes. Examples include "ompi" for an
application compiled with Open MPI, "mpich" for one built against
the MPICH library, or "oshmem" for an OpenSHMEM application compiled
against SUNY's reference library.
#
#  RUNTIME-OPTIONS
#
[runtime-options]
The "runtime-options" command line directive must be accompanied by a
comma-delimited list of case-insensitive options that control the runtime
behavior of the job. The full directive need not be provided - only enough
characters are required to uniquely identify the directive.

Runtime options are typically "true" or "false", though this is not a
requirement on developers. Since the value of each option may need to
be set (e.g., to override a default set by MCA parameter), the syntax
of the command line directive includes the use of an '=' character to
allow inclusion of a value for the option. For example, one can set
the ABORT-NONZERO-STATUS option to "true" by specifying it as
"ABORT-NONZERO-STATUS=1". Note that boolean options can be set to "true"
using a non-zero integer or a case-insensitive string of the word "true".
For the latter representation, the user need only provide at least the
'T' character. The same policy applies to setting a boolean option to
"false".

Note that a boolean option will default to "true" if provided without
a value. Thus, "--runtime-options abort-nonzero" is sufficient to set the
"ABORT-NONZERO-STATUS" option to "true".

Supported values include:

-   ERROR-NONZERO-STATUS[=(bool)] if set to false, this directs the runtime
    to treat a process that exits with non-zero status as a normal termination.
    If set to true, the runtime will consider such an occurrence as an
    error termination and take appropriate action - i.e. the job will be
    terminated unless a runtime option directs otherwise. This option
    defaults to a true value if the option is given without a value.

-   DONOTLAUNCH directs the runtime to map but not launch the specified
    job. This is provided to help explore possible process placement patterns
    before actually starting execution. No value need be passed as this is
    not an option that can be set by default in PRRTE.

-   SHOW-PROGRESS[=(bool)] requests that the runtime provide progress reports
    on its startup procedure - i.e., the launch of its daemons in support
    of a job. This is typically used to debug DVM startup on large systems.
    This option defaults to a true value if the option is given without a value.

-   NOTIFYERRORS[=(bool)] if set to true, requests that the runtime provide
    a PMIx event whenever a job encounters an error - e.g., a process fails.
    The event is to be delivered to each remaining process in the job. This
    option defaults to a true value if the option is given without a value.
    See "--help notifications" for more detail as to the PMIx event codes
    available for capturing failure events.

-   RECOVERABLE[=(bool)] if set to true, this indicates that the application
    wishes to consider the job as recoverable - i.e., the application is
    assuming responsibility for recovering from any process failure. This
    could include application-driven spawn of a substitute process or
    internal compensation for the missing process. This option defaults to
    a true value if the option is given without a value.

-   AUTORESTART[=(bool)] if set to true, this requests that the runtime
    automatically restart failed processes up to "max restarts" number of
    times. This option defaults to a true value if the option is given
    without a value.

-   CONTINUOUS[=(bool)] if set to true, this informs the runtime that the
    processes in this job are to run until explicitly terminated. Processes
    that fail are to be automatically restarted up to "max restarts"
    number of times. Notification of process failure is to be delivered
    to all processes in the application. This is the equivalent of specifying
    RECOVERABLE, NOTIFYERRORS, and AUTORESTART options except that the runtime,
    not the application, assumes responsibility for process recovery. This
    option defaults to a true value if the option is given without a value.

-   MAX-RESTARTS=<int> indicates the maximum number of times a given process
    is to be restarted. This can be set at the application or job level (which
    will then apply to all applications in that job).

-   EXEC-AGENT=<path> indicates the executable that shall be used to start
    an application process. The resulting command for starting an application
    process will be "<path> app <app-argv>". The path may contain its own
    command line arguments.

-   DEFAULT-EXEC-AGENT directs the runtime to use the system default exec
    agent to start an application process. No value need be passed as this is
    not an option that can be set by default in PRRTE.

-   OUTPUT-PROCTABLE[(=channel)] directs the runtime to report the convential
    debugger process table (includes PID and host location of each process in
    the application). Output is directed to stdout if the channel is "-",
    stderr if "+", or into the specified file otherwise. If no channel is
    specified, output will be directed to stdout.

-   STOP-ON-EXEC directs the runtime to stop the application process(es) immediately
    upon exec'ing them. The directive will apply to all processes in the job.

-   STOP-IN-INIT indicates that the runtime should direct the application process(es)
    to stop in PMIx_Init. The directive will apply to all processes in the job.

-   STOP-IN-APP indicates that the runtime should direct application processes to
    stop at some application-defined place and notify they are ready-to-debug. The
    directive will apply to all processes in the job.

-   TIMEOUT=<string> directs the runtime to terminate the job after it has executed for
    the specified time. Time is specified in colon-delimited format - e.g.,
    "01:20:13:05" to indicate 1 day, 20 hours, 13 minutes and 5 seconds. Time specified
    without colons will be assumed to have been given in seconds.

-   SPAWN-TIMEOUT=<string> directs the runtime to terminate the job if job launch is
    not completed within the specified time. Time is specified in colon-delimited
    format - e.g., "01:20:13:05" to indicate 1 day, 20 hours, 13 minutes and 5 seconds.
    Time specified without colons will be assumed to have been given in seconds.

-   REPORT-STATE-ON-TIMEOUT[(=bool)] directs the runtime to provide a detailed report
    on job and application process state upon job timeout. This option defaults to a
    true value if the option is given without a value.

-   GET-STACK-TRACES[(=bool)] requests that the runtime provide stack traces on all
    application processes still executing upon timeout. This option defaults to a true
    value if the option is given without a value.

-   REPORT-CHILD-JOBS-SEPARATELY[(=bool)] directs the runtime to report the exit status
    of any child jobs spawned by the primary job separately. If false, then the final
    exit status reported will be zero if the primary job and all spawned jobs exit
    normally, or the first non-zero status returned by either primary or child jobs.
    This option defaults to a true value if the option is given without a value.

-   AGGREGATE-HELP-MESSAGES[(=bool)] directs the runtime to aggregate help messages,
    reporting each unique help message once accompanied by the number of processes
    that reported it. This option defaults to a true value if the option is given
    without a value.

-   FWD-ENVIRONMENT[(=bool)] directs the runtime to forward the entire local environment
    in support of the application. This option defaults to a true value if the option is
    given without a value.

The runtime-options command line option has no qualifiers. Note that directives
are case-insensitive.
#
# NOTIFICATIONS
#
[notifications]
PRRTE provides notifications on a variety of process and job states. Each notification
includes not only the PMIx event code that generated it, but also information on
the cause of the event to the extent to which this is known.

Supported job events include:

-   PMIX_READY_FOR_DEBUG indicates that all processes in the reported nspace have
    reached the specified debug stopping point.

-   PMIX_LAUNCH_COMPLETE indicates that the reported nspace has been launched - i.e.,
    the involved PRRTE daemons all report that their respective child processes
    have completed fork/exec.

-   PMIX_ERR_JOB_CANCELED indicates that the job was cancelled by user command, usually
    issued via an appropriate PMIx-enabled tool.

-   PMIX_ERR_JOB_FAILED_TO_LAUNCH indicates that the specified job failed to launch.
    This can be due to a variety of factors that include inability to find the
    executable on at least one involved node.


Supported process events include:

-   PMIX_ERR_PROC_TERM_WO_SYNC indicates that at least one process in the job called
    PMIx_Init, thus indicating some notion of a global existence, and at least one
    process in the job subsequently exited without calling PMIx_Finalize. This usually
    indicates a failure somewhere in the application itself that precluded an orderly
    shutdown of the process. Notification will include the process ID that exited
    in this manner.

-   PMIX_EVENT_PROC_TERMINATED indicates that the reported process terminated normally.
    Notification will include the process ID that exited and its exit status.

-   PMIX_ERR_PROC_KILLED_BY_CMD indicates that the reported process was killed by
    PRRTE command. This typically occurs in response to a Ctrl-C (or equivalent)
    being applied to the PRRTE launcher, thereby instructing PRRTE to forcibly
    terminate its processes. The event currently will only be issued in the
    case where forcible termination is commanded via a tool that can pass the
    process IDs that are specifically to be terminated - otherwise, in the case
    of the Ctrl-C event previously described, all processes in the job will be
    terminated, leaving none to be notified. Notification will include the process
    ID that was terminated.

-   PMIX_ERR_PROC_SENSOR_BOUND_EXCEEDED indicates that the specified process
    exceeded a previously-set sensor boundary - e.g., it may have grown beyond
    a defined memory limit. Such events may or may not automatically trigger
    termination by command, depending upon the behavior of the sensor. Notification
    will include the process ID that exceeded the sensor boundary plus whatever
    information the sensor provides regarding measurements and bounds.

-   PMIX_ERR_PROC_ABORTED_BY_SIG indicates that the specified process was killed
    by a signal - e.g., a segmentation fault/violation or an externally applied
    signal. Notifications will include the process ID that was killed and the
    corresponding reported signal.

-   PMIX_ERR_PROC_REQUESTED_ABORT indicates that the specified process has aborted
    by calling the PMIx_Abort function. Notification will include the process ID
    that called abort and its exit status.

-   PMIX_ERR_EXIT_NONZERO_TERM indicates that the specified process terminated with
    a non-zero exit status. This notification is only generated in the case where
    the runtime option ERROR-NONZERO-STATUS is set to true, thereby indicating that
    a process exiting with non-zero status is to be considered an error. As PRRTE can
    be overwhelmed by a large job where every process exits with a non-zero status,
    only the _first_ process in a given job that exits with a non-zero status will
    generate a notification unless the RECOVERABLE runtime option is also provided
    as otherwise the job will be immediately terminated. Notifications will include
    the process ID that exited and the status it returned.

-   PMIX_ERR_PROC_RESTART indicates that the specified process has been restarted.
    Additional information may include the hostname where the process is now
    executing.
#
#  DVM
#
[dvm]
A required argument is passed to the "--dvm" directive to specify the
location of the DVM controller (e.g., "--dvm pid:12345") or by passing
the string "search" to instead search for an existing controller.

Supported options include:

search          directs the tool to search for available DVM controllers
                it is authorized to use, connecting to the first such
                candidate it finds.

pid:<arg>       provides the PID of the target DVM controller. This can
                be given as either the PID itself (arg = int) or the path
                to a file that contains the PID (arg = file:<path>)

file:<path>     provides the path to a PMIx rendezvous file that is
                output by PMIx servers - the file contains all the
                required information for completing the connection

uri:<arg>       specifies the URI of the DVM controller, or the name of
                the file (specified as file:filename) that contains that info

ns:<arg>        specifies the namespace of the DVM controller

system          exclusively find and use the system-level DVM controller

system-first    look for a system-level DVM controller, fall back to searching
                for an available DVM controller the command is authorized to
                use if a system-level controller is not found

Examples:
  <command> --dvm file:dvm_uri.txt --np 4 ./a.out

  <command> --dvm pid:12345 --np 4 ./a.out

  <command> --dvm uri:file:dvm_uri.txt --np 4 ./a.out

  <command> --dvm ns:prte-node1-2095 --np 4 ./a.out

  <command> --dvm pid:file:prte_pid.txt --np 4 ./a.out

  <command> --dvm search --np 4 ./a.out
