.. _man3-PMIx_server_init:

PMIx_server_init
================

.. include_body

``PMIx_server_init`` |mdash| Initialize the PMIx server support library.


SYNOPSIS
--------

.. code-block:: c

   #include <pmix_server.h>

   pmix_status_t PMIx_server_init(pmix_server_module_t *module,
                                  pmix_info_t info[], size_t ninfo);


Python Syntax
^^^^^^^^^^^^^

.. code-block:: python3

  from pmix import *

  foo = PMIxServer()
  # the directives is a list of Python ``pmix_info_t`` dictionaries
  pydirs = [{'key': PMIX_SERVER_NSPACE,
             'value': "SERVER", 'val_type': PMIX_STRING}]
  # map is a dictionary of server-module-function keys to Python
  # callback implementations (e.g. {'clientconnected': myconnfn, ...})
  rc = foo.init(pydirs, map)


INPUT PARAMETERS
----------------

* ``module``: Pointer to a :ref:`pmix_server_module_t(5)
  <man5-pmix_server_module_t>` structure containing the
  host environment's callback functions |mdash| the "function-shipped" server
  module through which client requests are relayed to the resource manager.
  Any function the host does not support is indicated by leaving its function
  pointer ``NULL``; client calls to an unsupported operation return a
  "not supported" error. Passing ``NULL`` for ``module`` (or an empty
  structure) is permitted and indicates that the host will not support
  multi-node operations such as :ref:`PMIx_Fence(3) <man3-PMIx_Fence>`, but
  still intends to give local clients access to job information.
* ``info``: Pointer to an array of :ref:`pmix_info_t(5) <man5-pmix_info_t>`
  structures conveying directives that qualify server initialization (see
  `DIRECTIVES`_). A ``NULL`` value is supported when no directives are desired.
* ``ninfo``: Number of elements in the ``info`` array.


DESCRIPTION
-----------

Initialize the PMIx server support library and register the host environment's
callback module. ``PMIx_server_init`` is called by the process that hosts the
PMIx server |mdash| typically a resource manager daemon, launcher, or tool
that will accept client and/or tool connections.

The PMIx server provides a *function-shipping* approach to the server side of
the protocol: each client-visible PMIx operation is mirrored by an entry in the
``pmix_server_module_t`` structure. When a client request arrives, the PMIx
server library invokes the corresponding host function so that resource
managers can implement server behavior without being burdened with PMIx
internal details. For performance and scalability, host module functions are
required to return quickly and to execute their work asynchronously, completing
each operation by invoking the callback function supplied to them.

.. important:: A host must run against a PMIx library of the same major
   version as the PMIx headers it was compiled against. The library takes
   the host's ``pmix_server_module_t`` to be the size and layout defined by
   its own headers, and expects the host's functions to behave as that
   version specifies. It cannot check either. New members are added to the
   structure only in a new major release. A host that runs against a
   library of a different major version is not supported, and its behavior
   is undefined.

Ownership rules for the module functions are important: all data passed *to* a
host server function is owned by the PMIx server library and must not be freed
by the host, while data returned by the host via a module callback is owned by
the host and may be released once the callback returns.

The ``info`` array supplies additional information the server may need when
initializing |mdash| for example, the namespace and rank to assign to the
server itself, the temporary directory in which to place the rendezvous socket,
or the ``PMIX_SERVER_TOOL_SUPPORT`` directive announcing that the daemon is
willing to accept connection requests from tools (see `DIRECTIVES`_).

The PMIx server library is reference counted. Repeated calls to
``PMIx_server_init`` are permitted and simply increment the reference count;
if a ``module`` is provided on a later call and none had been set previously,
the library adopts it. Each successful call must be balanced by a matching call
to :ref:`PMIx_server_finalize(3) <man3-PMIx_server_finalize>`.

``PMIx_server_init`` is a blocking call that completes synchronously; it does
not take a callback.

.. warning:: Use PMIx with caution in a setuid or setgid program. PMIx
   takes its configuration from the environment and from locations the
   environment names: the ``PMIX_*`` variables (including those that name
   the server to connect to and the directories components are loaded
   from), a per-user parameter file under ``$HOME``, and the session
   directory under ``$TMPDIR``. In a setuid or setgid program these are
   set by the user who ran it, not by the program's owner. PMIx cannot
   tell the difference, and it cannot ignore them, because the
   environment is how a process finds its server. A setuid or setgid
   program may need to set or clear these variables itself before calling
   this function.


DIRECTIVES
----------

The following attributes are relevant to this operation. Support for any given
attribute is optional and depends on how the PMIx implementation was built and
on the capabilities of the host environment. All are passed in the ``info``
array.

Several behaviors need more than one attribute, or a build option or site
setting as well: serving tools, being found as a scheduler, taking clients of
earlier releases. :ref:`label-settings-matrix` sets out each combination and
what it does, and :ref:`label-host-env-starting` what to pass for each goal.

Server identity and directories
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

* ``PMIX_SERVER_NSPACE`` (char\*) |mdash| namespace to assign to this PMIx
  server. If not given, the ``PMIX_SERVER_NSPACE`` environment variable is
  used - a launcher that starts the server itself sets it - and otherwise the
  namespace is ``pmix-server``.
* ``PMIX_SERVER_RANK`` (pmix_rank_t) |mdash| rank to assign to this PMIx server.
  If not given, the ``PMIX_SERVER_RANK`` environment variable is used, and
  otherwise the server's pid.
* ``PMIX_SERVER_TMPDIR`` (char\*) |mdash| temporary directory in which the
  server is to place its session-level files. The rendezvous files tools use
  to find it go under ``PMIX_SYSTEM_TMPDIR`` instead (see `Rendezvous
  files`_). If not given, the
  ``PMIX_SERVER_TMPDIR`` environment variable is used, then ``TMPDIR``,
  ``TEMP`` and ``TMP``, and finally ``/tmp``. The server's local socket is
  made here too, and its path must fit a socket address (see `The local
  socket`_). The directory must already exist. Its path is checked at init,
  with the system tmpdir's: see `The tmpdirs`_.
* ``PMIX_SYSTEM_TMPDIR`` (char\*) |mdash| temporary directory beneath which
  the server places its rendezvous files: those tools find it by, in
  ``pmix-user.<uid>``, and its system-level file (system server, scheduler,
  system controller), which a root server given no system tmpdir puts in
  ``/run/pmix`` instead. If not given, the ``PMIX_SYSTEM_TMPDIR``
  environment variable is used, then the same fallbacks as for
  ``PMIX_SERVER_TMPDIR``.

Role and support attributes
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

* ``PMIX_SERVER_TOOL_SUPPORT`` (bool) |mdash| the host is willing to accept
  connection requests from tools, and rendezvous files naming the server by PID
  and by namespace are written (see `Rendezvous files`_). Without it, every
  tool is refused with ``PMIX_ERR_NOT_SUPPORTED`` - including one a client or
  job script starts, which has the server's address in its environment. The
  files are readable only by the host's user ID, and tools running under
  other user IDs are refused, unless ``PMIX_SERVER_ALLOW_FOREIGN_TOOLS`` is
  also given; such tools also need the server's local socket widened to them
  (``PMIX_SOCKET_MODE``, ``PMIX_USERID``, ``PMIX_GRPID``), which a server
  does not do by itself. Tools of root, and of the user named by
  ``PMIX_USERID``, are not refused this way.
* ``PMIX_SERVER_SYSTEM_SUPPORT`` (bool) |mdash| the server is the node's
  system server: tools asking for the system server find it through a
  system-level rendezvous file. Takes effect only together with
  ``PMIX_SERVER_TOOL_SUPPORT``. As for that attribute, the file is readable
  only by the host's user ID unless ``PMIX_SERVER_ALLOW_FOREIGN_TOOLS`` is
  given.
* ``PMIX_SERVER_SESSION_SUPPORT`` (bool) |mdash| the server is the session
  server of its session, found by tools through a rendezvous file named by
  ``PMIX_SESSION_RNDZ_ID``, else by ``PMIX_SESSION_ID``: without either
  ``PMIx_server_init`` fails. Servers of different
  sessions share a node; a second server of one session on a node fails
  ``PMIx_server_init``. Takes effect only together with
  ``PMIX_SERVER_TOOL_SUPPORT``.
* ``PMIX_SESSION_RNDZ_ID`` (uint32_t) |mdash| the ID naming the session
  server's rendezvous file, unique among the session servers on a node; see
  ``PMIX_SERVER_SESSION_SUPPORT``. The host chooses it, so that it need
  not change ``PMIX_SESSION_ID``, the session's ID as the scheduler
  assigned it, to name the file - which is named by ``PMIX_SESSION_ID``
  when this is not given.
* ``PMIX_SERVER_GATEWAY`` (bool) |mdash| the server is acting as a gateway that
  can relay operations (e.g., logging) requiring routing to other nodes.
* ``PMIX_SERVER_SCHEDULER`` (bool) |mdash| the server is hosting the system
  scheduler (workload manager) and expects to receive scheduler-related
  requests. The runtimes that find it connect as tools, so it implies
  ``PMIX_SERVER_TOOL_SUPPORT``; passing that attribute false as well
  fails the init.
* ``PMIX_SERVER_SYS_CONTROLLER`` (bool) |mdash| the server is hosting the
  system controller. As for ``PMIX_SERVER_SCHEDULER``, the processes that
  find it are tools: it implies ``PMIX_SERVER_TOOL_SUPPORT``, and passing
  that attribute false as well fails the init.
* ``PMIX_SERVER_LEGACY_PEERS`` (bool) |mdash| take (or refuse) connections
  from clients and tools of PMIx releases before v7.0, which connect over
  TCP. Overrides the ``ptl_base_legacy_clients`` MCA parameter (and
  ``ptl_base_legacy_peers``, which sets it in a build with legacy clients), which defaults to ``false``. With it, the server listens on a loopback
  interface for them and gives its clients the TCP address in the
  variables those releases read; without it the server has no TCP
  listener, and a client or tool of an earlier release cannot connect. Only a PMIx
  configured ``--enable-legacy-clients`` can take them (the default is
  not to): in one without it, asking fails the init - by this attribute, or
  by ``ptl_base_legacy_clients``; ``ptl_base_legacy_peers`` asks such a
  build for nothing. A host that does not pass it leaves the decision to
  the site's parameter.
* ``PMIX_SERVER_REMOTE_CONNECTIONS`` (bool) |mdash| **deprecated**. A server
  takes connections from its own host only: ``true`` fails the init, and
  ``false`` changes nothing. A client or tool does not dial a server at an
  address that is not its own host's, and the server refuses a TCP peer at
  such an address.
* ``PMIX_SERVER_ALLOW_FOREIGN_TOOLS`` (bool) |mdash| mark the rendezvous
  files as readable by all users and allow tools running under user IDs other
  than that of the server to connect. Defaults to ``false``: without it, a
  connection request from a tool under another user ID is refused. The host
  retains ultimate authority over such connections and may restrict what
  foreign tools are permitted to do (for example, limiting them to queries) as
  dictated by host policy. The permissions of an existing directory the files
  are placed in are never changed, so a directory other users cannot search
  still keeps their tools out (see `Rendezvous files`_).

Connection attributes
^^^^^^^^^^^^^^^^^^^^^

* ``PMIX_SOCKET_MODE`` (uint32_t) |mdash| the mode (nine permission bits)
  of the server's local socket. Default 0600: only the server's user can
  connect. See `The local socket`_.
* ``PMIX_USERID`` (uint32_t, or a user name as char\*) |mdash| the owner
  of the server's local socket. Default: the server's user. Its directory
  stays the server's. Tools of this user are taken without
  ``PMIX_SERVER_ALLOW_FOREIGN_TOOLS``, as the server's own user's are.
* ``PMIX_GRPID`` (uint32_t, or a group name as char\*) |mdash| the group
  of the server's local socket and of its directory. Default: the
  server's effective group.
* ``PMIX_LAUNCHER_RENDEZVOUS_FILE`` (char\*) |mdash| full pathname of an
  additional rendezvous file in which the server is to write its contact
  information |mdash| for example, so that a debugger can find a launcher.
  Missing directories on the path are created. The file is written whether or
  not ``PMIX_SERVER_TOOL_SUPPORT`` is given; a tool that reads it is accepted
  only with that attribute.
* ``PMIX_CONNECT_TO_SYSTEM``, ``PMIX_CONNECT_SYSTEM_FIRST``,
  ``PMIX_CONNECT_TO_SCHEDULER``, ``PMIX_CONNECT_TO_SYS_CONTROLLER`` (bool),
  ``PMIX_CONNECTION_ORDER`` (char\*), ``PMIX_SERVER_PIDINFO`` (pid_t),
  ``PMIX_TOOL_ATTACHMENT_FILE`` (char\*) |mdash| connect the server to the
  server these name, as for :ref:`PMIx_tool_init(3) <man3-PMIx_tool_init>`,
  and make it this server's own server |mdash| the one, for example, an
  allocation request is sent to. A malformed value fails the init. With
  ``PMIX_TOOL_CONNECT_OPTIONAL`` a server that cannot be reached is not an
  error.
* ``PMIX_TCP_REPORT_URI`` (char\*) |mdash| report the server's URI: ``-``
  writes the local socket's URI to stdout and ``+`` to stderr; ``tcp-`` and
  ``tcp+`` write the TCP URI instead, for a reader of a release before v7.0,
  and fail the init when the server takes no legacy clients and so has no TCP
  listener. An integer writes to that open file descriptor, which is then
  closed: the TCP URI's string and then the local socket's, each ending in a
  NUL, when the server takes legacy clients, else the local socket's alone.
  Anything else is taken as the name of a file to write it to.
* ``PMIX_TCP_IF_INCLUDE`` (char\*) |mdash| comma-delimited list of devices
  and/or CIDR notation to listen on. Cannot be combined with
  ``PMIX_TCP_IF_EXCLUDE``.
* ``PMIX_TCP_IF_EXCLUDE`` (char\*) |mdash| comma-delimited list of devices
  and/or CIDR notation not to listen on.
* ``PMIX_TCP_IPV4_PORT`` (int or char\*) |mdash| IPv4 port to listen on, or
  (as a string) a comma-delimited list of ports and ranges to try in order.
  ``0`` or ``-1`` lets the kernel choose.
* ``PMIX_TCP_IPV6_PORT`` (int or char\*) |mdash| as ``PMIX_TCP_IPV4_PORT``,
  for IPv6.
* ``PMIX_TCP_DISABLE_IPV4`` (bool) |mdash| do not listen on IPv4 addresses.
* ``PMIX_TCP_DISABLE_IPV6`` (bool) |mdash| do not listen on IPv6 addresses.

Topology and behavior attributes
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

* ``PMIX_TOPOLOGY2`` (pmix_topology_t) |mdash| pointer to an
  implementation-specific description of the local node topology.
* ``PMIX_SERVER_SHARE_TOPOLOGY`` (bool) |mdash| the server is to scalably
  expose the node topology to its local clients (for example, via shared-memory
  backing stores), cleaning up any artifacts at finalize.
* ``PMIX_HOMOGENEOUS_SYSTEM`` (bool) |mdash| the nodes in the system are
  topologically identical, so the server need not compute or exchange
  per-node topology descriptions.
* ``PMIX_SERVER_ENABLE_MONITORING`` (bool) |mdash| enable the internal
  monitoring capabilities of the PMIx server.
* ``PMIX_EXTERNAL_PROGRESS`` (bool) |mdash| the host will progress the PMIx
  library as needed via calls to ``PMIx_Progress`` rather than PMIx spawning
  its own internal progress thread.
* ``PMIX_SINGLETON`` (char\*) |mdash| the server is operating in support of a
  singleton process; the value is the ``nspace.rank`` string of that singleton.
* ``PMIX_IOF_LOCAL_OUTPUT`` (bool) |mdash| write output streams to the server's
  own stdout/stderr.
* ``PMIX_BIND_PROGRESS_THREAD`` (char\*) |mdash| comma-delimited ranges of CPUs to
  which the internal PMIx progress thread is to be bound.
* ``PMIX_BIND_REQUIRED`` (bool) |mdash| return an error if the internal PMIx
  progress thread cannot be bound as requested.
* ``PMIX_EXTERNAL_AUX_EVENT_BASE`` (void\*) |mdash| pointer to an ``event_base``
  the library is to use for auxiliary functions (e.g., capturing signals) that
  would otherwise interfere with the host.


RETURN VALUE
------------

Returns ``PMIX_SUCCESS`` on success. On error, a negative value corresponding
to a PMIx error constant is returned, including:

* ``PMIX_ERR_INIT`` |mdash| the PMIx server library could not be initialized
  (for example, a required transport was explicitly disabled, or a tight race
  between concurrent ``PMIx_server_init`` calls left the library in an unusable
  state).
* ``PMIX_ERR_NOT_SUPPORTED`` |mdash| a provided directive requested behavior the
  implementation does not support.
* ``PMIX_ERR_BAD_PARAM`` |mdash| a directive or an environmental value carried
  something the implementation cannot use |mdash| for example a
  ``PMIX_SINGLETON`` that is not of the form ``"nspace.rank"``, or a
  ``PMIX_SERVER_RANK`` environment variable that is not a simple non-negative
  number.

Any other negative value indicates an appropriate error condition. PMIx error
constants are defined in ``pmix_common.h``.

A failed ``PMIx_server_init`` leaves nothing behind. Anything the call had
already brought up is torn down before the error is returned, so the caller may
correct whatever was rejected and call ``PMIx_server_init`` again. It must
**not** call :ref:`PMIx_server_finalize(3) <man3-PMIx_server_finalize>`, which
answers ``PMIX_ERR_INIT`` because no initialization completed. The one
exception is a failure of the underlying runtime itself, which cannot be
unwound; there the library reports the error and is unusable for the life of
the process.


NOTES
-----

``PMIx_server_init`` is intended for use only by processes acting as a PMIx
*server* |mdash| processes that host the server support library on behalf of a
resource manager or launcher. Client processes call
:ref:`PMIx_Init(3) <man3-PMIx_Init>` and tools call
:ref:`PMIx_tool_init(3) <man3-PMIx_tool_init>` instead.

After a successful ``PMIx_server_init``, the host is expected to register each
participating namespace with
:ref:`PMIx_server_register_nspace(3) <man3-PMIx_server_register_nspace>` and
each local client with
:ref:`PMIx_server_register_client(3) <man3-PMIx_server_register_client>` before
launching that client, then set up the client's environment with
``PMIx_server_setup_fork``.


The local socket
^^^^^^^^^^^^^^^^

Every server listens on a local (``AF_UNIX``) socket, where the clients and
tools of this release on its host connect. It listens on TCP, on a loopback
interface, only for peers of earlier releases (``PMIX_SERVER_LEGACY_PEERS``).
The kernel identifies the process at
each end of a connection to the local socket: the server knows each peer's
uid, gid and pid, and each peer can check the server's uid and gid.

The socket is ``pmix-<pid>``, in a directory of its own the server makes
beneath ``PMIX_SERVER_TMPDIR``: ``<tmpdir>/pmix-XXXXXX``. By default the
directory is mode 0700 and the socket 0600, so only the server's user can
connect. A host that wants other users' processes to connect - a client it
starts under another user ID, a tool it allows with
``PMIX_SERVER_ALLOW_FOREIGN_TOOLS`` - must widen it: ``PMIX_SOCKET_MODE``
sets the socket's mode, and ``PMIX_USERID`` and ``PMIX_GRPID`` its owner and
group. The directory stays the server's: it is searchable by whom the mode
lets connect - and by everyone, when the socket is given to another user -
and readable by no one but the server's user. The server never widens it on
its own.

The socket's path must fit a socket address - about 100 characters. A
temporary directory too long for it makes ``PMIx_server_init`` fail; there
is no fallback. The default ``$TMPDIR`` on macOS is too long: pass a shorter
``PMIX_SERVER_TMPDIR``.

The server removes the socket and its directory at finalize. A server that
is killed leaves them behind, as ``pmix-XXXXXX`` directories in its
temporary directory.

``PMIx_server_setup_fork`` gives a child the socket's address and the
server's uid and gid in ``PMIX_SERVER_LOCAL_URI``; a child of this release
connects there and nowhere else, and only to a server with that uid and gid.

Rendezvous files
^^^^^^^^^^^^^^^^

A tool finds a server through a *rendezvous file*: a small file holding the
server's URI, version and PID, and the address of its local socket. Which ones are written depends on the
directives:

.. list-table::
   :header-rows: 1

   * - Directive
     - File
     - Directory
   * - ``PMIX_LAUNCHER_RENDEZVOUS_FILE``
     - the name given
     - as given
   * - ``PMIX_SERVER_SCHEDULER``
     - ``pmix.sched.<hostname>``
     - the system directory (below)
   * - ``PMIX_SERVER_SYS_CONTROLLER``
     - ``pmix.sysctrlr.<hostname>``
     - the system directory
   * - ``PMIX_SERVER_TOOL_SUPPORT`` with ``PMIX_SERVER_SYSTEM_SUPPORT``
     - ``pmix.sys.<hostname>``
     - the system directory
   * - ``PMIX_SERVER_TOOL_SUPPORT`` with ``PMIX_SERVER_SESSION_SUPPORT``
     - ``pmix.<hostname>.session.<rendezvous ID>``
     - ``<PMIX_SYSTEM_TMPDIR>/pmix-user.<uid>``
   * - ``PMIX_SERVER_TOOL_SUPPORT``
     - ``pmix.<hostname>.tool.<pid>`` and ``pmix.<hostname>.tool.<nspace>``
     - ``<PMIX_SYSTEM_TMPDIR>/pmix-user.<uid>``

A scheduler or system controller writes only its own file; the files under
``PMIX_SERVER_TOOL_SUPPORT`` are written by any other server that gives it.

The tool files go in a directory of the server's user's own, whatever the
server's session tmpdir: ``pmix-user.<uid>`` beneath the system tmpdir. The
server makes it mode 0700 - 0711 with ``PMIX_SERVER_ALLOW_FOREIGN_TOOLS``, so
other users' tools can open the files at their names but not list them. One
already there is used only if it is a directory of the server's user, not a
link, that no other user can write; anything else is reported, and the server
runs on without writing its tool files. The system directory is
``PMIX_SYSTEM_TMPDIR`` when the host names one; otherwise ``/run/pmix`` for a
root server (``/var/run/pmix`` where there is no ``/run``), and the server's
``pmix-user`` directory for any other.

By default each file is readable only by the server's user ID. With
``PMIX_SERVER_ALLOW_FOREIGN_TOOLS`` it is also readable by all users.

A file at the same name that no running server holds is removed and replaced.
One a running server holds belongs to that server, and ``PMIx_server_init``
fails rather than take it over: one system server, scheduler or system
controller per node, one session server per session on a node, and one server
per namespace. A server whose host names no namespace takes
``pmix-server-<hostname>-<pid>``.

A file named by ``PMIX_TCP_REPORT_URI`` is not a rendezvous file and none of
the above applies to it. A file already at that name is reused only if the
server's user owns it; one of any other user's - root's too, unless the server
is root - fails ``PMIx_server_init``. The file, reused or new, is given mode
0644 - readable by all, writable only by the server's user - and is removed at
finalize.

The tmpdirs
^^^^^^^^^^^

Before it makes anything in them, the server checks the path of its session
and system tmpdirs, links resolved, from ``/``. Every directory above a tmpdir
must be owned by root, the server's user, or a user from outside the server's
user namespace, and must not be writable by anyone else unless it has the
sticky bit, or its group's only member is its owner. The tmpdir itself may be
any user's - a root server is often given a directory it made for the job's
user - provided only its owner can write it; or else it must pass the rule
for the directories above, as ``/tmp`` does. Otherwise another user could
move the server's directories aside, and its clients would fail to connect.

A path that fails makes ``PMIx_server_init`` fail, with a message naming the
directory. The ``ptl_base_tmpdir_check`` MCA parameter changes that:
``refuse`` (the default), ``warn`` - report it and continue - or ``silent``,
not to check. The last two are for a directory no other user shares that the
rule misjudges, such as a Kubernetes ``emptyDir`` mounted at ``/tmp``.


.. seealso::
   :ref:`PMIx_server_finalize(3) <man3-PMIx_server_finalize>`,
   :ref:`PMIx_server_register_nspace(3) <man3-PMIx_server_register_nspace>`,
   :ref:`PMIx_server_register_client(3) <man3-PMIx_server_register_client>`,
   :ref:`PMIx_Init(3) <man3-PMIx_Init>`,
   :ref:`PMIx_tool_init(3) <man3-PMIx_tool_init>`,
   :ref:`pmix_server_module_t(5) <man5-pmix_server_module_t>`,
   :ref:`pmix_info_t(5) <man5-pmix_info_t>`,
   :ref:`pmix_status_t(5) <man5-pmix_status_t>`
