.. _man5-pmix_server_module_t:

pmix_server_module_t
====================

.. include_body

`pmix_server_module_t` |mdash| The set of callback functions a host
environment provides to the PMIx server library.

SYNTAX
------

C Syntax
^^^^^^^^

.. code-block:: c

   #include <pmix_server.h>

   typedef struct pmix_server_module_4_0_0_t {
       /* v1x interfaces */
       pmix_server_client_connected_fn_t   client_connected; // DEPRECATED
       pmix_server_client_finalized_fn_t   client_finalized;
       pmix_server_abort_fn_t              abort;            // DEPRECATED
       pmix_server_fencenb_fn_t            fence_nb;
       pmix_server_dmodex_req_fn_t         direct_modex;
       pmix_server_publish_fn_t            publish;
       pmix_server_lookup_fn_t             lookup;
       pmix_server_unpublish_fn_t          unpublish;
       pmix_server_spawn_fn_t              spawn;
       pmix_server_connect_fn_t            connect;
       pmix_server_disconnect_fn_t         disconnect;
       pmix_server_register_events_fn_t    register_events;
       pmix_server_deregister_events_fn_t  deregister_events;
       pmix_server_listener_fn_t           listener;
       /* v2x interfaces */
       pmix_server_notify_event_fn_t       notify_event;
       pmix_server_query_fn_t              query;
       pmix_server_tool_connection_fn_t    tool_connected; // DEPRECATED
       pmix_server_log_fn_t                log;            // DEPRECATED
       pmix_server_alloc_fn_t              allocate;
       pmix_server_job_control_fn_t        job_control;
       pmix_server_monitor_fn_t            monitor;
       /* v3x interfaces */
       pmix_server_get_cred_fn_t           get_credential;
       pmix_server_validate_cred_fn_t      validate_credential;
       pmix_server_iof_fn_t                iof_pull;
       pmix_server_stdin_fn_t              push_stdin;
       /* v4x interfaces */
       pmix_server_grp_fn_t                group;
       pmix_server_fabric_fn_t             fabric;
       /* v6x interfaces */
       pmix_server_client_connected2_fn_t  client_connected2;
       pmix_server_tool_connection2_fn_t   tool_connected2;
       pmix_server_log2_fn_t               log2;
       /* pending interfaces */
       pmix_server_session_control_fn_t    session_control;
       pmix_server_resource_block_fn_t     resource_block;
       /* v7x interfaces */
       pmix_server_abort2_fn_t             abort2;

   } pmix_server_module_t;


DESCRIPTION
-----------

The `pmix_server_module_t` structure is the primary integration point between
the PMIx server library and its host environment |mdash| a resource manager,
launcher, or other system-management-stack (SMS) element. It is a table of
function pointers, one per PMIx server operation, that the host implements and
passes by pointer to :ref:`PMIx_server_init(3) <man3-PMIx_server_init>`.

PMIx uses a "function-shipping" model: each PMIx client request that requires
host action is mirrored by an entry in this structure. When the PMIx server
library receives such a request from a local client (or tool), it invokes the
corresponding callback in the host's module, handing the host the parameters
needed to service the request. This lets a resource manager implement the
server role without any knowledge of PMIx internals.

**Unimplemented callbacks may be left NULL.** A host is not required to support
every operation. Any function the host does not provide is indicated by a NULL
pointer in the structure; the library detects the NULL entry and returns
``PMIX_ERR_NOT_SUPPORTED`` to the requesting client for that operation rather
than dereferencing the pointer (see, for example, ``pmix_server_publish`` in
``src/server/pmix_server_ops.c``, which returns ``PMIX_ERR_NOT_SUPPORTED`` when
``NULL == pmix_host_server.publish``). This behavior is documented at the top of
``include/pmix_server.h``: "Any functions not supported by the RM can be
indicated by a NULL for the function pointer. Client calls to such functions
will have a 'not supported' error returned."

**Callbacks are asynchronous.** For performance and scalability, the host is
required to return from every callback as quickly as possible, performing the
actual work asynchronously. Nearly all callbacks are therefore non-blocking:
they accept a completion callback function (``cbfunc``) and an opaque
``cbdata`` pointer, and the host is expected to invoke ``cbfunc`` with the
result once the operation completes. The type of the completion function varies
by operation (for example, ``pmix_op_cbfunc_t`` for simple status returns,
``pmix_modex_cbfunc_t`` for fence/modex data, ``pmix_info_cbfunc_t`` for query,
allocate, job-control, monitor, group, fabric, and session operations, and so
on).

**Return-value contract.** The value a callback returns tells the library how
completion will be signaled:

* ``PMIX_SUCCESS`` |mdash| the request has been accepted and is being processed
  asynchronously. The host **will** invoke the supplied ``cbfunc`` later with
  the final result.
* ``PMIX_OPERATION_SUCCEEDED`` |mdash| the operation completed immediately and
  successfully. The host **must not** invoke ``cbfunc``; the library treats the
  operation as already finished. **Not permitted where the operation's result
  can only be delivered through the callback** |mdash| see below.
* Any other (error) value |mdash| the request failed and the host **will not**
  invoke ``cbfunc``. The library reports the error to the client directly and
  releases any state it was holding for the request.

.. important:: ``PMIX_OPERATION_SUCCEEDED`` is **not** an allowed return from
               ``spawn``, ``lookup``, ``direct_modex``, ``get_credential`` or
               ``validate_credential``, and is treated as an error by the
               library. Each of those operations produces a result |mdash| the
               namespace of the launched job, the published data that was
               asked for, the requested data blob, and the credential or its
               validation |mdash| that the return code cannot carry, so the
               only channel for it is ``cbfunc``. The library always supplies a
               non-``NULL`` ``cbfunc`` to all of them, so a host that has
               completed the work immediately loses nothing by using it:
               **invoke the callback, which may be done before returning, and
               return ``PMIX_SUCCESS``**. A host that returns
               ``PMIX_OPERATION_SUCCEEDED`` from any of them receives a
               diagnostic naming the operation, and the request is failed to
               its requestor with ``PMIX_ERR_NOT_SUPPORTED`` rather than being
               reported as a success carrying no result.

**Data ownership.** All data passed into a host callback is owned by the PMIx
server library and must not be freed by the host. Conversely, data the host
returns through a completion callback is owned by the host, which may release it
once the callback returns.

**The requester's identity.** When the library makes an up-call on behalf of a
client or tool, it tells the host who is asking, so that the host can decide
what that process may see or do:

* The requester's ``pmix_proc_t`` is passed as the up-call's process argument.
  ``iof_pull`` and ``register_events`` have no process argument, so for them
  the requester is named in the directives or info with ``PMIX_REQUESTOR``,
  in place of any ``PMIX_REQUESTOR`` the requester supplied.
* The up-call's info or directives array carries exactly one ``PMIX_USERID``
  and one ``PMIX_GRPID``, and at most one ``PMIX_REQUESTOR_PID`` and one
  ``PMIX_REQUESTED_GRPID``. For ``query`` they are in the qualifiers of every
  query.
* ``PMIX_USERID`` and ``PMIX_GRPID`` are always the user and group IDs
  established when the process connected. Any the requester put in its own
  request are removed. These describe the process connected to this server
  |mdash| a relay, if the request was relayed |mdash| and are what the host
  authorizes on.
* ``PMIX_REQUESTOR_PID`` is the process ID the kernel reported for the
  requester's connection. It is included only where the kernel reports one
  (the local transport, on platforms that do), never for a peer connected
  over TCP. Any the requester supplied is removed.
* ``PMIX_REQUESTED_GRPID`` is the group the requester asked to have the work
  charged to, if its request names one |mdash| typically the group, or
  account, the work is to be charged to. The requester may name it with
  ``PMIX_REQUESTED_GRPID``, or with ``PMIX_GRPID`` as earlier releases
  expected, and as a number or a group name; the library resolves a name and
  always passes a ``uint32_t`` to the host. A group that cannot be resolved
  fails the request (``PMIX_ERR_NOT_FOUND`` for an unknown name) rather than
  charging the work elsewhere, and a request that names more than one is
  refused (``PMIX_ERR_BAD_PARAM``). The library does not check that the
  requester belongs to a group it names: the host decides whether the
  requester may use it, and refuses the request (for example with
  ``PMIX_ERR_NO_PERMISSIONS``) if not.
* A host relaying a request names the process that first made it with
  ``PMIX_ORIGIN_USERID``, ``PMIX_ORIGIN_GRPID``, ``PMIX_ORIGIN_PID``,
  ``PMIX_ORIGIN_IS_TOOL`` and ``PMIX_ORIGIN_IS_CLIENT``. The library passes
  these on untouched |mdash| only the host can decide whether the relay may
  speak for another |mdash| and refuses a request that carries any of them
  more than once (``PMIX_ERR_BAD_PARAM``). They never replace the relay's own
  identity. See "Requests that pass through more than one server" in
  :doc:`/hosts/authorization`.

The library itself relays no peer's request. A request it cannot serve goes
to the host's up-call, and is refused (``PMIX_ERR_NOT_SUPPORTED``) when the
host provides none |mdash| including a tool's ``spawn`` or ``query`` sent to
a launcher that is attached to a server of its own. A launcher attached to
nothing still launches a ``spawn`` itself, for a requester of its own user.

Work the library does itself for a request |mdash| cleanup registered through
``job_control``, a spawn it launches when the host provides no ``spawn``
entry, forwarding of output, local monitoring |mdash| always uses the identity
established at connection, never a group named in the request. What the
library does after the host completes a request happens only when the host
reports success.

This applies to ``publish``, ``lookup``, ``unpublish``, ``spawn``,
``register_events``, ``notify_event`` for an event a client raised,
``query`` (including the queries behind
:ref:`PMIx_Resolve_peers(3) <man3-PMIx_Resolve_peers>` and
:ref:`PMIx_Resolve_nodes(3) <man3-PMIx_Resolve_nodes>`), ``log`` and
``log2``, ``allocate``, ``job_control``,
``monitor``, ``get_credential``, ``validate_credential``, ``iof_pull``,
``push_stdin``, ``session_control``, ``resource_block``, ``fabric`` and
``abort2``. The deprecated ``abort`` has no info array, so it carries no
requester identity beyond its process argument.

``tool_connected`` and ``tool_connected2`` carry only the identity the tool's
connection established: exactly one ``PMIX_USERID`` and one ``PMIX_GRPID``,
and ``PMIX_REQUESTOR_PID`` where the kernel reports a pid. Any
``PMIX_REQUESTOR_PID``, ``PMIX_REQUESTED_GRPID``, ``PMIX_ORIGIN_*`` or
``PMIX_REQUESTOR`` the tool put in its own info is removed before the
up-call. There is no process argument: a tool that asked
for an identity of its own is named by ``PMIX_NSPACE`` and ``PMIX_RANK``.

``fence_nb``, ``connect``, ``disconnect`` and ``group`` are made for all of
their participants rather than for one requester, so they carry no
requester identity: any ``PMIX_USERID`` or ``PMIX_GRPID`` a participant put in
its directives |mdash| or any ``PMIX_REQUESTOR_PID``,
``PMIX_REQUESTED_GRPID`` or ``PMIX_ORIGIN_*`` |mdash| is removed before the
up-call.

``publish`` carries ``PMIX_USERID`` and ``PMIX_GRPID`` but neither
``PMIX_REQUESTOR_PID`` nor ``PMIX_REQUESTED_GRPID``: its info array is the data
being published, and the library adds nothing to it a host might take for a
published key. A host must still skip ``PMIX_USERID``, ``PMIX_GRPID`` and any
``PMIX_ORIGIN_*`` entries when it stores the data.

``direct_modex`` is made for the target process, and may stand for several
local requesters that share one identity; its info array names that
identity with ``PMIX_REQUESTOR``, ``PMIX_USERID`` and ``PMIX_GRPID``, each
exactly once, and ``PMIX_REQUESTOR_PID`` where there is one. It carries no
requested group: the group is always the one established at connection,
never one named in the request: the server that holds the data
decides by group membership, and a group the requester merely names must
not count. The host passes these on to the holding server with
:ref:`PMIx_server_dmodex_request2(3) <man3-PMIx_server_dmodex_request2>`.
Requesters of different identities are sent up separately. See
:doc:`/hosts/access-control`.

When the host itself calls a PMIx API that is serviced through its own module,
the requester is the host: the up-call carries the server's own process
identifier and the library adds nothing to the caller's arrays.


ABI STABILITY
-------------

.. important:: A host must run against a PMIx library of the same major
   version as the PMIx headers it was compiled against. The library takes
   the host's ``pmix_server_module_t`` to be the size and layout defined by
   its own headers, and expects the host's functions to behave as that
   version specifies. It cannot check either. New members are added to the
   structure only in a new major release. A host that runs against a
   library of a different major version is not supported, and its behavior
   is undefined.

The `pmix_server_module_t` structure is consumed by host environments that may
initialize it using **positional** (non-designated) initializers. Consequently
the order of fields is a binary-compatibility contract and is **frozen**:

* Existing fields are never removed and never reordered.
* New callbacks are only ever **appended** at the end of the structure. The
  ``/* v1x interfaces */`` ... ``/* pending interfaces */`` group comments record
  the release series in which each block of members was added.
* When a callback's signature must change, the original field is **retained in
  place** (marked ``// DEPRECATED``) and a replacement is appended at the end
  with a numeric suffix on its name.

The structure currently carries four deprecation pairs. In each case the
library prefers the newer member and falls back to the deprecated one only if
the newer member is NULL:

* ``client_connected`` |rarrow| ``client_connected2``. The original
  ``pmix_server_client_connected_fn_t`` takes only ``proc``, ``server_object``,
  ``cbfunc``, and ``cbdata``. The replacement
  ``pmix_server_client_connected2_fn_t`` inserts an ``info[]``/``ninfo`` array so
  additional information can be passed about the connecting client.
* ``tool_connected`` |rarrow| ``tool_connected2``. The original
  ``pmix_server_tool_connection_fn_t`` returns ``void``; the replacement
  ``pmix_server_tool_connection2_fn_t`` has an otherwise identical signature but
  returns a ``pmix_status_t`` so the host can synchronously reject a tool
  connection request.
* ``log`` |rarrow| ``log2``. The original ``pmix_server_log_fn_t`` returns
  ``void``; the replacement ``pmix_server_log2_fn_t`` has an identical parameter
  list but returns a ``pmix_status_t`` so the host can indicate synchronously
  whether the log request was accepted.
* ``abort`` |rarrow| ``abort2``. The original ``pmix_server_abort_fn_t`` has no
  info array; the replacement ``pmix_server_abort2_fn_t`` adds an
  ``info[]``/``ninfo`` array, ahead of ``cbfunc``, which carries the
  requester's identity.


CALLBACK FUNCTIONS
------------------

The members are described below in structure order.

.. _man5-pmix_server_module_t-client_connected:

client_connected
^^^^^^^^^^^^^^^^^

``client_connected`` (``pmix_server_client_connected_fn_t``) |mdash|
**DEPRECATED**; use ``client_connected2``. Invoked when a registered client
connects to the local server. The client is held blocked until the host invokes
``cbfunc``, allowing the library to release it. This original form conveys only
the client's ``proc`` and ``server_object``.

.. _man5-pmix_server_module_t-client_finalized:

client_finalized
^^^^^^^^^^^^^^^^^

``client_finalized`` (``pmix_server_client_finalized_fn_t``) |mdash| Notifies the
host that a local client has called `PMIx_Finalize`, or a connected tool has
called `PMIx_tool_finalize`. The process is held blocked until the host invokes
``cbfunc``. The host should release any per-client state it associated with the
process via ``PMIx_server_register_client``. For a tool the host did not
register, ``server_object`` is NULL; see ``tool_connected2`` for how long that
tool's namespace lasts.

.. _man5-pmix_server_module_t-abort:

abort
^^^^^

``abort`` (``pmix_server_abort_fn_t``) |mdash| **DEPRECATED**; use ``abort2``.
Services :ref:`PMIx_Abort(3) <man3-PMIx_Abort>`. A local client has requested that a set
of processes be terminated; the ``procs`` array names the targets, or a NULL
array means all processes in the caller's namespace. The requesting client is
held blocked until the host invokes ``cbfunc``. The host performs the requested
termination and reports the result.

.. _man5-pmix_server_module_t-fence_nb:

fence_nb
^^^^^^^^

``fence_nb`` (``pmix_server_fencenb_fn_t``) |mdash| Services
:ref:`PMIx_Fence(3) <man3-PMIx_Fence>` and its non-blocking form. Called once all
local participants of a fence have contributed; the host collectively exchanges
the supplied ``data`` blob among all servers hosting participants and returns the
aggregated result through a ``pmix_modex_cbfunc_t``. A NULL ``data`` means the
local processes had nothing to contribute. Directives in the ``info`` array
steer the collective and are optional unless the mandatory flag is set.

**Ownership of the** ``data`` **blob transfers to the host on this call**, and
the host must ``free()`` it once it is done with it. This direction carries no
release function with which the library could reclaim the blob, and it cannot
free it when the call returns: the host is entitled to park the pointer and
read it later from another thread, so freeing it here would be a use-after-free.
Note that :ref:`PMIx_Data_embed(3) <man3-PMIx_Data_embed>` *copies* rather than
consumes, so a host that embeds the blob into a message of its own still owns
the original and must still free it. ``fence_nb`` is the only module function
that hands the host a bare data blob this way; the other collectives carry
their payloads in ``pmix_info_t`` arrays, which the library owns.

The ``info`` array may include ``PMIX_LOCAL_COLLECTIVE_STATUS``
(pmix_status_t), by which the PMIx server library reports to the host the status
of the local portion of the collective |mdash| for example, an error detected
among the local participants. A host that receives a failing status should
propagate it into the collective result rather than proceeding with the data
exchange, with two values to read with care:

* ``PMIX_ERR_TIMEOUT`` |mdash| the local phase ran out of time and the server
  handed the collective up with the participants it had (see below). A host
  **must not** complete such a collective as a success. One that ignores the
  local status, as a host could before v7.0 without harm, now completes a
  timed-out collective without the members that never called.
* ``PMIX_ERR_LOST_CONNECTION`` |mdash| a local participant was lost before it
  contributed, and the local phase completed on the survivors. For a group
  construct that asked for ``PMIX_GROUP_FT_COLLECTIVE`` this is a **degraded
  success**, not a failure: the group is to form on the survivors, who are told
  of the loss by a ``PMIX_GROUP_MEMBER_FAILED`` event, and a host that fails the
  construct on this status defeats the fault tolerance the caller asked for.

From v7.0 the library also supplies ``PMIX_LOCAL_PARTICIPANTS``
(pmix_data_array_t*), an array of ``pmix_proc_t`` identifying the local processes
that contributed to the collective: sorted, each process once, and a namespace
all of whose processes on this node contributed given once with rank
``PMIX_RANK_LOCAL_PEERS``. It may supply ``PMIX_SORTED_PROC_ARRAY`` (bool) to
indicate that the array of participating processes has already been sorted into
canonical order so the host need not sort it again. These attributes apply
equally to the other collective module functions below (``connect``,
``disconnect``, and ``group``). See :ref:`collective-participants-label`.

From v7.0 a collective whose local phase times out |mdash| a participant gave
``PMIX_TIMEOUT`` and not every local participant called in time |mdash| is
handed to the host as it stands, with ``PMIX_ERR_TIMEOUT`` as its
``PMIX_LOCAL_COLLECTIVE_STATUS`` and the participants it has, rather than
failed locally; the host can also hand it up early, at a deadline of its own,
with :ref:`PMIx_server_collective_expire(3) <man3-PMIx_server_collective_expire>`.
A host must therefore not complete a collective whose local status is
``PMIX_ERR_TIMEOUT`` as a success: that completes it without the members that
never called.

.. _man5-pmix_server_module_t-direct_modex:

direct_modex
^^^^^^^^^^^^

``direct_modex`` (``pmix_server_dmodex_req_fn_t``) |mdash| Requests that the host
contact the remote server hosting a given process and retrieve that process's
modex blob on demand (used when a :ref:`PMIx_Get(3) <man3-PMIx_Get>` cannot be
satisfied from local data). The blob is returned through a
``pmix_modex_cbfunc_t``. A timeout directive may be supplied to bound the wait.
The request may carry a ``PMIX_REQUIRED_KEY`` (char*) naming the specific key the
requester is waiting on, allowing the host to defer its response until that key
has been posted by the target process. Like ``spawn``, this operation's result
reaches the requestor only through the callback, so
``PMIX_OPERATION_SUCCEEDED`` is not an allowed return from it.

.. _man5-pmix_server_module_t-publish:

publish
^^^^^^^

``publish`` (``pmix_server_publish_fn_t``) |mdash| Services
:ref:`PMIx_Publish(3) <man3-PMIx_Publish>`. Stores the provided key/value data
in the host's data store subject to the requested range and persistence
directives, recording the publisher's identity for later lookup. Completion is
reported through a ``pmix_op_cbfunc_t``.

.. _man5-pmix_server_module_t-lookup:

lookup
^^^^^^

``lookup`` (``pmix_server_lookup_fn_t``) |mdash| Services
:ref:`PMIx_Lookup(3) <man3-PMIx_Lookup>`. Retrieves previously published data
for a NULL-terminated array of string keys, honoring any wait/timeout directives,
and returns the results through a ``pmix_lookup_cbfunc_t``. Like ``spawn``, the
data this operation retrieves reaches the requestor only through the callback,
so ``PMIX_OPERATION_SUCCEEDED`` is not an allowed return from it.

.. _man5-pmix_server_module_t-unpublish:

unpublish
^^^^^^^^^

``unpublish`` (``pmix_server_unpublish_fn_t``) |mdash| Services
:ref:`PMIx_Unpublish(3) <man3-PMIx_Unpublish>`. Deletes previously published data
matching a NULL-terminated array of keys (within any specified range) and reports
completion through a ``pmix_op_cbfunc_t``.

.. _man5-pmix_server_module_t-spawn:

spawn
^^^^^

``spawn`` (``pmix_server_spawn_fn_t``) |mdash| Services
:ref:`PMIx_Spawn(3) <man3-PMIx_Spawn>`. Launches the given array of applications.
A failure to start any process causes the entire request to be terminated and an
error returned. The namespace of the spawned job is returned through a
``pmix_spawn_cbfunc_t``, which is the **only** channel for it |mdash| so this
operation must not answer ``PMIX_OPERATION_SUCCEEDED``, even when the launch
completed before the callback returned. See the return-value contract above. The job-level information accompanying the request may
include ``PMIX_REQUESTOR_IS_TOOL`` or ``PMIX_REQUESTOR_IS_CLIENT`` (bool),
indicating whether the process that issued the spawn request is a tool or a
client, respectively; a host may use this to apply different policies to
tool-initiated launches. When the request named any of ``PMIX_FWD_STDOUT``,
``PMIX_FWD_STDERR`` or ``PMIX_FWD_STDDIAG``, the job-level information names
all three, each with the value the server resolved it to; the host registers
them with the spawned job (see
:ref:`PMIx_server_register_nspace(3) <man3-PMIx_server_register_nspace>`).

.. _man5-pmix_server_module_t-connect:

connect
^^^^^^^

``connect`` (``pmix_server_connect_fn_t``) |mdash| Services
:ref:`PMIx_Connect(3) <man3-PMIx_Connect>`. Records the specified processes as
"connected" so the host treats the failure of any of them as a reportable event.
This is a client-side collective, so the callback fires once all participants
have contributed; completion is reported through a ``pmix_op_cbfunc_t``. As with
``fence_nb``, the ``info`` array may carry ``PMIX_LOCAL_COLLECTIVE_STATUS``,
``PMIX_LOCAL_PARTICIPANTS``, and ``PMIX_SORTED_PROC_ARRAY`` describing the local
portion of the collective.

.. _man5-pmix_server_module_t-disconnect:

disconnect
^^^^^^^^^^

``disconnect`` (``pmix_server_disconnect_fn_t``) |mdash| Services
:ref:`PMIx_Disconnect(3) <man3-PMIx_Disconnect>`. Reverses a prior ``connect`` of
the same set of processes; an error is returned if the set was not previously
connected. Completion is reported through a ``pmix_op_cbfunc_t``. As with
``fence_nb``, the ``info`` array may carry ``PMIX_LOCAL_COLLECTIVE_STATUS``,
``PMIX_LOCAL_PARTICIPANTS``, and ``PMIX_SORTED_PROC_ARRAY`` describing the local
portion of the collective.

.. _man5-pmix_server_module_t-register_events:

register_events
^^^^^^^^^^^^^^^

``register_events`` (``pmix_server_register_events_fn_t``) |mdash| Part of the
support for :ref:`PMIx_Register_event_handler(3) <man3-PMIx_Register_event_handler>`.
Tells the host that the library wishes to receive notification of the specified
system (environmental) event codes; the host translates its internal codes to
the corresponding PMIx codes when it later notifies. Completion is reported
through a ``pmix_op_cbfunc_t``.

The library reference-counts each code across all of its local clients and its
own registrations, and calls this function only for the codes that acquire
their *first* registrant |mdash| so a code the host is already forwarding is
never requested again, and only the codes needing action appear in the array.
Non-system codes are never passed to the host: the host is required to report
events relating to a registered namespace regardless of any registration.

.. _man5-pmix_server_module_t-deregister_events:

deregister_events
^^^^^^^^^^^^^^^^^

``deregister_events`` (``pmix_server_deregister_events_fn_t``) |mdash| The
companion to ``register_events`` (see
:ref:`PMIx_Deregister_event_handler(3) <man3-PMIx_Deregister_event_handler>`).
Cancels a prior registration for the specified event codes. The host remains
obligated to report job-related events regardless.

This is called when a code loses its *last* registrant, counting the library's
local clients and its own registrations together, so the host can stop
forwarding a code nothing is listening for. A subsequent registration for that
code will call ``register_events`` again. A host that provides
``register_events`` should provide this as well; leaving it NULL simply means
the host keeps forwarding codes the library no longer wants.

.. _man5-pmix_server_module_t-listener:

listener
^^^^^^^^

``listener`` (``pmix_server_listener_fn_t``) |mdash| Optionally lets the host own
the rendezvous listening socket. The host monitors the descriptor, accepts
incoming client connections, and passes each accepted socket to the provided
``pmix_connection_cbfunc_t``. Leaving this member NULL causes the library to
spawn its own internal listener thread.

.. _man5-pmix_server_module_t-notify_event:

notify_event
^^^^^^^^^^^^

``notify_event`` (``pmix_server_notify_event_fn_t``) |mdash| Services
:ref:`PMIx_Notify_event(3) <man3-PMIx_Notify_event>` when an event generated by
the server or one of its local clients must be delivered beyond the local node.
The host propagates the event (identified by ``code``, ``source``, and ``range``)
and reports completion through a ``pmix_op_cbfunc_t``.

For an event a client raised, ``source`` is that client and the info array
carries its identity (see *The requester's identity* above). The host decides
with it whether to pass the event on. For ``PMIX_DEBUGGER_RELEASE`` the host
**must**, before relaying or delivering the event, check that this user may
access every job the release can reach |mdash| each job named in a custom
range (a wildcard namespace meaning every job), the source's own job for
``PMIX_RANGE_NAMESPACE``, and every job the host would deliver it to for any
wider range |mdash| and otherwise refuse it (``PMIX_ERR_NO_PERMISSIONS``)
through the callback. The library applies that rule itself to the jobs
registered with this server, and passes the rest up for the host to decide.

.. _man5-pmix_server_module_t-query:

query
^^^^^

``query`` (``pmix_server_query_fn_t``) |mdash| Services
:ref:`PMIx_Query_info(3) <man3-PMIx_Query_info>`. The host answers an array of
``pmix_query_t`` requests on behalf of the identified process and returns the
results through a ``pmix_info_cbfunc_t``. ``proct`` is the process that asked,
and each query's qualifiers carry its ``PMIX_USERID`` and ``PMIX_GRPID`` (see
*The requester's identity* above). A query that names no namespace is about
the requester's own job. The same entry point services the queries behind
:ref:`PMIx_Resolve_peers(3) <man3-PMIx_Resolve_peers>` and
:ref:`PMIx_Resolve_nodes(3) <man3-PMIx_Resolve_nodes>`.

.. _man5-pmix_server_module_t-tool_connected:

tool_connected
^^^^^^^^^^^^^^

``tool_connected`` (``pmix_server_tool_connection_fn_t``) |mdash| **DEPRECATED**;
use ``tool_connected2``. Invoked when a tool connects and needs a namespace/rank
assignment. The host returns the assigned identifier through a
``pmix_tool_connection_cbfunc_t``. This original form returns ``void`` and so
cannot synchronously reject the connection.

.. _man5-pmix_server_module_t-log:

log
^^^

``log`` (``pmix_server_log_fn_t``) |mdash| **DEPRECATED**; use ``log2``. Services
:ref:`PMIx_Log(3) <man3-PMIx_Log>` on behalf of a client. The host must not call
`PMIx_Log` from within this callback (doing so would create an infinite loop);
instead it forwards the data to a gateway/logging channel or returns a
not-supported error. Completion is reported through a ``pmix_op_cbfunc_t``. This
original form returns ``void``.

.. _man5-pmix_server_module_t-allocate:

allocate
^^^^^^^^

``allocate`` (``pmix_server_alloc_fn_t``) |mdash| Services
:ref:`PMIx_Allocation_request(3) <man3-PMIx_Allocation_request>`. The host acts
on the requested allocation modification (identified by a
``pmix_alloc_directive_t``) and returns any results through a
``pmix_info_cbfunc_t``. The directives may include ``PMIX_REQUESTOR``
(pmix_proc_t*) identifying the process on whose behalf the request was made when
it was relayed by another process.

.. _man5-pmix_server_module_t-job_control:

job_control
^^^^^^^^^^^

``job_control`` (``pmix_server_job_control_fn_t``) |mdash| Services
:ref:`PMIx_Job_control(3) <man3-PMIx_Job_control>`. The host executes the
requested control action (e.g., pause, resume, signal, terminate) against the
target processes and returns results through a ``pmix_info_cbfunc_t``. The
directives may include ``PMIX_REQUESTOR`` (pmix_proc_t*) identifying the process
on whose behalf the request was made when it was relayed by another process.

.. _man5-pmix_server_module_t-monitor:

monitor
^^^^^^^

``monitor`` (``pmix_server_monitor_fn_t``) |mdash| Services
:ref:`PMIx_Process_monitor(3) <man3-PMIx_Process_monitor>`. The host arranges the
requested monitoring (e.g., heartbeat or file-based) of the requestor, raising
the specified error status on failure, and returns results through a
``pmix_info_cbfunc_t``.

.. _man5-pmix_server_module_t-get_credential:

get_credential
^^^^^^^^^^^^^^

``get_credential`` (``pmix_server_get_cred_fn_t``) |mdash| Services
:ref:`PMIx_Get_credential(3) <man3-PMIx_Get_credential>`. The host requests a
security credential from the SMS on behalf of the identified process and returns
it through a ``pmix_credential_cbfunc_t``, which is the only channel for it |mdash|
so ``PMIX_OPERATION_SUCCEEDED`` is not an allowed return from it.

.. _man5-pmix_server_module_t-validate_credential:

validate_credential
^^^^^^^^^^^^^^^^^^^

``validate_credential`` (``pmix_server_validate_cred_fn_t``) |mdash| Services
:ref:`PMIx_Validate_credential(3) <man3-PMIx_Validate_credential>`. The host asks
the SMS to validate the supplied credential and returns the result through a
``pmix_validation_cbfunc_t``, which is the only channel for it |mdash| so
``PMIX_OPERATION_SUCCEEDED`` is not an allowed return from it.

.. _man5-pmix_server_module_t-iof_pull:

iof_pull
^^^^^^^^

``iof_pull`` (``pmix_server_iof_fn_t``) |mdash| Services
:ref:`PMIx_IOF_pull(3) <man3-PMIx_IOF_pull>`. Registers the local server to
receive the specified IO channels (a ``pmix_iof_channel_t`` bitmask) forwarded
from a set of source processes; the ``PMIX_IOF_STOP`` directive instead removes
the server from the distribution list. Note that ``stdin`` cannot be pulled with
this call. Completion is reported through a ``pmix_op_cbfunc_t``.

When the request comes from a client or tool, the directives carry the
requester's ``PMIX_USERID`` and ``PMIX_GRPID`` (see *The requester's identity*
above), and the host decides whether the requester may receive output from
the named sources, which may be in any namespace. The server forwards nothing
for the request until the host reports success; output that arrives before
then is held and delivered once the request is approved. If the host refuses,
the request is discarded.

.. _man5-pmix_server_module_t-push_stdin:

push_stdin
^^^^^^^^^^

``push_stdin`` (``pmix_server_stdin_fn_t``) |mdash| Services
:ref:`PMIx_IOF_push(3) <man3-PMIx_IOF_push>`. Passes a ``stdin`` byte object from
a source process to the host for delivery to the named target processes (a
WILDCARD rank targets all processes in a namespace). Completion is reported
through a ``pmix_op_cbfunc_t``.

.. _man5-pmix_server_module_t-group:

group
^^^^^

``group`` (``pmix_server_grp_fn_t``) |mdash| Services
:ref:`PMIx_Group_construct(3) <man3-PMIx_Group_construct>` and the related group
operations. The ``op`` argument (a ``pmix_group_operation_t``) selects construct,
destruct, or cancel of the named group across its member processes; directives
may request, for example, assignment of a group context ID. Results are returned
through a ``pmix_info_cbfunc_t``. As with ``fence_nb``, the ``info`` array may
carry ``PMIX_LOCAL_COLLECTIVE_STATUS``, ``PMIX_LOCAL_PARTICIPANTS``, and
``PMIX_SORTED_PROC_ARRAY`` describing the local portion of the collective. Read
the status as described for ``fence_nb``: a construct whose local status is
``PMIX_ERR_TIMEOUT`` must not be completed as a success, while
``PMIX_ERR_LOST_CONNECTION`` on a ``PMIX_GROUP_FT_COLLECTIVE`` construct means
it is to form on the survivors.

A host that knows which members of a construct never took part |mdash| for
example because it timed out, or completed a ``PMIX_GROUP_FT_COLLECTIVE``
construct without them |mdash| may name them in the ``info`` of its completion
as ``PMIX_COLLECTIVE_NONPARTICIPANTS`` (pmix_data_array_t*, of
``pmix_proc_t``); the server passes the list to each participant in the
construct's results, whatever the construct's status.

The server may make more than one ``group`` call for the same group ID and
``op``: bootstrap leaders and added members each arrive on a call of their own.
When the host completes any one of these calls, the server delivers that result
to every call with the same group ID and ``op`` that it is still waiting on,
because a host that combines them answers only once. The host must therefore
keep a group ID to one operation of each kind at a time, across all servers.
A request that names an ID already in use, from processes that are not part of
the operation under way, must be refused, not run alongside it. See
:ref:`label-host-env-operations`.

.. _man5-pmix_server_module_t-fabric:

fabric
^^^^^^

``fabric`` (``pmix_server_fabric_fn_t``) |mdash| Services
:ref:`PMIx_Fabric_register(3) <man3-PMIx_Fabric_register>` and related fabric
operations. The host retrieves fabric-related information from the server
supporting the system scheduler, per the ``pmix_fabric_operation_t`` op, and
returns it through a ``pmix_info_cbfunc_t``.

.. _man5-pmix_server_module_t-client_connected2:

client_connected2
^^^^^^^^^^^^^^^^^

``client_connected2`` (``pmix_server_client_connected2_fn_t``) |mdash| Replaces
the deprecated ``client_connected``. Invoked when a registered client connects;
the library prefers this member and only falls back to ``client_connected`` when
this is NULL. Adds an ``info[]``/``ninfo`` array carrying additional information
about the connecting client. The client is held blocked until the host invokes
``cbfunc``. The host answers exactly once: return an error, or return
``PMIX_SUCCESS`` and later call ``cbfunc`` once, from any thread, or return
``PMIX_OPERATION_SUCCEEDED`` to accept at once without calling it. Two mistakes
are survived: a callback followed by an error return refuses the client with
that error, and a second call of ``cbfunc`` made before the library has acted on
the first is ignored. Any other late call - after an error return, or once the
library has accepted or refused the connection - reaches memory already
released.

It is called only once the client's identity has been settled and every check
the server can make beforehand has passed. It is not called for a second
connection for a rank that already has one, which is refused with
``PMIX_ERR_EXISTS``. What the host itself may change
while it decides is checked again once it accepts: a client, or a job, it
deregistered meanwhile is refused then, as is one whose wire format another
client of its job has meanwhile fixed differently, and one that has gone before
it could be told.

The host is asked to answer within the time given in ``PMIX_TIMEOUT``. The
server waits a little longer than that; then it refuses the client with
``PMIX_ERR_TIMEOUT``, and keeps the client's rank held until the host does
answer. An answer of yes that comes too late, or is followed by one of the
refusals above, is followed by a ``PMIX_ERR_LOST_CONNECTION`` event naming the
client, delivered to the event handlers the host registered - as for any
process that leaves without finalizing - unless another connection has the
name by then.

The array carries:

* ``PMIX_USERID`` and ``PMIX_GRPID`` - the identity the server established when
  the client connected, which the security component checked against the
  kernel and against the client's registration.
* ``PMIX_CRED_TYPE`` - the name of the security component that established
  them (``native``, ``munge``).
* ``PMIX_PROC_PID`` - the client's pid. The client's PMIx library obtains it with
  ``getpid()`` and sends it when it connects, since the kernel does not report the
  pid at the other end of a TCP connection. A client in a container with its own
  process-ID namespace sends its pid within that namespace.
* ``PMIX_REALUID`` and ``PMIX_REALGID`` - the client's real (as opposed to
  effective) user and group IDs, as the client's PMIx library sent them.
* ``PMIX_TIMEOUT`` - the seconds the host is asked to answer within: what the
  client has left of the time it waits for this reply - the server's
  ``ptl_base_handshake_wait_time``, less what it has waited already. Absent
  when that wait is unbounded. A process with none left is refused with
  ``PMIX_ERR_TIMEOUT``, and the host is not asked.

.. _man5-pmix_server_module_t-tool_connected2:

tool_connected2
^^^^^^^^^^^^^^^

``tool_connected2`` (``pmix_server_tool_connection2_fn_t``) |mdash| Replaces the
deprecated ``tool_connected``. Invoked when a tool connects and needs a
namespace/rank assignment; the library prefers this member over
``tool_connected``. Unlike the original, it returns a ``pmix_status_t`` so the
host can synchronously reject the connection. The assigned identifier is returned
through a ``pmix_tool_connection_cbfunc_t``, so the host accepts only by returning
``PMIX_SUCCESS`` and calling it, once; ``PMIX_OPERATION_SUCCEEDED`` gives the tool
no identity, and the tool is refused. The rules for answering are otherwise those
of ``client_connected2``.

The info array is what the tool sent, with ``PMIX_USERID``, ``PMIX_GRPID``,
``PMIX_CRED_TYPE`` and - where the kernel reports one - ``PMIX_PROC_PID``
replaced by the values the server established for the connection, as for
``client_connected2``, and ``PMIX_TIMEOUT`` set as for ``client_connected2``.
``PMIX_REQUESTOR_PID`` is the same kernel pid, present only where the kernel
reports one; any ``PMIX_REQUESTOR_PID``, ``PMIX_REQUESTED_GRPID``,
``PMIX_ORIGIN_*`` or ``PMIX_REQUESTOR`` the tool sent is removed (see *The
requester's identity* above). The rest
is passed on as the tool sent it: ``PMIX_REALUID`` and ``PMIX_REALGID``, which the
tool's PMIx library obtains from the operating system, its command line, and the
roles it asked for.

The host is asked about one tool of a name at a time, so it never decides on two
tools for one name at once; a tool that cannot be accepted once another has
been - one of the same name - is refused without the host being asked. Tools of
other names, and tools that need an identity, are put to the host together. The
host must keep the names it gives unique; a tool it gives a name that another
tool took for itself meanwhile is refused with ``PMIX_ERR_EXISTS`` when the
host accepts it. A slow answer delays the tools of that name behind it, and
their time waiting counts against their own ``PMIX_TIMEOUT``. An answer
that is too late, and an accepted tool that cannot be connected after all, are
handled as for ``client_connected2``: the tool is refused with
``PMIX_ERR_TIMEOUT``, its name held until the answer, and a yes is followed by a
``PMIX_ERR_LOST_CONNECTION`` event naming the tool.

A tool the host did not start (a self-started tool, as opposed to a tool-client
the host registered with
:ref:`PMIx_server_register_client(3) <man3-PMIx_server_register_client>`) has a
namespace of its own, which holds self-started tools only:

* It may join an existing namespace only if that is a self-started tool's of the
  same user, or one the host registers for it during this upcall.
* It lasts only while one of its processes is connected. When the last one
  disconnects, finalized or not, the library removes the namespace and
  everything held for it - including any job data the host registered for it
  during this upcall - as
  :ref:`PMIx_server_deregister_nspace(3) <man3-PMIx_server_deregister_nspace>`
  would, and reports the lost connection at once. A later deregistration by the
  host does nothing. A tool that reconnects under the same namespace is a new
  connection: this upcall is made again.
* The host registering the same name in full with
  :ref:`PMIx_server_register_nspace(3) <man3-PMIx_server_register_nspace>`
  takes the name back: the tool's namespace is removed, any tool still connected
  in it is closed, and the host's job is built fresh.

A library that does this defines ``PMIX_CAP_TOOL_NSPACE_RELEASE``.

.. _man5-pmix_server_module_t-log2:

log2
^^^^

``log2`` (``pmix_server_log2_fn_t``) |mdash| Replaces the deprecated ``log`` and
services :ref:`PMIx_Log(3) <man3-PMIx_Log>`; the library prefers this member and
falls back to ``log`` only when it is NULL. Identical in parameters to ``log``
but returns a ``pmix_status_t``. As with ``log``, the host must not call
`PMIx_Log` from within this callback.

.. _man5-pmix_server_module_t-session_control:

session_control
^^^^^^^^^^^^^^^

``session_control`` (``pmix_server_session_control_fn_t``) |mdash| Services
:ref:`PMIx_Session_control(3) <man3-PMIx_Session_control>`. The host executes the
requested control operation against the identified session (``sessionID``) and
returns results through a ``pmix_info_cbfunc_t``. The directives may include
``PMIX_REQUESTOR`` (pmix_proc_t*) identifying the process on whose behalf the
request was made when it was relayed by another process.

.. _man5-pmix_server_module_t-resource_block:

resource_block
^^^^^^^^^^^^^^

``resource_block`` (``pmix_server_resource_block_fn_t``) |mdash| Services
:ref:`PMIx_Resource_block(3) <man3-PMIx_Resource_block>`. The host defines,
deletes, extends, or removes resources from a named resource block per the
``pmix_resource_block_directive_t`` and the supplied array of
``pmix_resource_unit_t`` units. Completion is reported through a
``pmix_op_cbfunc_t``.

.. _man5-pmix_server_module_t-abort2:

abort2
^^^^^^

``abort2`` (``pmix_server_abort2_fn_t``) |mdash| Replaces the deprecated
``abort`` and services :ref:`PMIx_Abort(3) <man3-PMIx_Abort>`; the library
prefers this member and falls back to ``abort`` only when it is NULL. The
parameters are those of ``abort`` with an ``info``/``ninfo`` array added
ahead of ``cbfunc``. For a request from a client or tool, the array carries
the requester's ``PMIX_USERID`` and ``PMIX_GRPID``, as established when it
connected (see the requester's identity above). When the host itself calls
`PMIx_Abort`, the array is empty.


.. seealso::
   :ref:`PMIx_server_init(3) <man3-PMIx_server_init>`,
   :ref:`PMIx_server_register_nspace(3) <man3-PMIx_server_register_nspace>`,
   :ref:`PMIx_Abort(3) <man3-PMIx_Abort>`,
   :ref:`PMIx_Fence(3) <man3-PMIx_Fence>`,
   :ref:`PMIx_Publish(3) <man3-PMIx_Publish>`,
   :ref:`PMIx_Lookup(3) <man3-PMIx_Lookup>`,
   :ref:`PMIx_Unpublish(3) <man3-PMIx_Unpublish>`,
   :ref:`PMIx_Spawn(3) <man3-PMIx_Spawn>`,
   :ref:`PMIx_Connect(3) <man3-PMIx_Connect>`,
   :ref:`PMIx_Disconnect(3) <man3-PMIx_Disconnect>`,
   :ref:`PMIx_Register_event_handler(3) <man3-PMIx_Register_event_handler>`,
   :ref:`PMIx_Deregister_event_handler(3) <man3-PMIx_Deregister_event_handler>`,
   :ref:`PMIx_Notify_event(3) <man3-PMIx_Notify_event>`,
   :ref:`PMIx_Query_info(3) <man3-PMIx_Query_info>`,
   :ref:`PMIx_Log(3) <man3-PMIx_Log>`,
   :ref:`PMIx_Allocation_request(3) <man3-PMIx_Allocation_request>`,
   :ref:`PMIx_Job_control(3) <man3-PMIx_Job_control>`,
   :ref:`PMIx_Process_monitor(3) <man3-PMIx_Process_monitor>`,
   :ref:`PMIx_Get_credential(3) <man3-PMIx_Get_credential>`,
   :ref:`PMIx_Validate_credential(3) <man3-PMIx_Validate_credential>`,
   :ref:`PMIx_IOF_pull(3) <man3-PMIx_IOF_pull>`,
   :ref:`PMIx_IOF_push(3) <man3-PMIx_IOF_push>`,
   :ref:`PMIx_Group_construct(3) <man3-PMIx_Group_construct>`,
   :ref:`PMIx_Fabric_register(3) <man3-PMIx_Fabric_register>`,
   :ref:`PMIx_Session_control(3) <man3-PMIx_Session_control>`,
   :ref:`PMIx_Resource_block(3) <man3-PMIx_Resource_block>`,
   :ref:`pmix_info_t(5) <man5-pmix_info_t>`,
   :ref:`pmix_proc_t(5) <man5-pmix_proc_t>`
