.. _man3-PMIx_server_register_nspace:

PMIx_server_register_nspace
===========================

.. include_body

``PMIx_server_register_nspace`` |mdash| Register a namespace and its job-level
information with the PMIx server library.


SYNOPSIS
--------

.. code-block:: c

   #include <pmix_server.h>

   pmix_status_t PMIx_server_register_nspace(const pmix_nspace_t nspace, int nlocalprocs,
                                             pmix_info_t info[], size_t ninfo,
                                             pmix_op_cbfunc_t cbfunc, void *cbdata);


Python Syntax
^^^^^^^^^^^^^

.. code-block:: python3

  from pmix import *

  foo = PMIxServer()
  # ... after a successful foo.init(pydirs, map) ...
  # the directives is a list of Python ``pmix_info_t`` dictionaries
  pydirs = [{'key': PMIX_UNIV_SIZE,
             'value': 4, 'val_type': PMIX_UINT32},
            {'key': PMIX_JOB_SIZE,
             'value': 4, 'val_type': PMIX_UINT32}]
  rc = foo.register_nspace("myjob", 4, pydirs)


INPUT PARAMETERS
----------------

* ``nspace``: The namespace (a character array of maximum length
  ``PMIX_MAX_NSLEN``) of the job being registered.
* ``nlocalprocs``: The number of processes from this namespace that will be
  launched locally |mdash| i.e., that will connect to this PMIx server. This
  count is required so that the server library can correctly determine when a
  collective operation is locally complete, even if the collective is called
  before all local processes have started.

  A **negative** value has a distinct meaning: it declares the call to be an
  *update* to a namespace the library already holds, rather than the
  registration of a new one. See `Updating a registered namespace`_.
* ``info``: Pointer to an array of :ref:`pmix_info_t(5) <man5-pmix_info_t>`
  structures conveying the session-, job-, application-, node-, and
  process-realm information for the namespace (see `DIRECTIVES`_). A ``NULL``
  value is supported when no data is to be registered (for example, when using
  the ``PMIX_REGISTER_NODATA`` directive).
* ``ninfo``: Number of elements in the ``info`` array.
* ``cbfunc``: Callback function of type :ref:`pmix_op_cbfunc_t <man5-pmix_op_cbfunc_t>` invoked when the
  registration completes. A ``NULL`` value makes the call *blocking* (see
  `DESCRIPTION`_).
* ``cbdata``: Opaque pointer that is passed, unmodified, to ``cbfunc``.


DESCRIPTION
-----------

Register a namespace (job) with the PMIx server library so that its job-level
information can be provided to the processes in that job as they connect. The
PMIx connection procedure gives the host PMIx server an opportunity to pass
job-related information down to each child process |mdash| for example, the
number of processes in the job, the relative local ranks of the processes, and
the node and process maps. The host is free to determine which of the supported
elements it provides; the defined values are described in `DIRECTIVES`_.

The host **must** register *every* namespace that will participate in
collective operations involving local processes |mdash| even a namespace from
which this server hosts no local processes |mdash| if any local process might
at some point perform a collective operation involving one or more processes
from that namespace. This is required so the collective can determine when it
is locally complete.

The host **must** register a namespace before it starts any of that
namespace's local processes. A client that connects to this server before its
namespace is registered is refused (``PMIX_ERR_INVALID_CRED``), and the host
is not asked about it |mdash| even when its rank was registered with
:ref:`PMIx_server_register_client(3) <man3-PMIx_server_register_client>` and
prepared with ``PMIx_server_setup_fork``.

The host **must** include ``PMIX_JOB_SIZE`` whenever it registers a
namespace's job data, either at the top level of ``info`` or inside its
``PMIX_JOB_INFO_ARRAY``. A registration (``nlocalprocs`` zero or more) without
it is refused with ``PMIX_ERR_BAD_PARAM`` and changes nothing the library
holds for ``nspace``. A registration carrying ``PMIX_REGISTER_NODATA``
registers no job data and need not include it; the update (negative
``nlocalprocs``) that later delivers that namespace's
``PMIX_JOB_INFO_ARRAY`` must. An update of a namespace whose job data is
already registered need not repeat the size.

Blocking and non-blocking forms
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``PMIx_server_register_nspace`` supports both a non-blocking and a blocking
mode of operation, selected by the ``cbfunc`` argument. This function is the
canonical example of the PMIx thread-shifting pattern: in both modes the
request parameters are packaged and posted to the library's internal progress
thread, where the actual registration work is performed.

When ``cbfunc`` is **non-**\ ``NULL``, the call is *non-blocking*: the function
posts the request to the progress thread and returns ``PMIX_SUCCESS``
immediately, and the provided ``cbfunc`` is invoked with the final status once
registration completes. As with all non-blocking PMIx APIs, the caller **must**
keep the ``info`` array valid until ``cbfunc`` has been invoked.

When ``cbfunc`` is ``NULL``, the call is *blocking*: the library substitutes an
internal callback, posts the request, and waits for the progress thread to
finish before returning. In this case the result is carried by the return value
itself, and on success the function returns ``PMIX_OPERATION_SUCCEEDED`` to
indicate that the operation completed inline and no callback will fire.


Updating a registered namespace
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

A job's description is not always fixed for the life of the job. Resources may
be added to it, a value the host computed early may be refined, a node may be
lost. **This API is the supported way to revise the job-level data of a
namespace that is already registered**, and it is the method host environments
should use.

Pass a **negative** ``nlocalprocs``. The ``info`` array is then read as a
revision of what the library holds for ``nspace`` rather than as a fresh
registration, and no local process count is implied or changed.

The array may take either of two shapes, and they are treated identically:

* **only the values that changed**, or
* **the whole description**, with some values in it different from before.

Restating a value that has not changed is explicitly permitted and costs
nothing: each datastore compares every entry against what it already answers
for that key and discards the ones that match, so a host that periodically
restates its full job description does not accumulate anything. Only entries
that are new, or whose value differs, are stored and pushed to clients. This is
not merely an optimization |mdash| the shared-memory datastore cannot reclaim
what it has published, so a restatement that was treated as a change would grow
a job's footprint for the life of the job.

Values are applied at the **job level** (they are stored against
``PMIX_RANK_WILDCARD``) and are pushed to every local client of that namespace
that is already running, so a process that called
:ref:`PMIx_Init(3) <man3-PMIx_Init>` before the update will see the revised
value from its own datastore without having to ask the server for it.

A ``PMIX_PROC_INFO_ARRAY`` in an update revises the data of the single process
that array describes, and a ``PMIX_JOB_INFO_ARRAY`` is unwrapped and its
contents applied as job-level values.

.. note::

   There is currently **no way to remove a key** through an update. An update
   may add a key or change the value of an existing one; a key once registered
   remains registered for the life of the namespace. Use
   :ref:`PMIx_server_deregister_resources(3) <man3-PMIx_server_deregister_resources>`
   for values that were registered as non-namespace resources.


DIRECTIVES
----------

Information is passed through the ``info`` array, organized by *data realm*.
Realm information may be passed either as individual
:ref:`pmix_info_t(5) <man5-pmix_info_t>` entries or grouped inside a
``pmix_data_array_t`` labeled with the corresponding ``*_INFO_ARRAY``
attribute. The following attributes are required to be supported by all PMIx
libraries for use with this API:

* ``PMIX_REGISTER_NODATA`` (bool) |mdash| the registration is for the namespace
  only; do not copy any job data. Used to pre-register a namespace whose data
  will follow later.
* ``PMIX_SESSION_INFO_ARRAY`` (pmix_data_array_t\*) |mdash| an array of
  session-realm information for the session containing this job. A
  session can also be registered on its own, before its first job and
  beyond its last: see
  :ref:`PMIx_server_register_session(3) <man3-PMIx_server_register_session>`.
* ``PMIX_JOB_INFO_ARRAY`` (pmix_data_array_t\*) |mdash| an array of job-realm
  information for this namespace.
* ``PMIX_APP_INFO_ARRAY`` (pmix_data_array_t\*) |mdash| an array of
  application-realm information; required (one array per application) when the
  job contains more than one application.
* ``PMIX_PROC_INFO_ARRAY`` (pmix_data_array_t\*) |mdash| an array of
  process-realm information for a process in the job.
* ``PMIX_NODE_INFO_ARRAY`` (pmix_data_array_t\*) |mdash| an array of node-realm
  information for a node participating in the job.

Session-realm information
^^^^^^^^^^^^^^^^^^^^^^^^^^

* ``PMIX_UNIV_SIZE`` (uint32_t) |mdash| number of process slots allocated to
  the session.
* ``PMIX_MAX_PROCS`` (uint32_t) |mdash| maximum number of processes; must be
  provided if ``PMIX_UNIV_SIZE`` is not given.
* ``PMIX_SESSION_ID`` (uint32_t) |mdash| session identifier; required whenever
  the PMIx server library may host multiple sessions.

Job-realm information
^^^^^^^^^^^^^^^^^^^^^

* ``PMIX_NSPACE`` (char\*) |mdash| namespace of the job being registered.
* ``PMIX_JOBID`` (char\*) |mdash| job identifier assigned by the resource
  manager.
* ``PMIX_JOB_SIZE`` (uint32_t) |mdash| total number of processes in the job.
  **Required** whenever the job's data is registered; see `DESCRIPTION`_.
* ``PMIX_NODE_MAP`` (char\*) |mdash| regular-expression representation of the
  nodes hosting the job (see ``PMIx_generate_regex2``).
* ``PMIX_PROC_MAP`` (char\*) |mdash| regular-expression representation of the
  process-to-node mapping.
* ``PMIX_NODE_MAP_RAW`` (char\*) |mdash| comma-delimited list of the nodes
  containing processes for this job, provided as an alternative to the
  regular-expression form in ``PMIX_NODE_MAP``.
* ``PMIX_PROC_MAP_RAW`` (char\*) |mdash| semicolon-delimited list of strings,
  each a comma-delimited list of the ranks on the corresponding node, provided
  as an alternative to the regular-expression form in ``PMIX_PROC_MAP``.
* ``PMIX_ANL_MAP`` (char\*) |mdash| process mapping in ANL notation, as used by
  PMI-1 and PMI-2. A host need not supply it: when a local process asks for
  it and it was not registered, the server works it out from the job's node
  list and per-node local peers, keeps it as job-level data, and answers
  from it. No value is produced when that data does not place every rank or
  when the mapping cannot be expressed within the 1024 bytes a PMI-2 value
  may occupy.
* ``PMIX_APP_MAP_TYPE`` (char\*) |mdash| type of mapping used to lay out the
  application (e.g., ``cyclic``).
* ``PMIX_APP_MAP_REGEX`` (char\*) |mdash| regular expression describing the
  result of the mapping.
* ``PMIX_JOB_NUM_APPS`` (uint32_t) |mdash| number of applications in the job;
  required when the job contains more than one application.
* ``PMIX_SERVER_NSPACE`` (char\*) |mdash| namespace of the PMIx server itself.
* ``PMIX_SERVER_RANK`` (pmix_rank_t) |mdash| rank of the PMIx server itself.

The host environment is expected to supply a broad range of additional
session-, job-, application-, node-, and process-realm information as required
by the job. See the "Reserved Keys" chapter of the PMIx Standard for the full
list and for the rules governing how each key is retrieved.

Spawned jobs and output forwarding
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The output of a job created by
:ref:`PMIx_Spawn(3) <man3-PMIx_Spawn>` is forwarded to whoever receives its
parent's output, formatted the way that receiver formats the parent's. The
server that receives the job's output need not be the one that processed the
spawn, and only sees what the host registers with it. For a spawned job, the
host therefore registers the namespace with every server that may receive its
output for a tool, and includes:

* ``PMIX_PARENT_ID`` (pmix_proc_t\*) |mdash| the process that spawned the job,
  in the job-level or per-process data. This is how the server finds the
  subscriptions the job inherits.
* the **output formatting directives** the spawn request carried |mdash|
  ``PMIX_IOF_TAG_OUTPUT``, ``PMIX_IOF_TAG_DETAILED_OUTPUT``,
  ``PMIX_IOF_TAG_FULLNAME_OUTPUT``, ``PMIX_IOF_RANK_OUTPUT``,
  ``PMIX_IOF_TIMESTAMP_OUTPUT``, ``PMIX_IOF_XML_OUTPUT``,
  ``PMIX_IOF_MERGE_STDERR_STDOUT`` and ``PMIX_IOF_OUTPUT_RAW`` |mdash| with
  the value each was given, a ``false`` included. The server sends them to each
  receiver with the first of the job's output (``PMIX_IOF_JOB_DIRECTIVES``).
* the **output channels** the spawn request named, if it named any |mdash|
  ``PMIX_FWD_STDOUT``, ``PMIX_FWD_STDERR`` and ``PMIX_FWD_STDDIAG``, as the
  server passed them to the host's ``spawn`` function: all three, each with the
  value the server resolved it to. A job registered with them inherits its
  parent's forwarding on the channels set ``true`` and no others, and a server
  that writes the job's output to its own ``stdout`` and ``stderr`` writes only
  those channels there. The job's output files are written as always.
* optionally, ``PMIX_IOF_INHERIT`` (bool) set to ``false``, if the job is not to
  inherit its parent's forwarding at all.

The host does not need to copy the parent's formatting onto the child: PMIx
supplies that itself. A host that omits the request's own directives loses
only those, and the child is formatted exactly as its parent is.


CALLBACK FUNCTION
-----------------

When ``cbfunc`` is provided, it has the signature ``pmix_op_cbfunc_t``:

.. code-block:: c

   typedef void (*pmix_op_cbfunc_t)(pmix_status_t status, void *cbdata);

The library invokes ``cbfunc`` from its progress thread once registration
completes. ``status`` is ``PMIX_SUCCESS`` if the namespace and its data were
registered successfully, or a negative PMIx error constant otherwise.
``cbdata`` is the opaque pointer passed to ``PMIx_server_register_nspace``,
allowing the host to correlate the completion with its originating request.


RETURN VALUE
------------

For the non-blocking form (``cbfunc`` is non-``NULL``), a return of
``PMIX_SUCCESS`` indicates only that the request was accepted for processing;
the final status is delivered to ``cbfunc``.

For the blocking form (``cbfunc`` is ``NULL``), the return value carries the
result directly:

* ``PMIX_OPERATION_SUCCEEDED`` |mdash| the registration completed successfully
  inline; no callback is or will be invoked.
* ``PMIX_ERR_INIT`` |mdash| the PMIx server library has not been initialized.
* ``PMIX_ERR_NOT_AVAILABLE`` |mdash| the operation cannot be serviced because
  the library's progress engine has been stopped.
* ``PMIX_ERR_BAD_PARAM`` |mdash| ``nspace`` is ``NULL`` or empty, or the
  registration carries the job's data without ``PMIX_JOB_SIZE``.
* ``PMIX_ERR_NOT_FOUND`` |mdash| a ``PMIX_USERID`` or ``PMIX_GRPID`` was given
  as a name that does not resolve to a user or group.
* ``PMIX_ERR_WOULD_BLOCK`` |mdash| the blocking form was called from the
  library's own progress thread (for example, from inside a
  :ref:`pmix_server_module_t(5) <man5-pmix_server_module_t>` upcall), where
  waiting for the request would be waiting for the caller. Use the
  non-blocking form from that context.

Both forms return ``PMIX_ERR_INIT``, ``PMIX_ERR_NOT_AVAILABLE`` and, for a
``NULL`` or empty ``nspace``, ``PMIX_ERR_BAD_PARAM`` directly, since they are
detected before the request is accepted. A missing ``PMIX_JOB_SIZE`` is
detected once the request is processed, so the non-blocking form reports it
to ``cbfunc``.

Any other negative value indicates an appropriate error condition. PMIx error
constants are defined in ``pmix_common.h``.


NOTES
-----

There is no requirement that a namespace be registered before its clients are
registered; if
:ref:`PMIx_server_register_client(3) <man3-PMIx_server_register_client>` is
called for an as-yet-unregistered namespace, the library creates a placeholder
namespace in anticipation of the eventual ``PMIx_server_register_nspace`` call.
However, collective operations cannot complete locally until the namespace has
been registered with its ``nlocalprocs`` count, and the namespace must be
registered before any of its clients is started: a client that connects while
its namespace is only a placeholder is refused.

Large ``PMIX_NODE_MAP`` and ``PMIX_PROC_MAP`` values are commonly generated in
compressed form using ``PMIx_generate_regex2`` prior to registration.

A self-started tool - one the host did not start - may have taken a namespace
name for itself. Registering that name in full (``nlocalprocs`` zero or more)
takes it back: the tool's namespace, and everything the library held for it, is
removed, any tool process still connected in it is closed, and the host's job
is built fresh as though the name had been free. An update (negative
``nlocalprocs``) revises what the tool's namespace holds and leaves the tool
connected. A host that registers a namespace for a tool during its
``tool_connected2`` upcall gives the tool that name; see
:ref:`pmix_server_module_t(5) <man5-pmix_server_module_t>`.

The job's ``PMIX_USERID`` and ``PMIX_GRPID`` may be given either as numbers
(``uint32_t``) or as a user and group name (``char*``), at the top level of
``info`` or inside an info array such as ``PMIX_JOB_INFO_ARRAY``. The library
resolves a name when the namespace is registered, and stores and reports only
the number. The ``info`` array itself is not changed.

``PMIX_USER_GRPIDS`` (a ``pmix_data_array_t`` of group ids or names) gives the
groups the job's owner belongs to - primary and supplementary - and may also be
given inside ``PMIX_JOB_INFO_ARRAY``. **The host is responsible for providing
it**: the library never looks a user's groups up. Without it, the owner's
requests are judged by the one group each is made under, so a group in another
job's access list admits the owner only if it is exactly that group. A later
registration that gives it replaces the groups the library had for that user.

These identify the job's *owner*, and ``PMIX_ACCESS_PERMISSIONS`` (holding
``PMIX_ACCESS_USERIDS`` and/or ``PMIX_ACCESS_GRPIDS``, whose entries may also be
names) names further users and groups allowed to access the job. The library
records both when the namespace is registered; a later registration that names a
list replaces it, and one that does not keeps it. If no owner is given, the job
belongs to the user its clients are registered with. A malformed
``PMIX_ACCESS_PERMISSIONS`` fails the registration with ``PMIX_ERR_BAD_PARAM``.
See :doc:`/hosts/access-control` for how the library uses them.


.. include:: /man/no-blocking-in-progress-thread.rst


.. seealso::
   :ref:`PMIx_server_deregister_nspace(3) <man3-PMIx_server_deregister_nspace>`,
   :ref:`PMIx_server_register_client(3) <man3-PMIx_server_register_client>`,
   :ref:`PMIx_server_register_resources(3) <man3-PMIx_server_register_resources>`,
   :ref:`PMIx_server_register_session(3) <man3-PMIx_server_register_session>`,
   :ref:`PMIx_generate_regex2(3) <man3-PMIx_generate_regex2>`,
   :ref:`PMIx_server_init(3) <man3-PMIx_server_init>`,
   :ref:`pmix_info_t(5) <man5-pmix_info_t>`,
   :ref:`pmix_nspace_t(5) <man5-pmix_nspace_t>`,
   :ref:`pmix_status_t(5) <man5-pmix_status_t>`
