25.2.248. PMIx_server_init

PMIx_server_init — Initialize the PMIx server support library.

25.2.248.1. SYNOPSIS

#include <pmix_server.h>

pmix_status_t PMIx_server_init(pmix_server_module_t *module,
                               pmix_info_t info[], size_t ninfo);

25.2.248.1.1. Python Syntax

from pmix import *

foo = PMIxServer()
# the directives is a list of Python ``pmix_info_t`` dictionaries
pydirs = [{'key': PMIX_SERVER_NSPACE,
           'value': "SERVER", 'val_type': PMIX_STRING}]
# map is a dictionary of server-module-function keys to Python
# callback implementations (e.g. {'clientconnected': myconnfn, ...})
rc = foo.init(pydirs, map)

25.2.248.2. INPUT PARAMETERS

  • module: Pointer to a pmix_server_module_t(5) structure containing the host environment’s callback functions — the “function-shipped” server module through which client requests are relayed to the resource manager. Any function the host does not support is indicated by leaving its function pointer NULL; client calls to an unsupported operation return a “not supported” error. Passing NULL for module (or an empty structure) is permitted and indicates that the host will not support multi-node operations such as PMIx_Fence(3), but still intends to give local clients access to job information.

  • info: Pointer to an array of pmix_info_t(5) structures conveying directives that qualify server initialization (see DIRECTIVES). A NULL value is supported when no directives are desired.

  • ninfo: Number of elements in the info array.

25.2.248.3. DESCRIPTION

Initialize the PMIx server support library and register the host environment’s callback module. PMIx_server_init is called by the process that hosts the PMIx server — typically a resource manager daemon, launcher, or tool that will accept client and/or tool connections.

The PMIx server provides a function-shipping approach to the server side of the protocol: each client-visible PMIx operation is mirrored by an entry in the pmix_server_module_t structure. When a client request arrives, the PMIx server library invokes the corresponding host function so that resource managers can implement server behavior without being burdened with PMIx internal details. For performance and scalability, host module functions are required to return quickly and to execute their work asynchronously, completing each operation by invoking the callback function supplied to them.

Important

A host must run against a PMIx library of the same major version as the PMIx headers it was compiled against. The library takes the host’s pmix_server_module_t to be the size and layout defined by its own headers, and expects the host’s functions to behave as that version specifies. It cannot check either. New members are added to the structure only in a new major release. A host that runs against a library of a different major version is not supported, and its behavior is undefined.

Ownership rules for the module functions are important: all data passed to a host server function is owned by the PMIx server library and must not be freed by the host, while data returned by the host via a module callback is owned by the host and may be released once the callback returns.

The info array supplies additional information the server may need when initializing — for example, the namespace and rank to assign to the server itself, the temporary directory in which to place the rendezvous socket, or the PMIX_SERVER_TOOL_SUPPORT directive announcing that the daemon is willing to accept connection requests from tools (see DIRECTIVES).

The PMIx server library is reference counted. Repeated calls to PMIx_server_init are permitted and simply increment the reference count; if a module is provided on a later call and none had been set previously, the library adopts it. Each successful call must be balanced by a matching call to PMIx_server_finalize(3).

PMIx_server_init is a blocking call that completes synchronously; it does not take a callback.

Warning

Use PMIx with caution in a setuid or setgid program. PMIx takes its configuration from the environment and from locations the environment names: the PMIX_* variables (including those that name the server to connect to and the directories components are loaded from), a per-user parameter file under $HOME, and the session directory under $TMPDIR. In a setuid or setgid program these are set by the user who ran it, not by the program’s owner. PMIx cannot tell the difference, and it cannot ignore them, because the environment is how a process finds its server. A setuid or setgid program may need to set or clear these variables itself before calling this function.

25.2.248.4. DIRECTIVES

The following attributes are relevant to this operation. Support for any given attribute is optional and depends on how the PMIx implementation was built and on the capabilities of the host environment. All are passed in the info array.

Several behaviors need more than one attribute, or a build option or site setting as well: serving tools, being found as a scheduler, taking clients of earlier releases. Settings and What They Enable sets out each combination and what it does, and Starting the Server what to pass for each goal.

25.2.248.4.1. Server identity and directories

  • PMIX_SERVER_NSPACE (char*) — namespace to assign to this PMIx server. If not given, the PMIX_SERVER_NSPACE environment variable is used - a launcher that starts the server itself sets it - and otherwise the namespace is pmix-server.

  • PMIX_SERVER_RANK (pmix_rank_t) — rank to assign to this PMIx server. If not given, the PMIX_SERVER_RANK environment variable is used, and otherwise the server’s pid.

  • PMIX_SERVER_TMPDIR (char*) — temporary directory in which the server is to place its session-level files. The rendezvous files tools use to find it go under PMIX_SYSTEM_TMPDIR instead (see Rendezvous files). If not given, the PMIX_SERVER_TMPDIR environment variable is used, then TMPDIR, TEMP and TMP, and finally /tmp. The server’s local socket is made here too, and its path must fit a socket address (see The local socket). The directory must already exist. Its path is checked at init, with the system tmpdir’s: see The tmpdirs.

  • PMIX_SYSTEM_TMPDIR (char*) — temporary directory beneath which the server places its rendezvous files: those tools find it by, in pmix-user.<uid>, and its system-level file (system server, scheduler, system controller), which a root server given no system tmpdir puts in /run/pmix instead. If not given, the PMIX_SYSTEM_TMPDIR environment variable is used, then the same fallbacks as for PMIX_SERVER_TMPDIR.

25.2.248.4.2. Role and support attributes

  • PMIX_SERVER_TOOL_SUPPORT (bool) — the host is willing to accept connection requests from tools, and rendezvous files naming the server by PID and by namespace are written (see Rendezvous files). Without it, every tool is refused with PMIX_ERR_NOT_SUPPORTED - including one a client or job script starts, which has the server’s address in its environment. The files are readable only by the host’s user ID, and tools running under other user IDs are refused, unless PMIX_SERVER_ALLOW_FOREIGN_TOOLS is also given; such tools also need the server’s local socket widened to them (PMIX_SOCKET_MODE, PMIX_USERID, PMIX_GRPID), which a server does not do by itself. Tools of root, and of the user named by PMIX_USERID, are not refused this way.

  • PMIX_SERVER_SYSTEM_SUPPORT (bool) — the server is the node’s system server: tools asking for the system server find it through a system-level rendezvous file. Takes effect only together with PMIX_SERVER_TOOL_SUPPORT. As for that attribute, the file is readable only by the host’s user ID unless PMIX_SERVER_ALLOW_FOREIGN_TOOLS is given.

  • PMIX_SERVER_SESSION_SUPPORT (bool) — the server is the session server of its session, found by tools through a rendezvous file named by PMIX_SESSION_RNDZ_ID, else by PMIX_SESSION_ID: without either PMIx_server_init fails. Servers of different sessions share a node; a second server of one session on a node fails PMIx_server_init. Takes effect only together with PMIX_SERVER_TOOL_SUPPORT.

  • PMIX_SESSION_RNDZ_ID (uint32_t) — the ID naming the session server’s rendezvous file, unique among the session servers on a node; see PMIX_SERVER_SESSION_SUPPORT. The host chooses it, so that it need not change PMIX_SESSION_ID, the session’s ID as the scheduler assigned it, to name the file - which is named by PMIX_SESSION_ID when this is not given.

  • PMIX_SERVER_GATEWAY (bool) — the server is acting as a gateway that can relay operations (e.g., logging) requiring routing to other nodes.

  • PMIX_SERVER_SCHEDULER (bool) — the server is hosting the system scheduler (workload manager) and expects to receive scheduler-related requests. The runtimes that find it connect as tools, so it implies PMIX_SERVER_TOOL_SUPPORT; passing that attribute false as well fails the init.

  • PMIX_SERVER_SYS_CONTROLLER (bool) — the server is hosting the system controller. As for PMIX_SERVER_SCHEDULER, the processes that find it are tools: it implies PMIX_SERVER_TOOL_SUPPORT, and passing that attribute false as well fails the init.

  • PMIX_SERVER_LEGACY_PEERS (bool) — take (or refuse) connections from clients and tools of PMIx releases before v7.0, which connect over TCP. Overrides the ptl_base_legacy_clients MCA parameter (and ptl_base_legacy_peers, which sets it in a build with legacy clients), which defaults to false. With it, the server listens on a loopback interface for them and gives its clients the TCP address in the variables those releases read; without it the server has no TCP listener, and a client or tool of an earlier release cannot connect. Only a PMIx configured --enable-legacy-clients can take them (the default is not to): in one without it, asking fails the init - by this attribute, or by ptl_base_legacy_clients; ptl_base_legacy_peers asks such a build for nothing. A host that does not pass it leaves the decision to the site’s parameter.

  • PMIX_SERVER_REMOTE_CONNECTIONS (bool) — deprecated. A server takes connections from its own host only: true fails the init, and false changes nothing. A client or tool does not dial a server at an address that is not its own host’s, and the server refuses a TCP peer at such an address.

  • PMIX_SERVER_ALLOW_FOREIGN_TOOLS (bool) — mark the rendezvous files as readable by all users and allow tools running under user IDs other than that of the server to connect. Defaults to false: without it, a connection request from a tool under another user ID is refused. The host retains ultimate authority over such connections and may restrict what foreign tools are permitted to do (for example, limiting them to queries) as dictated by host policy. The permissions of an existing directory the files are placed in are never changed, so a directory other users cannot search still keeps their tools out (see Rendezvous files).

25.2.248.4.3. Connection attributes

  • PMIX_SOCKET_MODE (uint32_t) — the mode (nine permission bits) of the server’s local socket. Default 0600: only the server’s user can connect. See The local socket.

  • PMIX_USERID (uint32_t, or a user name as char*) — the owner of the server’s local socket. Default: the server’s user. Its directory stays the server’s. Tools of this user are taken without PMIX_SERVER_ALLOW_FOREIGN_TOOLS, as the server’s own user’s are.

  • PMIX_GRPID (uint32_t, or a group name as char*) — the group of the server’s local socket and of its directory. Default: the server’s effective group.

  • PMIX_LAUNCHER_RENDEZVOUS_FILE (char*) — full pathname of an additional rendezvous file in which the server is to write its contact information — for example, so that a debugger can find a launcher. Missing directories on the path are created. The file is written whether or not PMIX_SERVER_TOOL_SUPPORT is given; a tool that reads it is accepted only with that attribute.

  • PMIX_CONNECT_TO_SYSTEM, PMIX_CONNECT_SYSTEM_FIRST, PMIX_CONNECT_TO_SCHEDULER, PMIX_CONNECT_TO_SYS_CONTROLLER (bool), PMIX_CONNECTION_ORDER (char*), PMIX_SERVER_PIDINFO (pid_t), PMIX_TOOL_ATTACHMENT_FILE (char*) — connect the server to the server these name, as for PMIx_tool_init(3), and make it this server’s own server — the one, for example, an allocation request is sent to. A malformed value fails the init. With PMIX_TOOL_CONNECT_OPTIONAL a server that cannot be reached is not an error.

  • PMIX_TCP_REPORT_URI (char*) — report the server’s URI: - writes the local socket’s URI to stdout and + to stderr; tcp- and tcp+ write the TCP URI instead, for a reader of a release before v7.0, and fail the init when the server takes no legacy clients and so has no TCP listener. An integer writes to that open file descriptor, which is then closed: the TCP URI’s string and then the local socket’s, each ending in a NUL, when the server takes legacy clients, else the local socket’s alone. Anything else is taken as the name of a file to write it to.

  • PMIX_TCP_IF_INCLUDE (char*) — comma-delimited list of devices and/or CIDR notation to listen on. Cannot be combined with PMIX_TCP_IF_EXCLUDE.

  • PMIX_TCP_IF_EXCLUDE (char*) — comma-delimited list of devices and/or CIDR notation not to listen on.

  • PMIX_TCP_IPV4_PORT (int or char*) — IPv4 port to listen on, or (as a string) a comma-delimited list of ports and ranges to try in order. 0 or -1 lets the kernel choose.

  • PMIX_TCP_IPV6_PORT (int or char*) — as PMIX_TCP_IPV4_PORT, for IPv6.

  • PMIX_TCP_DISABLE_IPV4 (bool) — do not listen on IPv4 addresses.

  • PMIX_TCP_DISABLE_IPV6 (bool) — do not listen on IPv6 addresses.

25.2.248.4.4. Topology and behavior attributes

  • PMIX_TOPOLOGY2 (pmix_topology_t) — pointer to an implementation-specific description of the local node topology.

  • PMIX_SERVER_SHARE_TOPOLOGY (bool) — the server is to scalably expose the node topology to its local clients (for example, via shared-memory backing stores), cleaning up any artifacts at finalize.

  • PMIX_HOMOGENEOUS_SYSTEM (bool) — the nodes in the system are topologically identical, so the server need not compute or exchange per-node topology descriptions.

  • PMIX_SERVER_ENABLE_MONITORING (bool) — enable the internal monitoring capabilities of the PMIx server.

  • PMIX_EXTERNAL_PROGRESS (bool) — the host will progress the PMIx library as needed via calls to PMIx_Progress rather than PMIx spawning its own internal progress thread.

  • PMIX_SINGLETON (char*) — the server is operating in support of a singleton process; the value is the nspace.rank string of that singleton.

  • PMIX_IOF_LOCAL_OUTPUT (bool) — write output streams to the server’s own stdout/stderr.

  • PMIX_BIND_PROGRESS_THREAD (char*) — comma-delimited ranges of CPUs to which the internal PMIx progress thread is to be bound.

  • PMIX_BIND_REQUIRED (bool) — return an error if the internal PMIx progress thread cannot be bound as requested.

  • PMIX_EXTERNAL_AUX_EVENT_BASE (void*) — pointer to an event_base the library is to use for auxiliary functions (e.g., capturing signals) that would otherwise interfere with the host.

25.2.248.5. RETURN VALUE

Returns PMIX_SUCCESS on success. On error, a negative value corresponding to a PMIx error constant is returned, including:

  • PMIX_ERR_INIT — the PMIx server library could not be initialized (for example, a required transport was explicitly disabled, or a tight race between concurrent PMIx_server_init calls left the library in an unusable state).

  • PMIX_ERR_NOT_SUPPORTED — a provided directive requested behavior the implementation does not support.

  • PMIX_ERR_BAD_PARAM — a directive or an environmental value carried something the implementation cannot use — for example a PMIX_SINGLETON that is not of the form "nspace.rank", or a PMIX_SERVER_RANK environment variable that is not a simple non-negative number.

Any other negative value indicates an appropriate error condition. PMIx error constants are defined in pmix_common.h.

A failed PMIx_server_init leaves nothing behind. Anything the call had already brought up is torn down before the error is returned, so the caller may correct whatever was rejected and call PMIx_server_init again. It must not call PMIx_server_finalize(3), which answers PMIX_ERR_INIT because no initialization completed. The one exception is a failure of the underlying runtime itself, which cannot be unwound; there the library reports the error and is unusable for the life of the process.

25.2.248.6. NOTES

PMIx_server_init is intended for use only by processes acting as a PMIx server — processes that host the server support library on behalf of a resource manager or launcher. Client processes call PMIx_Init(3) and tools call PMIx_tool_init(3) instead.

After a successful PMIx_server_init, the host is expected to register each participating namespace with PMIx_server_register_nspace(3) and each local client with PMIx_server_register_client(3) before launching that client, then set up the client’s environment with PMIx_server_setup_fork.

25.2.248.6.1. The local socket

Every server listens on a local (AF_UNIX) socket, where the clients and tools of this release on its host connect. It listens on TCP, on a loopback interface, only for peers of earlier releases (PMIX_SERVER_LEGACY_PEERS). The kernel identifies the process at each end of a connection to the local socket: the server knows each peer’s uid, gid and pid, and each peer can check the server’s uid and gid.

The socket is pmix-<pid>, in a directory of its own the server makes beneath PMIX_SERVER_TMPDIR: <tmpdir>/pmix-XXXXXX. By default the directory is mode 0700 and the socket 0600, so only the server’s user can connect. A host that wants other users’ processes to connect - a client it starts under another user ID, a tool it allows with PMIX_SERVER_ALLOW_FOREIGN_TOOLS - must widen it: PMIX_SOCKET_MODE sets the socket’s mode, and PMIX_USERID and PMIX_GRPID its owner and group. The directory stays the server’s: it is searchable by whom the mode lets connect - and by everyone, when the socket is given to another user - and readable by no one but the server’s user. The server never widens it on its own.

The socket’s path must fit a socket address - about 100 characters. A temporary directory too long for it makes PMIx_server_init fail; there is no fallback. The default $TMPDIR on macOS is too long: pass a shorter PMIX_SERVER_TMPDIR.

The server removes the socket and its directory at finalize. A server that is killed leaves them behind, as pmix-XXXXXX directories in its temporary directory.

PMIx_server_setup_fork gives a child the socket’s address and the server’s uid and gid in PMIX_SERVER_LOCAL_URI; a child of this release connects there and nowhere else, and only to a server with that uid and gid.

25.2.248.6.2. Rendezvous files

A tool finds a server through a rendezvous file: a small file holding the server’s URI, version and PID, and the address of its local socket. Which ones are written depends on the directives:

Directive

File

Directory

PMIX_LAUNCHER_RENDEZVOUS_FILE

the name given

as given

PMIX_SERVER_SCHEDULER

pmix.sched.<hostname>

the system directory (below)

PMIX_SERVER_SYS_CONTROLLER

pmix.sysctrlr.<hostname>

the system directory

PMIX_SERVER_TOOL_SUPPORT with PMIX_SERVER_SYSTEM_SUPPORT

pmix.sys.<hostname>

the system directory

PMIX_SERVER_TOOL_SUPPORT with PMIX_SERVER_SESSION_SUPPORT

pmix.<hostname>.session.<rendezvous ID>

<PMIX_SYSTEM_TMPDIR>/pmix-user.<uid>

PMIX_SERVER_TOOL_SUPPORT

pmix.<hostname>.tool.<pid> and pmix.<hostname>.tool.<nspace>

<PMIX_SYSTEM_TMPDIR>/pmix-user.<uid>

A scheduler or system controller writes only its own file; the files under PMIX_SERVER_TOOL_SUPPORT are written by any other server that gives it.

The tool files go in a directory of the server’s user’s own, whatever the server’s session tmpdir: pmix-user.<uid> beneath the system tmpdir. The server makes it mode 0700 - 0711 with PMIX_SERVER_ALLOW_FOREIGN_TOOLS, so other users’ tools can open the files at their names but not list them. One already there is used only if it is a directory of the server’s user, not a link, that no other user can write; anything else is reported, and the server runs on without writing its tool files. The system directory is PMIX_SYSTEM_TMPDIR when the host names one; otherwise /run/pmix for a root server (/var/run/pmix where there is no /run), and the server’s pmix-user directory for any other.

By default each file is readable only by the server’s user ID. With PMIX_SERVER_ALLOW_FOREIGN_TOOLS it is also readable by all users.

A file at the same name that no running server holds is removed and replaced. One a running server holds belongs to that server, and PMIx_server_init fails rather than take it over: one system server, scheduler or system controller per node, one session server per session on a node, and one server per namespace. A server whose host names no namespace takes pmix-server-<hostname>-<pid>.

A file named by PMIX_TCP_REPORT_URI is not a rendezvous file and none of the above applies to it. A file already at that name is reused only if the server’s user owns it; one of any other user’s - root’s too, unless the server is root - fails PMIx_server_init. The file, reused or new, is given mode 0644 - readable by all, writable only by the server’s user - and is removed at finalize.

25.2.248.6.3. The tmpdirs

Before it makes anything in them, the server checks the path of its session and system tmpdirs, links resolved, from /. Every directory above a tmpdir must be owned by root, the server’s user, or a user from outside the server’s user namespace, and must not be writable by anyone else unless it has the sticky bit, or its group’s only member is its owner. The tmpdir itself may be any user’s - a root server is often given a directory it made for the job’s user - provided only its owner can write it; or else it must pass the rule for the directories above, as /tmp does. Otherwise another user could move the server’s directories aside, and its clients would fail to connect.

A path that fails makes PMIx_server_init fail, with a message naming the directory. The ptl_base_tmpdir_check MCA parameter changes that: refuse (the default), warn - report it and continue - or silent, not to check. The last two are for a directory no other user shares that the rule misjudges, such as a Kubernetes emptyDir mounted at /tmp.