11.16. Connection Lifecycle: Clients and Tools

This page describes the life of a connection between a PMIx server and one of its clients or tools, from the server opening its listener to the connection ending. It covers:

  1. how a server sets up its listener;

  2. how a server tells the clients it starts where to connect;

  3. how a client or tool opens a connection;

  4. how the server accepts it: the handshake, the security checks, and the call to the host;

  5. how a connection ends: by finalize, by disconnect, by a process dying, and by the host.

Two related subjects are covered elsewhere:

The security mechanisms themselves are in Authentication. What a host may do with the identity it is given is in Deciding What a Requester May Do.

11.16.1. The pieces

Piece

Where

Role

Listener

src/mca/ptl/base/ptl_base_listener.c, src/mca/ptl/tcp/ptl_tcp.c, src/mca/ptl/usock/ptl_usock.c

Has each transport open its listening socket - the local socket (ptl_usock.c) and TCP (ptl_tcp.c) - then publishes the URIs, writes the rendezvous files and accepts connections.

Connection handler

src/mca/ptl/base/ptl_base_connection_hdlr.c

Reads and checks each new connection’s request, runs the checks, calls the host, replies.

Connecting side

src/mca/ptl/base/ptl_base_connect.c, ptl_base_fns.c, src/mca/ptl/base/ptl_base_client.c

Builds and sends the connection request, reads the replies.

Security (psec)

src/mca/psec/{native,munge}

Creates a credential on the connecting side; checks it, and runs any further exchange, on the server.

Host up-calls

client_connected2, tool_connected2, client_finalized in pmix_server_module_t(5)

The host’s decision to accept a connection, and its notice that a process has finished.

Departure handling

lost_connection in ptl_base_sendrecv.c; the PMIX_FINALIZE_CMD case in pmix_server_switchyard.c; pmix_server_registration.c

Unwinds a connection and the server state that depends on it.

There is no listener thread. Accepting, reading a connection request, and every check after it run on the PMIx progress thread.

11.16.2. Rules the connection path follows

Every phase below is written to these rules. They are what makes the path predictable for the peer and for the host.

  1. One identity, settled before anyone acts on it. The peer’s user and group are settled - its credential checked and any further exchange complete - before any policy check is made and before the host is told anything.

  2. A connection changes nothing until it is accepted. Checks run against a scratch copy of the peer’s identity. Nothing the host registered, and nothing another connection relies on, is modified by a connection that may yet be refused.

  3. Every check that can refuse runs before the host is asked. Once the host accepts a connection, only the final replies remain.

  4. The progress thread is never held. No step waits on a peer without a time limit, and a slow peer does not stop the server serving everyone else.

  5. A refused peer is told why. Every refusal sends a status before closing, so the peer can report something more useful than “server unreachable”.

  6. Every failure undoes exactly what was done. Nothing is left behind, and nothing is released twice.

  7. Clients and tools are checked alike. The same checks run in the same order on both paths, except where a check only makes sense for one of them.

  8. The host is told where each item came from: the identity the security component settled, the registration the host made, or what the peer’s PMIx library supplied.

  9. Once the host has accepted a connection, it can learn how it ends: a finalize through client_finalized, and a lost connection through PMIX_ERR_LOST_CONNECTION delivered to the host’s own event handlers. A client is the host’s child, so the host also sees it end directly.

  10. A departure is handled the same way however it is noticed - by the socket closing without a finalize, or by the host deregistering the process. A finalize alone is not a departure: the process may initialize again.

11.16.3. 1. Setting up the listener

A server sets up its listener during PMIx_server_init. A tool sets one up only if it is a launcher or a scheduler, after it has connected to its own server.

Setting up happens in two steps:

  1. pmix_ptl_base_create_listener opens the sockets and publishes how to reach them. Nothing is accepted yet.

  2. pmix_ptl_base_start_listening, at the very end of init, starts accepting.

Between the two, the server finishes initializing. A peer that connects in that window waits in the kernel’s backlog until the server starts accepting.

A server listens on its local socket, where every client and tool of this release on its host connects. It listens on TCP only for those who use it: clients and tools of earlier releases, when it takes them (ptl_base_legacy_clients, or the host’s PMIX_SERVER_LEGACY_PEERS; off by default, and only in a PMIx configured --enable-legacy-clients). A server takes connections from its own host only: a host that asks for connections from other hosts (PMIX_SERVER_REMOTE_CONNECTIONS set to true) fails PMIx_server_init. A server that cannot open a socket it needs fails PMIx_server_init too.

Important

A server that does not take legacy clients cannot run clients of earlier releases. Such a client - an Open MPI 5 process, say, with its bundled PMIx v5.0 - is given only the local socket’s URI, which it cannot use. Its connection fails, it logs one PMIX ERROR: BAD-PARAM line, and it comes up as a singleton: an N-process job runs as N separate one-process jobs. Nothing the server could set instead makes every earlier release stop rather than run alone - to such a client, no server and an unusable one both mean “singleton”. The remedy is the site’s: build with --enable-legacy-clients and turn legacy clients on (ptl_base_legacy_clients, or the host’s PMIX_SERVER_LEGACY_PEERS), or update the clients along with their servers.

The other direction needs only a build option. In a PMIx configured --enable-legacy-servers, a client or tool of this release started by a server of an earlier release (v3.2 on) - a container’s application under an older resource manager - connects to it over TCP, at the address and in the wire format the server’s environment or rendezvous file gives it. Without that option, the default, such a server is refused: before the process connects, naming the server’s release, and with PMIX_ERR_NOT_SUPPORTED, which fails a client’s PMIx_Init rather than making it a singleton. Any server address over TCP is refused the same way, since only a server of an earlier release is reached at one. With neither direction built - the default - the library has no TCP transport at all.

11.16.3.1. The local socket

The local socket is an AF_UNIX socket. The kernel identifies the process at the far end of each connection to it - its uid, gid and pid - whatever that process claims.

  • It is pmix-<pid>, in a directory of its own that the server makes beneath its temporary directory: <tmpdir>/pmix-XXXXXX.

  • The directory is mode 0700 and the socket 0600: only the server’s own user can connect. A host widens that by passing PMIX_SOCKET_MODE, and PMIX_USERID and PMIX_GRPID for the socket’s owner and group, to PMIx_server_init. The socket takes its mode and owner while the directory is still closed; the directory, which stays the server’s, is then opened to whom the mode lets in - search permission, never read. The server holds the directory open from the moment it makes it, and makes every change in it through that descriptor - the bind too, where /proc/self/fd reaches the directory by it.

  • The path must fit an AF_UNIX address, about 100 characters. A temporary directory too long for it fails init; there is no fallback. The default $TMPDIR on macOS is too long - pass a shorter PMIX_SERVER_TMPDIR.

  • At finalize the server removes the socket and its directory. A server that is killed leaves them behind.

11.16.3.2. Choosing an interface

For TCP, the server considers each network interface that has an IPv4 address (or IPv6, if enabled), skipping any the if_include / if_exclude settings rule out. Then:

  • It listens on loopback only. Only processes on the same node can reach it.

  • With no loopback interface left, it fails its init. It never takes a public interface instead.

The port is the first free one from ptl_base_ipv4_ports (or the PMIX_TCP_IPV4_PORT directive), and by default is chosen by the kernel. Each socket is close-on-exec and non-blocking.

11.16.3.3. Publishing the address

Each address is a URI:

<nspace>.<rank>;usock://<path>
<nspace>.<rank>;tcp4://<address>:<port>

(tcp6:// for IPv6). The server’s URI is its TCP one when it listens on TCP, and its local socket’s when it does not. The server stores it under PMIX_SERVER_URI, reports it, and puts it first in its rendezvous files. A TCP URI reaches older peers, which cannot use the local socket. PMIX_MYSERVER_URI is always the local socket’s URI: it is the one to hand a process of this release, such as a launcher that is to connect back (PMIX_LAUNCHER_RNDZ_URI). To give the server’s address to a process of a release before v7.0, use PMIX_SERVER_URI instead. When the server has one, the local socket’s URI is given beside it where a process on the host reads it: the environment of the processes the server starts (see below) and the rendezvous files.

The server can also report the URI on request (ptl_base_report_uri or the PMIX_TCP_REPORT_URI directive):

Value

Legacy clients on

Legacy clients off

- (stdout), + (stderr)

the local socket’s URI

the local socket’s URI

tcp- (stdout), tcp+ (stderr)

the TCP URI, for a reader of an earlier release

refused at init: there is no TCP listener

a number (an open pipe, then closed)

<varnames>;<tcp uri> and <varnames>;<usock uri>, each ending in a NUL, in one write

<varnames>;<usock uri> and a NUL

anything else (a file name)

line 1 the TCP URI, line 2 the version, then the local:, bfrops: and psec: lines

line 1 the local socket’s URI, line 2 the version

So - and + always print what a client or tool of this release uses, and tcp-/tcp+ are for a person pointing a reader of an earlier release at this server. A program reading the pipe and linked to an earlier release - Open MPI’s, say - takes the first string, as it always has; one linked to this release takes the second. A file is left mode 0644: only the server’s user can write it. One already at the name is reused only if it is that user’s; another user’s stops the server starting.

11.16.3.4. Rendezvous files

A server that supports tools writes rendezvous files, through which tools find it, in a directory only its user can write: <system tmpdir>/pmix-user.<uid> - or, when something other than its own directory is at that name, $XDG_RUNTIME_DIR/pmix, where that is a directory of its user’s that no other user can write. A root system server, scheduler or system controller writes its file in /run/pmix. A tool looks there, and in the directories of the other users whose servers it may talk to; it walks the whole system tmpdir, as an earlier release did, only with legacy servers on (ptl_base_legacy_servers). The details are in Transport Layer (Client/Server/Tool Connections). Each file holds the URI, the server’s version, pid, uid and gid, the local socket’s URI, the wire formats and security mechanisms it accepts, and the lock that shows it is running (see A server the kernel cannot name). The server removes the files it wrote when it finalizes.

11.16.3.5. Accepting

Once listening, the progress thread runs the accept handler whenever a connection arrives:

  1. accept the connection and make it non-blocking and close-on-exec.

  2. Add it to the list of pending connections - accepted, but not yet delivered their connection request.

  3. Start a timer of ptl_base_connect_ack_timeout seconds (default 5). A connection that has not delivered its whole request in that time is closed.

  4. If ptl_base_max_pending_connections connections are already pending (default: an eighth of the process’s file-descriptor limit), close the oldest of them. Tools waiting for the host’s decision are not pending connections and are never the ones closed. They have a limit of their own, the same number: once that many tools are waiting, a further tool is refused with PMIX_ERR_OUT_OF_RESOURCE. A user other than the server’s, root and the host’s PMIX_USERID may have at most a quarter of that number waiting (at least one); a further tool of that user is refused the same way.

  5. Allocate a connection’s request buffer as its bytes arrive: 4 KB at first, doubling up to the length the request names. If the buffers of all pending connections would then pass ptl_base_max_pending_bytes (default 16 MiB, 0 for no limit), close that connection.

If accept itself fails, the handler retries per-connection errors at once. When the process is out of descriptors or memory, it pauses accepting for a second at a time until the condition clears.

At finalize the server closes its listening sockets, closes any pending connections, and removes its rendezvous and URI files.

11.16.4. 2. Telling a client where to connect

A client never searches for its server. The host tells it, through the environment, before the client starts. The host:

  1. registers the job (PMIx_server_register_nspace);

  2. registers the client, with the uid and gid it will run as (PMIx_server_register_client);

  3. calls PMIx_server_setup_fork(3) to add the connection information to the client’s environment;

  4. starts the client with that environment.

A client is local to its server by definition: the host registers it and prepares its environment on the server’s node. It connects to the local socket.

PMIx_server_setup_fork sets:

Variable

Value

Used by the client to

PMIX_NAMESPACE, PMIX_RANK

the client’s identity

name itself to the server

PMIX_SERVER_LOCAL_URI

the local socket’s URI, then ;<uid>:<gid> - the effective uid and gid the server runs as

connect to the local socket, and check that the server there is the one that started it. Clients of earlier releases ignore it

PMIX_CONNECT_TOKEN

<server nspace>.<server rank>; then 128 random bits, in hex, that that server made for this rank and keeps

present them to the server of that name, however it reaches it, and to no other. Clients of earlier releases ignore it

PMIX_SERVER_URI<NN>, one per wire format the server accepts (for example PMIX_SERVER_URI70, PMIX_SERVER_URI61)

the server’s TCP URI when it takes clients of earlier releases, else its local socket’s URI

choose the newest wire format both sides share: the first variable the client recognizes wins. A client of an earlier release also connects to this URI. Given the local socket’s, which it cannot use, it depends on the release: a v4.x or v5.0 client initializes as a singleton and returns PMIX_ERR_UNREACH (a v5.0 client also prints a PMIX_ERR_BAD_PARAM error line as it does); a v2.1 or v3.2 client does not come up as a singleton - its PMIx_Init fails

PMIX_VERSION

the server’s version

know which features the server has

PMIX_SECURITY_MODE

the security mechanisms the server accepts

choose its own mechanism from that list

PMIX_BFROP_BUFFER_TYPE

the buffer type

format its messages as the server expects

PMIX_GDS_MODULE

the datastores the server offers

choose how to receive job data

PMIX_SERVER_TMPDIR, PMIX_SYSTEM_TMPDIR

the server’s temporary directories

place its own temporary files

PMIX_HOSTNAME

the server’s name for this node

use the same node name as the server

A client takes these out of its environment as PMIx_Init begins - all but PMIX_SERVER_TMPDIR, PMIX_SYSTEM_TMPDIR and PMIX_HOSTNAME - and keeps them for its own process, where a later PMIx_Init of the same process reads them. A tool does the same at PMIx_tool_init. Both also take PMIX_LAUNCHER_RNDZ_URI, PMIX_SERVER_NSPACE and PMIX_SERVER_RANK, which a launcher sets for a process it starts itself. The token is overwritten where the environment held it. A process the client or tool starts once it has initialized - forked, or forked and exec’d - and a program it execs inherit none of them, and a process forked from it does not find the ones it kept. Only the C library’s environment is changed: a language runtime’s own copy of it, such as Python’s os.environ, still holds them (the PMIx Python bindings take them out of that copy too).

A server takes none of them out: the host’s environment is the host’s. The server adds the variables above to each client’s environment one client at a time, and does not pass its own environment on. The one place the library does - a process it starts itself, with no host to start it (PMIx_Spawn from a launcher, or a server whose host has no spawn up-call) - it passes its environment on less these variables, and gives the process its own.

A process started with none of these is a singleton. It looks for a system-level server on its node (the pmix.sys.<host> rendezvous file) and, if it finds one, connects to it; otherwise it runs without a server.

11.16.5. 3. Opening a connection

A client connects inside PMIx_Init, and a tool inside PMIx_tool_init, on the caller’s thread. Each blocks until the connection is complete or has failed. A tool attaching to another server later (PMIx_tool_attach_to_server) connects from the progress thread, without blocking it.

A client takes the local socket’s URI from its environment (or a URI from a PMIX_SERVER_URI directive to PMIx_Init). A tool finds its server as described in Transport Layer (Client/Server/Tool Connections); a rendezvous file of this release gives it the local socket. A tool given a TCP URI directly (PMIX_SERVER_URI or PMIX_TCP_URI, or the launcher’s PMIX_LAUNCHER_RNDZ_URI) does not connect, and fails with PMIX_ERR_NOT_SUPPORTED (help topic tcp-uri-alone): a server of this release takes it on its local socket, and one of an earlier release through its rendezvous file (file:<path>), which gives the server’s version. From there the two do the same:

  1. Open a connection to the URI’s address - only an address of this host. A TCP address that is not this host’s - not a loopback address, and not an address one of its interfaces holds, by the interface list PMIx made at init or by the kernel’s own list - is not dialled, and the process says why. A tool fails with PMIX_ERR_UNREACH; a client, which was told where its server is, fails PMIx_Init with PMIX_ERR_COMM_FAILURE rather than coming up as a singleton. Each attempt is bounded by ptl_base_handshake_wait_time (default 60 seconds). A process given the local socket connects there and nowhere else: if it cannot reach the socket, it fails to initialize, and says why.

  2. Check the server (below).

  3. Send the connection request. On the local socket that is an offer, answered by the server’s choice, and then the credential (see The connection request). Each send is bounded by ptl_base_handshake_wait_time too, so a server that stops reading ends the connection with PMIX_ERR_TIMEOUT.

  4. Read the replies, each also bounded by ptl_base_handshake_wait_time.

  5. Arm the connection for normal traffic.

11.16.5.1. Checking the server

On the local socket the kernel names the server as well: the uid and gid of the process listening there. A process checks them before it sends its request:

  • A process the server started was told who the server is, in PMIX_SERVER_LOCAL_URI. The kernel’s uid and gid must equal those. A process in a user namespace of its own - a rootless container, say - is shown the server’s ids as its namespace maps them; it maps them back to its parent namespace’s, through /proc/self/uid_map and gid_map, before comparing. A server whose ids its namespace does not map - the kernel shows them as the overflow id, 65534 - cannot be named; see “A server the kernel cannot name” below.

  • Any other process - a tool that found the server through a rendezvous file, say - talks to a server running as its own user or as root, or as a user or group the ptl_base_server_access parameter names (users=<list>,groups=<list>). The gid checked is the server’s effective gid.

A server that fails the check is not talked to. The process reports the uid and gid it found and what it expected.

Over TCP, the kernel names the owner of the server’s end of the connection when the server is on the same host, from its table of TCP connections - a user, but no group. A process applies the same rule by user alone: a server running as its own user, as root, or as a user ptl_base_server_access names. A server the table does not show is refused (PMIX_ERR_INVALID_CRED), and so is one the process could not ask about because it is out of descriptors or memory. A table that cannot be read - /proc mounted with hidepid, or kept from a container - counts as no table. When the table has no answer:

  • a connection whose far end is no longer there - the server closed it - fails with PMIX_ERR_UNREACH;

  • a far end that is not at an address of this host - one reached through another network namespace, say - is refused with PMIX_ERR_UNREACH, saying the address is not this host’s;

  • a server on this host when the kernel has no table to ask is connected to only when its source vouches for it - see “A server the kernel cannot name” below.

11.16.5.2. A server the kernel cannot name

The kernel’s answer is used whenever it names a real user. It cannot in three cases:

  • TCP with no table of connections. Linux, macOS and FreeBSD 12 and later have one. Any other platform has none.

  • TCP with a table that cannot be read, such as /proc mounted with hidepid, or restricted in a container, where neither the sock_diag netlink interface nor /proc/net/tcp answers.

  • An id from outside the process’s user namespace. The kernel shows such an id as the overflow id (65534), on the local socket and in the TCP table alike. A process finds it has no entry in /proc/self/uid_map (or gid_map). This happens in a rootless container whose server runs on the host as root - Slurm’s OCI containers, for one. Linux only: elsewhere every id is named.

Such a server is connected to only when the address came from a source that vouches for it - one the user or the launcher answers for:

  • A client’s server, always: it is the one named for the client, by the environment it was started with or by its caller.

  • A server that named itself in the environment of a tool or other process it started.

  • A URI passed directly (PMIX_SERVER_URI or PMIX_TCP_URI), including the launcher’s PMIX_LAUNCHER_RNDZ_URI.

  • A file the caller named, as file:<path> or PMIX_TOOL_ATTACHMENT_FILE, when:

    • it is the file at the name, not a link to one nor one of several names for it, and a regular file only its owner can write;

    • its owner is the user the process was told to expect, the process’s own user, root, a user ptl_base_server_access names, or an id from outside the process’s user namespace. A process inside a namespace cannot make a file such an id owns, so the file was made outside, and the user named it.

    What release the file names, and whether its server is still running, play no part.

A server found by a search, or at a fixed name in a shared directory, never vouches. Neither does anything else.

The user is responsible for passing the right address. A source that vouches is taken at its word: PMIx cannot check, for such a server, who runs it.

A server nothing vouches for is refused with PMIX_ERR_INVALID_CRED, and the process says to name the server’s file or pass its URI (help topics server-unnamed and server-unmapped). A server of v7.0 or later also listens on a local socket, where the kernel names its user outside a user namespace; a process whose file names that socket connects there.

A server that runs as nobody, or as a service account, on the host itself has a mapped id - it is named, not unnamed - so the normal rule applies: it is refused unless ptl_base_server_access names that user. That holds too for a client of an earlier-release server run by a service account.

A process of an earlier release does none of this; the server checks it as any other process on its host - see The native Mechanism for a platform with no table.

11.16.5.3. The connection request

On the local socket, which only peers of this release use, the request is three messages:

  1. The offer, from the peer: a message header, then

    • the peer’s PMIx version;

    • the kind of peer (one byte, as in the table below);

    • its namespace and rank, for the kinds that send them;

    • the names of its psec components;

    • the names of its bfrops components, the one it prefers first;

    • the buffer type it prefers (one byte);

    • the names of its gds components, the one it prefers first.

    Each list is a one-byte count followed by that many names, each ending in a NUL. A list holds at most 16 names (PMIX_PTL_OFFER_MAX_NAMES) and a name at most 64 bytes (PMIX_PTL_OFFER_MAX_NAMELEN); an offer past either is refused with PMIX_ERR_BAD_PARAM. The offer carries no uid or gid: those are the kernel’s.

  2. The choice, from the server: a status, and on success the length of what follows and the chosen names - security mechanism, wire format, buffer type (one byte), datastore. The server chooses:

    • security: the first of the server’s active mechanisms, in priority order, that the peer offers - munge when both have it, else native. If the peer offers none of them it is refused with PMIX_ERR_NOT_SUPPORTED.

    • wire format and buffer type: the namespace’s, if a peer of the namespace has already connected and the kernel’s uid and gid for the peer are those of the rank it names (or, for a rank not yet in the namespace, of the namespace’s owner and every process in it) - a peer that does not offer it is refused with PMIX_ERR_NOT_SUPPORTED. Otherwise - a new namespace, or a peer of another uid or gid - the first wire format the peer offers that the server has, and the peer’s buffer type.

    • datastore: the first the peer offers that the server has. It is the peer’s own; the first peer’s also becomes its namespace’s default.

    The peer refuses a choice it did not offer: its init fails with PMIX_ERR_NOT_SUPPORTED.

  3. The credential, from the peer: a message header, the credential’s length and the credential, made with the chosen security mechanism (see below), and then the information field, packed in the chosen wire format and buffer type.

The server answers the credential with the replies below. The three messages share one deadline: the connection’s ptl_base_connect_ack_timeout, from when it was accepted.

Over TCP, which only peers of earlier releases use, the request is one message: a header followed by these fields, in this order. The order is a wire-format contract: a server and its peers can be different PMIx releases, so fields are only ever appended.

#

Field

Content

1

security mechanism

name of the psec component the peer uses

2

credential length

3

credential

created by that component (see below)

4

kind of peer

one byte: see the table below

5

identity

depends on the kind: some or all of uid, gid, namespace, rank

6

version

the peer’s PMIx version

7

wire format

name of the bfrops component the peer uses

8

buffer type

9

datastore

name of the gds component the peer uses

10

information

packed pmix_info_t array; everything to the end of the message

The kinds of peer:

Value

Kind

Sent by

Identity sent

0

simple client

a client started by the server

namespace, rank

1, 2

legacy tool, legacy launcher

tools and launchers of old releases only

uid, gid

3

tool needing an identifier

a tool that has no namespace or rank of its own

uid, gid

4

tool given an identifier

a tool given PMIX_TOOL_NSPACE and PMIX_TOOL_RANK; also a server connecting to another server

uid, gid, namespace, rank

5

tool started by a server

a tool whose environment names it (PMIX_NAMESPACE, PMIX_RANK)

uid, gid, namespace, rank

6, 7, 8

launcher needing / given / started with an identifier

as 3, 4, 5, for a launcher

as 3, 4, 5

9

singleton

a client with no server of its own, connecting to a system server

uid, gid, namespace, rank

10

scheduler

a scheduler

uid, gid, namespace, rank

The information field carries, from a client, its pid (PMIX_PROC_PID, which the client’s PMIx library obtains with getpid()) and command line (PMIX_CMD_LINE). From a tool it carries its pid, real and effective uid and gid, its roles (PMIX_LAUNCHER, PMIX_SERVER_SCHEDULER, PMIX_SERVER_SYS_CONTROLLER), its command line, and any directives it was given that are meant for the server.

11.16.5.4. The credential

Each psec component makes its own credential:

  • native sends the peer’s effective uid and gid. The server takes the uid from its kernel instead where it can, and takes the claim only where its kernel keeps no table of TCP connections and it takes legacy peers: see The native Mechanism.

  • munge sends a credential encoded by the MUNGE daemon. It carries no payload: the server refuses a munge credential that does. On the local socket, where the kernel names the server’s uid, the credential is restricted to that uid (MUNGE_OPT_UID_RESTRICTION), so only a process running as the server’s user can decode it. Over TCP, or where the uid differs between this process’s user namespace and its parent’s, it is unrestricted.

11.16.5.5. The replies

A client reads, in order:

  1. a status: success, or a request to continue with a security exchange;

  2. if requested, the client side of that exchange (no component of this release requests one);

  3. its index in the server’s table of peers.

A tool reads, in order:

  1. if requested, the security exchange;

  2. a status - the host’s decision;

  3. if it asked for one, its assigned namespace and rank;

  4. the server’s namespace and rank;

  5. a final status.

Any status other than success ends the connection, and the peer reports it. A connection the server closes without sending a status is reported as PMIX_ERR_UNREACH.

Once the replies are in, the peer marks itself connected, switches the socket to non-blocking, and arms it for normal traffic. A client then asks the server for its job information, completing PMIx_Init.

11.16.6. 4. Accepting a connection

Everything in this section runs on the server’s progress thread.

11.16.6.1. Reading the request

The connection handler reads the request as it arrives, without waiting: each time the socket is readable it takes what is there and keeps its place. A request larger than PMIX_MAX_CRED_SIZE (128 KB) is refused. Once the whole request is in, the connection leaves the pending list and its timer stops.

On the local socket the first message is the offer. The handler takes the peer’s uid and gid from the kernel, reads the offer, checks the version and kind below, and answers with its choice (see The connection request). The connection then goes back to waiting, on the same timer, for the credential, which it reads the same way. Over TCP the one message holds every field.

The handler then reads the fields in order. Two checks apply to every peer:

  • Kind of peer. An unknown value is refused.

  • Version. A peer from PMIx v1 or v2.0 is refused with PMIX_ERR_OUTDATED. See Version Numbers and Binary Compatibility for the supported floors. A peer of any release before v7.0 is refused with PMIX_ERR_NOT_SUPPORTED unless the server takes legacy clients, and on the local socket whether it does or not: no earlier release has that transport. A peer of v7.0 or later is refused on TCP with PMIX_ERR_LOCAL_URI_REQUIRED: it uses the local socket, where the kernel names it, and the TCP listener exists only to take earlier releases - on their claim, where the kernel cannot name them. The peer says which address it was given and what to use instead (help topics local-uri-required-uri and local-uri-required-file).

Before the security component is asked, a peer that connected over TCP from an address that is not this host’s - not a loopback address, and not the address of one of its interfaces - is refused with PMIX_ERR_NO_PERMISSIONS. A server takes peers on its own host only.

The rest depends on whether the peer is a client (kinds 0 and 9) or a tool (all the others).

11.16.6.2. Checks for a client

  1. Its components must be available. On the local socket the server chose them from the peer’s offer. Over TCP, the security, wire-format, and datastore components it names must be ones this server has loaded, or it is refused with PMIX_ERR_NOT_SUPPORTED.

  2. The client must be registered. Its namespace, and its rank within it, must have been registered by the host (PMIx_server_register_client). An unregistered client is refused with PMIX_ERR_INVALID_CRED.

  3. Its identity must be the registered one. The security component settles the client’s uid and gid (see Settling the identity), and they must equal the uid and gid the host registered for that rank. On the local socket the uid and gid the kernel reports for the client must equal them too, whichever security component is chosen; a client that differs is refused with PMIX_ERR_INVALID_CRED before its credential is validated.

  4. It must present its rank’s token. A rank the host registers with PMIx_server_register_client gets a token nobody is handed; preparing a process for it with PMIx_server_setup_fork replaces it with the one that process is given, and preparing it again replaces that. The server keeps it until the rank or its job is deregistered. A client of this release that names the rank must present it, on whichever socket it comes, or is refused with PMIX_ERR_INVALID_CRED before the host is asked - so until the rank is prepared, no client is taken for it. A rank neither registered nor prepared has none, and nothing is asked of it; nor is anything asked of the singleton a server was started for (PMIX_SINGLETON), which no host prepared. A client of an earlier release presents none, and is not asked for one: with legacy clients, a connection over TCP that declares an earlier release is taken without a token.

  5. Its wire format must match its job’s. All clients of one job use one wire format, set by the first to connect. This is checked once the client’s identity and token have been, and a client that differs is refused with PMIX_ERR_NOT_SUPPORTED.

A PMIX_USERID or PMIX_GRPID in a client’s information field is ignored. The identity the security component settles is the only one the server uses.

11.16.6.3. Checks for a tool

  1. Its security component must be available - over TCP. On the local socket the server chose it.

  2. Its identity is settled by the security component (see Settling the identity). The settled uid and gid replace whatever the tool claimed.

  3. A tool of another user is refused unless the host started the server with PMIX_SERVER_ALLOW_FOREIGN_TOOLS. A tool of the server’s own uid, of root’s, or of the uid the host named to PMIx_server_init as PMIX_USERID is not another user’s.

  4. A tool that names an identity must be entitled to it:

    • a tool started by a server (kind 5 or 8) whose rank is already known must have that rank’s uid and gid;

    • a tool of this release that names a rank with a token must present it, as a client must. Once a client connection has bound the token to its pid, a tool on the local socket from another pid is refused with PMIX_ERR_EXISTS, as for a rank that is connected; a tool’s connection binds nothing;

    • a tool joining a namespace the server already knows must have that namespace’s uid and gid, and the namespace must belong to a tool, or to a process this server started.

  5. Its wire format must be one this server accepts.

A tool the host has not given an identity - a self-started tool with no namespace - gets a new namespace, created by the server and named by the host.

11.16.6.4. Settling the identity

The security component - the one the server chose, on the local socket, or the one the peer named, over TCP - settles its uid and gid. How depends on the component:

  • native, on the local socket, takes the uid and gid the kernel reports for the connection. Over TCP it asks the server’s own kernel who owns the peer’s end of the connection, which gives the uid; a peer the table does not show is refused. A peer at an address that is not this host’s never reaches it (see above). On a platform with no table to ask, a TCP peer is accepted on its claim: the server has a TCP listener only when it takes legacy clients. A table the server could not read - it is out of descriptors or memory, say - is no answer, and the peer is refused. See The native Mechanism.

  • munge decodes the credential through the MUNGE daemon. The decoded uid and gid must equal the peer’s registered (client) or claimed (tool) identity. On the local socket a tool’s identity is the one the kernel reports, and a client’s kernel uid and gid have already been compared with its registration (see Checks for a client).

The result is one uid and one gid. They are what the host is given, and what every later access decision for this peer uses.

A peer from outside the server’s user namespace. A server in a user namespace of its own - in a container, say - is shown each peer’s ids as its namespace numbers them, and an id the namespace does not map as the overflow id (65534). On Linux, a peer the kernel reports with the overflow uid or gid, when the server’s /proc/self/uid_map (or gid_map) has no entry for that id, is refused with PMIX_ERR_INVALID_CRED - on the local socket at its first message, and over TCP when the owner the kernel reports is that uid. Where the map does have an entry for the overflow id - a subordinate-id range that covers it, as for a container whose processes run as nobody - that number is the id of a user of the namespace, and a peer reported with it is taken as that user, as is any other. The kernel shows a peer from outside the namespace with the same number, and nothing in the connection tells the two apart.

Settling the identity is a single step: no component runs an exchange with the peer.

It can wait on a service, though, and does so on the progress thread, where the server serves no other connection or event meanwhile. munge’s decode asks the MUNGE daemon, for every connection. And where the kernel does not give the peer’s groups - a TCP peer, or a platform with no such list - its group is checked against the passwd and group databases, which may ask a directory service through NSS; those answers are cached, so a later connection by the same user does not ask again while they last. A host whose MUNGE daemon or name service can be slow to answer should expect each delay to be the server’s too.

11.16.6.5. Calling the host

When every check has passed, the server asks the host. The peer waits.

A client goes to client_connected2 (or, if that is NULL, the deprecated client_connected). The host receives:

Item

Content

Comes from

proc argument

the client’s namespace and rank

the host’s registration; the client’s identity was checked against it

server_object argument

the pointer the host gave when it registered the client

the host

PMIX_USERID, PMIX_GRPID

the settled uid and gid

the security component

PMIX_CRED_TYPE

the name of the security component that settled them

the server

PMIX_PROC_PID

the client’s process ID

the client’s PMIx library, which calls getpid(). The kernel does not report the pid of a TCP peer. A client in a container with its own process-ID namespace sends its pid within that namespace, which is not the pid the host sees.

PMIX_REALUID, PMIX_REALGID

the client’s real uid and gid, if it sent them

the client’s PMIx library

PMIX_TIMEOUT

the seconds the host is asked to answer within

the server’s ptl_base_handshake_wait_time, less what the peer has waited for this reply already; absent when that is 0

A tool goes to tool_connected2 (or the deprecated tool_connected). The host receives the tool’s whole information field, with these set by the server:

Item

Content

Comes from

PMIX_USERID, PMIX_GRPID

the settled uid and gid

the security component

PMIX_CRED_TYPE

the name of the security component that settled them

the server

PMIX_NSPACE, PMIX_RANK

the identity the tool gave, if any

the tool; checked as above where the server already knows it

PMIX_VERSION_INFO

the tool’s PMIx version

the tool’s PMIx library, in its connection request

PMIX_TIMEOUT

the seconds the host is asked to answer within

the server’s ptl_base_handshake_wait_time, less what the peer has waited for this reply already; absent when that is 0

The rest is passed on as the tool sent it: PMIX_PROC_PID, PMIX_REALUID and PMIX_REALGID, which the tool’s PMIx library obtains from the operating system; PMIX_CMD_LINE; and the roles the tool asked for.

If the host provides neither connection up-call, a client is accepted, and so is a tool that brought its own identity. A tool that needs an identity assigned is refused.

The host decides on one tool of a name at a time. A tool that brings its own namespace and rank waits in a queue, in the order the tools’ identities were settled, while the host decides on a tool of that name - or while one of that name is not yet done with after a late answer (see When the host’s answer does not stand). So the host never decides on two tools for one name at once:

  • where a tool belongs - the namespace and rank it joins, and whether that rank already has a connection - is checked when its turn comes, so it sees every tool accepted ahead of it;

  • when the host accepts a tool, any queued tool that names the same namespace and rank is refused with PMIX_ERR_EXISTS at once, and any that names the same namespace as another user is refused with PMIX_ERR_NOT_SUPPORTED. The host is not asked about either;

  • a queued tool that has closed its connection by its turn - its own connect timed out, say - is dropped without asking the host;

  • when the server finalizes, the tools still queued are refused with PMIX_ERR_UNREACH.

Tools of other names, and tools that need an identity - which have no name until the host gives them one - are put to the host together. The host keeps the names it gives unique; it cannot know of a name a tool took for itself until it is asked. A tool the host names while a tool that took that name is still being decided on is checked again when it is accepted: whichever is accepted second is refused with PMIX_ERR_EXISTS, and a tool given a rank in a namespace that appeared while the host decided - another tool’s, accepted meanwhile - joins it only if every process in it is the tool’s user.

A slow answer therefore delays the tools of that name behind it, and their time in the queue counts against their own wait.

11.16.6.6. What the host must do

  • Decide. Check that the client is a process you started, with the pid given in PMIX_PROC_PID; check that the tool’s user may use this server, in the roles it asked for. See Accepting Connections.

  • Record the identity. Keep the PMIX_USERID and PMIX_GRPID you were given. They are the identity for every later request from this peer, and the library passes them again on every up-call it makes for it. See Deciding What a Requester May Do.

  • Answer exactly once. Either return an error, or return PMIX_SUCCESS and later call the callback, once, from any thread. Never both. client_connected2 may instead return PMIX_OPERATION_SUCCEEDED to accept at once without calling the callback. tool_connected2 may not, because the tool’s assigned identity can only travel through the callback.

  • Answer within PMIX_TIMEOUT. The peer is waiting, bounded by its own ptl_base_handshake_wait_time, and tools of its name behind it in the queue wait too. PMIX_TIMEOUT is what the peer has left of that wait; a peer with none left is refused with PMIX_ERR_TIMEOUT without the host being asked. The up-call itself runs on the progress thread, so it must not block. A host that cannot decide in time should answer no.

  • Handle PMIX_ERR_LOST_CONNECTION for a process it accepted. See When the host’s answer does not stand.

11.16.6.7. When the host’s answer does not stand

The server waits for the host’s answer for PMIX_TIMEOUT and two seconds more. Then it refuses the peer with PMIX_ERR_TIMEOUT, and keeps only what it needs for the answer, which the host still owes: the host holds the callback’s cbdata. Until the answer comes, the peer’s name stays held - a client’s rank is refused to any other connection, and a tool of the same name stays in the queue - so that what follows cannot name a process accepted since.

A yes the server cannot act on is followed by a PMIX_ERR_LOST_CONNECTION event naming the process, delivered to the event handlers the host registered in its own process, as for any process that leaves without finalizing. That happens when:

  • the yes comes after the server stopped waiting;

  • the peer has gone by the time the server replies to it;

  • the host deregistered the client, its job, or the tool’s rank while it decided;

  • the peer’s wire format no longer fits its job;

  • the server could not record the connection.

The name stays held until the event has been given to the host’s handlers, as it is while the server waits for the answer, so nothing accepted under the name meanwhile can be the process the event names. The event is not raised when a live connection has the name by then: it would name that connection. A host with no handler for the event is not told, as for any lost connection.

A no, early or late, is followed by nothing.

11.16.6.8. Replying to the peer

When the host accepts, the server:

  1. adds the peer to its table of peers;

  2. sends the remaining replies (see The replies);

  3. makes the socket non-blocking and arms it for normal traffic;

  4. sends the peer any event notifications cached for it.

When the host refuses, the server sends the host’s status to the peer and closes the connection.

11.16.6.9. One connection per rank

A rank has at most one connection at a time. A process a client starts once it has initialized inherits none of its identity (see 2. Telling a client where to connect), and calls PMIx_Init as a singleton. One that does have it - started before the client initialized, or given it by the host or a job script - and connects as the same rank is a clone, in the Standard’s terms. The server does not accept clones.

  • A client connection for a rank that already has a connection - live (not finalized, socket open) or waiting on the host’s answer - is refused with PMIX_ERR_EXISTS, before the host is asked.

  • A tool joining a rank that already has a connection is refused the same way - checked before its wire format, so a tool is told its rank is taken rather than that its format does not fit the namespace. A tool that took its namespace and rank from its environment - one started from a job script, say, which inherits the script’s - then connects again as a tool that needs an identity, and the host gives it a namespace of its own; if that connection fails too, the tool is told PMIX_ERR_EXISTS. Tools of earlier releases do not connect again: their PMIx_tool_init returns PMIX_ERR_EXISTS. A tool connects again only while PMIx_tool_init runs: a tool that attaches to a server later keeps the identity it was initialized with, and so does a tool whose caller gave it one (PMIX_TOOL_NSPACE and PMIX_TOOL_RANK).

  • The first process to connect holds the rank. A tool that takes a job script’s rank before the script’s application connects leaves the application refused: the server cannot tell such a tool from one that is the script’s only PMIx process, so the script must start its tools without the script’s PMIX_NAMESPACE and PMIX_RANK, or with an identity of their own.

  • A tool with the server’s address in its environment but no namespace and rank asks the host for an identity. A tool of an earlier release - v7.0.0rc1 among them - in that position says instead that it is a tool on a client’s rank, and sends an empty name: the server takes it as a tool that needs an identity, and the host names it, but the name is not sent to the tool, which reads none. Any other connection that names itself with an empty namespace or an invalid rank is refused with PMIX_ERR_BAD_PARAM: an empty namespace would match every namespace where names are compared.

  • A rank whose earlier connection finalized, or was lost, can connect again.

Every socket the library creates or accepts is close-on-exec, so a process’s connection closes when it execs - on the local socket and over TCP alike, for a client and a tool. The server handles that as it handles any connection that ends without a finalize: the rank is lost at once (see When a connection is lost), whether the process runs on or not. The program the process execs is a new process to the server.

The server never compares a new connection’s pid with the pid of the rank’s live connection: a second connection for a rank that has one is refused with PMIX_ERR_EXISTS whatever pid it comes from.

Libraries of earlier releases leave their connection open across an exec. The program such a process execs inherits the socket, and cannot use it: the rank keeps its connection until that program exits, and if the program calls PMIx_Init or PMIx_tool_init for the rank, it is refused with PMIX_ERR_EXISTS meanwhile. A process of this release whose server is of an earlier release closes its connection when it execs, as it does with any server, and that server handles the connection as lost.

A rank’s process that died can connect again - restarted by the host, say - before the server has read its old connection’s close. Before refusing, the server looks at the old socket: one that has closed in order or been reset is handled as a lost connection there and then, and the new connection goes ahead. One with unread data on it, or one that reports any other error, is live.

PMIX_ALL_CLONES_PARTICIPATE, which the Standard defines for collectives, has no effect: a rank has one process.

11.16.7. 5. Ending a connection

A connection ends in one of five ways. The server must handle each one completely: release the peer, account for it in every operation it was part of, and let the host know.

11.16.7.1. When a client finalizes

On the client, PMIx_Finalize:

  1. returns PMIX_ERR_WOULD_BLOCK if called from the progress thread;

  2. if the client was started with PMIX_EMBED_BARRIER, joins a fence of its job;

  3. sends PMIX_FINALIZE_CMD to the server and waits for the reply, at most pmix_server_client_fintime seconds (the finalize_timeout MCA parameter, default 10);

  4. stops its progress thread, closes the connection, and releases its state.

On the server, PMIX_FINALIZE_CMD:

  1. removes the client’s event registrations and output-forwarding requests, fails requests waiting on its data, and drops cached notifications addressed to it;

  2. calls the host’s client_finalized;

  3. once the host calls back, marks the peer finalized and replies.

When the client’s connection then closes, the server releases the connection. It keeps the rank’s entry - a tombstone - so that the same rank can initialize again later and be recognized. See Client Init/Finalize Lifecycle and State Management.

A finalized rank remains a member of its job, because the process may call PMIx_Init again and rejoin. An MPI Sessions program does this once per session, and a fast rank can open the next session’s fence before a slow one has finished the last. So a collective that names a finalized rank keeps waiting for it to come back and contribute. See Specification: Tracking Local Collective Participation, “A graceful finalize is not a departure”.

The server cannot tell a rank that will return from one that will not. The host can: deregistering the client (see When the host ends it) tells the server the process is gone for good. Where the host does not deregister, a collective whose participant finalized and never returns waits until its PMIX_TIMEOUT, or indefinitely if it has none.

What the host must do:

  • release what it holds for that process;

  • call the callback, which releases the client. The client waits only up to its own timeout;

  • not deregister the client from inside the up-call before calling the callback.

11.16.7.2. When a tool finalizes or disconnects

PMIx_tool_finalize sends PMIX_FINALIZE_CMD to the tool’s primary server, waits for the reply, and closes all its connections. The primary server handles a tool’s finalize as it does a client’s, and calls client_finalized for it too; server_object is then NULL. (The capability flag PMIX_CAP_TOOL_FINALIZED says the library does this.) Any other server sees its connection close, and handles it as a lost connection (see When a connection is lost). That is deliberate: for a tool, whether it exited or crashed makes no difference to a server.

PMIx_tool_disconnect ends the connection to one server, without a message. It is not a finalize, but the server cannot tell it from one that failed, and handles it as a lost connection. Requests the tool still had outstanding with that server complete with PMIX_ERR_UNREACH.

The server treats a disconnected tool as gone, because once a tool has disconnected neither the server nor the host can tell when it later ends: if its state were kept for a return, it would never be released. So a namespace the server created for a self-started tool is removed when the last process using it has gone, whether by finalize, disconnect or failure. A tool that connects again arrives as a new tool.

11.16.7.3. When a connection is lost

A process that exits, or is killed, without finalizing closes its connection without notice; so does a tool that disconnects. The server notices when a read or write on the socket fails - a process that execs closes its connection the same way (see One connection per rank). The server:

  1. closes the connection and stops its events;

  2. accounts for the process in every collective it was part of: a fence, connect, disconnect, or group construct still waiting for it completes with PMIX_ERR_PARTIAL_SUCCESS if others remain, or PMIX_ERR_LOST_CONNECTION if not;

  3. removes its event registrations and output requests, fails requests waiting on its data with PMIX_ERR_LOST_CONNECTION, and stops any monitoring of it. A request for its data made afterwards, while the rank is lost, is answered PMIX_ERR_NOT_FOUND at once unless the data had already arrived;

  4. raises PMIX_ERR_LOST_CONNECTION, naming the process, with range PMIX_RANGE_PROC_LOCAL. It goes to the event handlers the host registered in its own process with PMIx_Register_event_handler, and to no client or tool. It is not cached: a handler registered later does not receive it.

A process whose PMIx_Finalize request reached the server has finalized, even if its connection closes before the host answers that request - the client gives up waiting after a while. It is not counted as lost, and no event is raised for it.

If the rank connects again - the host started a new process for it - the server releases what it kept of the lost connection, and the new process starts with no data published: a request for its data waits for the new process’s, not the lost one’s.

For a client this is a second notice: the client is the host’s child, and the host sees it end. For a tool the host did not start, the event is the only notice that it left without finalizing.

A process that hangs without closing its connection is not detected by the server. The host’s own process monitoring covers that case.

What the host must do: register a handler for PMIX_ERR_LOST_CONNECTION if it accepts tools it did not start, and release what it holds for such a tool when the event names it (PRRTE does this); notice a client’s death through its own monitoring; generate the termination events applications expect (PMIX_PROC_TERMINATED, and PMIX_ERR_PROC_TERM_WO_SYNC for processes connected to it), and deregister the process when it no longer needs it.

11.16.7.4. When the server goes away

When a client’s or tool’s connection to its server fails:

  1. every request waiting on that server completes with PMIX_ERR_UNREACH;

  2. the process raises PMIX_ERR_LOST_CONNECTION, if the server was its primary one and it has not begun to finalize;

  3. from then on, calls that need the server return PMIX_ERR_UNREACH.

There is no automatic reconnection. A tool can attach to another server.

11.16.7.5. When the host ends it

Deregistering a client (PMIx_server_deregister_client(3)) tells the server that a process has ended and will not connect again. It is the only way the server learns that a finalized rank is gone for good: finalizing alone leaves the rank a member of its job (see When a client finalizes).

A host need not call it. Without it, the rank stays a member until its job is deregistered, and:

  • a collective that names a finalized rank that never returns waits until its PMIX_TIMEOUT, or indefinitely;

  • a request for data that rank never posted, made after it finalized, waits instead of being answered.

When to call it: once the process has terminated. Not from client_finalized - the process may still initialize again. PRRTE calls it for each local process when the process has terminated: its exit has been collected and its output drained. Its client_finalized only records that the process finalized.

What the server does:

  1. if the client is still connected, closes the connection and handles it as a lost connection, without calling the host back or raising an event: its event and output registrations end, and a request waiting on its data is answered PMIX_ERR_LOST_CONNECTION. So is one waiting on a rank that never connected. The host deregisters a live client only when it is about to kill it or believes it already dead, so the connection may fail on its own while this runs; the departure is still handled once;

  2. releases the network and GPU resources assigned to the process, stops monitoring it, and runs any epilogs registered for it;

  3. sends any reply still queued for it - typically the reply to its finalize;

  4. removes the rank’s entry. A later connection by that rank is refused, because it is no longer registered;

  5. records the rank as departed, so a later request for its data is answered PMIX_ERR_NOT_FOUND at once;

  6. accounts for its departure in every active collective that names it and has not received its contribution, as for a lost connection, and stops counting it among the job’s local processes, so a collective over the whole job started afterwards does not wait for it either. Whichever of this and the lost connection comes first does it, once;

  7. once every local process of the job has finalized or been deregistered, tells the network and GPU support that the job’s local processes are done.

The departure is counted because the rank can never contribute now, and because a collective’s PMIX_TIMEOUT is optional: without this step, a job whose rank finalized and exited would hang.

Deregistering a job (PMIx_server_deregister_nspace(3)) deregisters every client in it, removes the job’s data, events, and group memberships, and accounts for the departure of each of its ranks in every active collective that names it and has not received the rank’s contribution, so that collective completes on the participants that remain.

Finalizing the server (PMIx_server_finalize(3)) closes every connection. Clients and tools see their server go away.

11.16.8. Host obligations, in one place

  • Register a job and each of its clients before starting them, and call PMIx_server_setup_fork for each client.

  • Decide every connection in client_connected2 or tool_connected2: check that a client is a process you started, with the pid given in PMIX_PROC_PID, and that a tool’s user may use the server in the roles it asked for.

  • Record the PMIX_USERID and PMIX_GRPID you are given. They are the peer’s identity for every later authorization.

  • Answer each up-call exactly once, promptly, without blocking - a connection up-call within the PMIX_TIMEOUT it carries.

  • Be ready to give an identity to a tool you launched: one whose namespace and rank are already in use connects again without one.

  • In client_finalized, release your state for the process and call the callback. Do not deregister the process there: it may initialize again.

  • Watch your processes. Generate termination events when they end.

  • Register a handler for PMIX_ERR_LOST_CONNECTION: it is how you learn that a tool you did not start has gone without finalizing, and that a process you accepted never became connected.

  • Optionally, deregister each client once its process has terminated. Until you do, a rank that finalized is still expected in its job’s collectives.

A peer that vanishes while the server writes to it costs the server an error on that connection, never a SIGPIPE: no write PMIx makes to a connection’s socket can raise it, nor the write of its URI to a pipe the host names (PMIX_SERVER_REPORT_URI). Nor can a write forwarding output or input to a pipe or file of the library’s own, such as a child’s stdin. A host need not change its own handling of that signal for any of these. Output written to the process’s own stdout and stderr is written as the process writes it: if their reader goes, that write raises SIGPIPE as any other write the process makes there would.