3.5.2. Installing and Administering PMIx v7
For people who install PMIx, and the host environment that uses it (PRRTE, a resource manager), on a system. For what people running jobs will notice, see Running Jobs with PMIx v7.
The reference pages for this role are System Administration and its sub-pages; each section below links to the detail.
3.5.2.1. Installing v7
Keep one PMIx installation, the latest. Applications do not need to be rebuilt: a program built against PMIx v5.0 runs against the v7 library. See Choosing a Release.
Go from v5.0 straight to v7.0. Skip v6.x, which was announced “do not use”.
Upgrade the host environment at the same time. A host environment must be built against the PMIx release it runs with.
PRRTE v5 requires PMIx v7, and PRRTE v3 (paired with PMIx v5) refuses it. There is no PRRTE release that works with both. See PMIx and PRRTE Releases.
Rebuild any other host environment - a resource manager’s PMIx plugin, say - against v7.
Build requirements changed:
hwloc 2.1.0 or later (v5.0 took 1.11);
libevent only - support for libev is gone, along with
--with-libev;a C11 compiler with atomics;
--enable-c11-atomicsand--enable-builtin-atomicsare gone;from a git clone, Python 3.6 or later; flex is no longer needed;
Solaris and Sun Studio are no longer supported.
MUNGE is built only when asked for. v5.0 built the munge
component whenever MUNGE was installed. v7 builds it only with
--with-munge. If you use MUNGE authentication, add the option.
Compression: build every node the same way. v7 adds the zstd
and lz4 compression components, and prefers zstd to the zlib
ones when its development headers are installed. Compressed data
crosses nodes, and a node using zlib cannot read zstd or
lz4 data, so every node in a job must pick the same component.
This follows from a shared installation; it matters during a rolling
upgrade. See Optional support libraries.
3.5.2.2. Containers and applications of earlier releases
What changed. v7 clients and tools connect over a local socket. Releases before v7 connect over TCP, and a v7 server takes them only when both of these hold:
PMIx was configured with
--enable-legacy-clients. It is off by default.The support is turned on at run time: the MCA parameter
ptl_base_legacy_clients=true, or the host’sPMIX_SERVER_LEGACY_PEERSattribute. PRRTE does not pass the attribute, so for PRRTE set the parameter where its daemons read it (a parameter file, say).
Even then, the server listens for them on loopback only.
Warning
Do not set ptl_base_legacy_clients in a PMIx built
without --enable-legacy-clients. Every server, and every
launcher or scheduler (prte, prun), that reads it fails to
start. ptl_base_legacy_peers is safe to set in any build: it
turns on only the directions the build has. See
Processes of releases before v7.0.
What users see without it. Each process of an older application - a container with PMIx v5 inside, for example - prints one error line and runs as a singleton, unaware of its job. The job does not fail; it just does not work as a job.
What to do. Decide whether your systems run containers or
applications with an older PMIx inside. If they do, build PMIx with
--enable-legacy-clients and turn it on. If they do not, leave it off,
and tell users to update the PMIx in their images.
The other direction is a build option too. A v7 application runs
under servers from PMIx v3.2 on only if the PMIx it uses was configured
--enable-legacy-servers; by default it refuses a server of an
earlier release, and its PMIx_Init fails. Configure it where v7
applications - a container image, say - may run under an older resource
manager. --enable-legacy-peers turns on both directions. With
neither, the TCP transport and the earlier releases’ wire formats are
left out of the library.
3.5.2.3. Tools on login nodes
What changed. A tool - prun, pterm, a debugger, a monitor
- reaches a PMIx server only on its own node. PMIx no longer takes
connections from other nodes, and the ssl authentication that
served them is gone. A server asked to take them
(PMIX_SERVER_REMOTE_CONNECTIONS) fails to start.
What to do. Users run tools on the node where the server runs.
For a launcher or a debugger, that means ssh to the node where the DVM
controller (prte or mpirun) is running - the approach the DDT
debugger recommends. Make sure users can reach that node.
Coming. Launching from a login node will remain possible. The
method is being worked out: most likely prun will send its launch
request through PRRTE’s own messaging rather than through PMIx.
Debuggers are not settled yet.
3.5.2.5. Temporary directories
Keep
$TMPDIRshort. The local socket is created under it, and a socket path is limited to about 100 characters. The default on macOS is too long.The server checks its temporary directory at start. It refuses a directory another user could change: one on a path not owned by root or the server’s user, or one others can write without the sticky bit. Some container volumes are like that - a Kubernetes
emptyDir, for example.ptl_base_tmpdir_checkset towarnorsilentlets the server run there.Rendezvous files moved. Each user’s servers write theirs in
<tmpdir>/pmix-user.<uid>/. A system server, scheduler, or system controller running as root writes its files in/run/pmix, unless the host names a system temporary directory.Cleaning up. A rendezvous file whose server is gone is reclaimed automatically. Session and socket directories still accumulate after a crash; clean out old
pmix*entries as before.
See the “Temporary files” section of Controlling Authentication.
3.5.2.6. Scripts that read a server’s address
--report-uri -and+(PMIX_TCP_REPORT_URI) now print the local socket’s address, which v7 tools use.tcp-andtcp+print the TCP address instead, for a tool of an earlier release. They need legacy clients on.A v7 tool given a TCP address for a v7 server is refused with
PMIX_ERR_LOCAL_URI_REQUIRED, and told to use the local address.
3.5.2.7. Settings that now fail
What changed. PMIx no longer ignores a setting it cannot honor.
Naming a component that no longer exists, a user in
ptl_base_server_access the system does not know, or a relative
socket directory makes the process fail to start, with a message.
What to do. Check parameter files (pmix-mca-params.conf) and
PMIX_MCA_* settings in module files and job scripts against this
list.
v5.0 component |
In v7 |
|---|---|
|
Replaced by |
|
Renamed |
|
Replaced by |
|
Removed. |
|
Removed. |
|
Removed. |
|
Removed (they served releases v7 no longer talks to). |
the |
Removed. |
3.5.2.8. New and changed parameters
Parameter |
Default |
What it does |
|---|---|---|
|
|
Take clients and tools of releases before v7 (needs
|
|
|
With no server named, also look for servers of releases before
v7 where they write their files (needs
|
|
|
Both of the above, for whichever directions the build has. Unlike either of them, it makes nothing fail to start in a build that lacks a direction: it asks for nothing the build does not have. |
|
none |
Servers besides one’s own user’s and root’s that a client or tool will talk to. |
|
|
What a server does with a temporary directory another user could
change: |
|
5 |
Seconds a new connection has to send its whole request. |
|
0 |
Connections still sending their request, and, separately, tools waiting for the host’s decision. 0 means an eighth of the file-descriptor limit; -1 no limit. A user other than the server’s, root’s, and the host’s own may have at most a quarter of the waiting tools. |
|
16 MiB |
Total buffer allocated to connection requests still arriving. |
|
32 |
Largest message accepted, in megabytes. 0 means no limit. |
|
512 MiB |
Largest decompressed size accepted. |
|
1024 |
Distinct log messages remembered for duplicate suppression. |
See Settings and What They Enable for how these combine with build
options and host attributes, Controlling Authentication for
the full list, and
pmix_info --param all all for every parameter and its value.