2. Settings and What They Enable
Most of what a PMIx server does is switched on in more than one place, and each place belongs to a different person. This page puts them side by side, so that whoever builds PMIx, whoever runs it, and whoever writes the program that hosts it can see what the others must also do.
Layer |
Set by |
What it decides |
|---|---|---|
|
whoever builds PMIx |
What is in the library at all. |
MCA parameters |
the system administrator, in a parameter file or the environment (see The Modular Component Architecture (MCA)) |
Site policy, read by each process as it initializes. |
Attributes passed to PMIx_server_init(3) - or to PMIx_tool_init(3) by a launcher or scheduler |
the developer of the host environment (PRRTE, a resource manager, a scheduler), in its code |
What this server is for: whether it serves tools, which role it plays, who may reach its socket. |
Four rules hold throughout:
A behavior happens only when every layer it needs says yes. A run-time setting cannot add what the build left out, and a host cannot use what the site turned off.
Asking for something the build left out is not always ignored. Some requests make the process fail to initialize; others quietly do nothing. The tables below say which.
Where an attribute and an MCA parameter control the same thing, the attribute wins. A host that passes an attribute overrides the site; a host that passes nothing leaves the decision to the site.
The settings that count are those of the process that reads them. Server policy belongs in the parameter files the host’s daemons read. Two settings are read by clients and tools instead -
ptl_base_server_accessandptl_base_legacy_servers- and belong where those processes read them: the users’ parameter files or environment.
pmix_info shows what an installation was built with and the value
of every parameter:
shell$ pmix_info # build options and components
shell$ pmix_info --param ptl all # connection parameters and values
2.1. Common surprises
Each line is a combination that does not do what it looks as though it should. The sections after this one give the full tables.
If… |
then… |
To get the behavior |
|---|---|---|
PMIx is configured |
the server has no TCP listener, and its environment and rendezvous files name only its local socket. A client of a release before v7.0 cannot use that address and runs as a singleton; a tool of such a release cannot connect. |
Set |
PMIx is configured without |
every server, and every launcher or scheduler ( |
Rebuild with |
PMIx is configured without |
every client and tool ( |
Rebuild with |
The site sets |
legacy clients are off. |
The host’s attribute wins; a host should pass it only when it means to override the site. |
|
the server still takes no older clients. The parameter only widens where a process told of no server searches for one. |
To take older clients, see the first row. |
The host does not pass |
the server writes none of the files tools find it by, and refuses
every tool with |
The host passes |
The host passes |
no system or session file is written, and tools are refused. |
Pass |
The host passes |
the file is written, and a debugger that reads it is refused. |
Pass |
The host passes |
the server fails to initialize, whether or not it supports tools. |
Pass one of the two IDs. |
The host passes |
tools of other users can read the rendezvous files, but the kernel refuses their connection to the socket (mode 0600). |
Widen the socket too: |
The host widens the socket ( |
tools of other users reach the socket and are refused by PMIx -
except root’s and those of the user named as |
Pass |
The server runs as a service account, not as root or as the users |
users’ tools will not talk to it: a tool talks only to a server running as its own user or root, or one that started it. |
Name the account in |
Something other than the server’s own directory is at
|
the server reports it and runs on, writing its tool files in
|
Remove what is at the name. |
A directory on the path to the server’s temporary directory can
be changed by another user (a Kubernetes |
the server fails to initialize. |
Use another directory, or set |
The temporary directory’s path is long (macOS’s default
|
the server fails to initialize: the socket’s path is limited to about 100 characters. |
Start the host with a short |
The host passes |
the server fails to initialize: it has no TCP address to report. |
Turn legacy clients on, or report |
The host passes |
the server fails to initialize. A server takes connections from its own host only. |
Remove the attribute. |
PMIx is configured without |
its |
Configure |
MUNGE is installed, but PMIx is not configured |
the |
Configure |
Nodes of one job are built with different compression libraries |
a node whose component is |
Build every node the same way. |
2.2. Processes of releases before v7.0
Processes of this release reach a server on their host over its local socket. Processes of earlier releases reach a server over TCP, and the two directions - a server taking older clients, and a client reaching an older server - are built and controlled separately. See clients of earlier releases.
2.2.1. A server taking clients and tools of earlier releases
Built with |
At run time |
The server |
|---|---|---|
default (no |
nothing set |
Has no TCP listener. Clients of earlier releases run as singletons; tools of earlier releases cannot connect. |
default |
|
Fails to initialize. So does a launcher or scheduler tool that reads the setting. |
default |
|
As in the first row. The parameter turns on only the directions the build has, and this build has neither. |
|
nothing set; or the host passes |
As in the first row: the support is in the library and nothing turns it on. |
|
|
Listens on TCP, on a loopback interface only, besides its local
socket. The address processes of earlier releases read - in the
environment of the processes it starts, first in its rendezvous
files, and from |
|
the host passes |
As in the row above, whatever the parameters say. |
While a server takes legacy clients, a connection over TCP that
declares a release before v7.0 is not checked for a connection token.
On a platform whose kernel keeps no table of TCP connections (any but
Linux, macOS and FreeBSD 12 or later), native takes such a peer’s
user and group as it gives them, as every earlier release did. See
Accepting Connections.
2.2.2. A client or tool reaching a server of an earlier release
Built with |
At run time |
A client or tool of this release |
|---|---|---|
default (no |
nothing set, or |
Refuses a server of an earlier release, and any server address
over TCP, naming the reason. A client’s |
default (no |
|
|
|
nothing set |
Connects to a server of v3.2 through v6.x that started it or that
it is told of, over TCP. Refuses a server older than v3.2
( |
|
|
Also, when told of no server, searches the whole system temporary directory, where earlier releases write their rendezvous files. |
With neither --enable-legacy-clients nor --enable-legacy-servers
- the default - the library has no TCP transport, and no wire format
but this release’s.
2.3. Tools, and the files they find a server by
A client is told where its server is, in the environment the server
gave it. A tool - a debugger, prun, a monitor, a runtime attaching
to a scheduler - finds a server through a rendezvous file. Which
files a server writes, and whether it accepts the tool that reads one,
depends only on the attributes its host passes.
The host passes |
Files written |
Tools |
|---|---|---|
none of the attributes below |
none |
All refused, |
|
|
Accepted from the server’s user, root, and the user named as
|
|
also |
As above. This is the server a tool asking for the system server finds. |
|
none |
All refused. |
|
also |
As above. A second server given the same ID on the node fails to initialize. |
|
none |
All refused. |
|
- |
The server fails to initialize. |
|
|
Accepted as for |
|
|
As for |
|
the file named, at the path given |
Refused unless |
|
the same files, readable by all users; the user directory, if the server makes it, mode 0711 |
Also accepted from other users - who must be able to reach the socket as well (next section). |
Where the files go. The user directory is
<system tmpdir>/pmix-user.<uid>; when something other than the
server’s own directory is at that name, $XDG_RUNTIME_DIR/pmix, if
that is a directory of the server’s user that nobody else can write;
else the files are not written and the server runs on. The system
directory is PMIX_SYSTEM_TMPDIR when the host names one, else
/run/pmix (/var/run/pmix) for a server running as root, else
the server’s user directory.
What a tool believes. A tool reads a file only if its owner is the
tool’s own user, root, or a user ptl_base_server_access names, and
talks only to a server that the kernel says runs as that user. A server
whose files tools of another user are to find runs as root, or as that
user, or is named in those users’ ptl_base_server_access.
A server whose namespace, system role or session ID is already held by a running server on the node fails to initialize, rather than take over that server’s file. A file whose server is gone is removed and replaced.
The rest is in PMIx_server_init(3) (“Rendezvous files”) and Transport Layer (Client/Server/Tool Connections).
2.4. Who can reach the socket
A process of another user needs two things to connect: the kernel must let it open the server’s local socket, and PMIx must accept its user. The host controls both, separately.
The process’s user |
The socket must admit it |
PMIx accepts a tool of that user (given
|
|---|---|---|
the server’s own |
by default (mode 0600) |
by default |
root |
always |
by default |
the user named as |
by default: |
by default |
a member of the group named as |
with |
only with |
anyone else |
with |
only with |
The same socket rule applies to a client the host starts under another
user: a server running as root for a job’s user names that user as
PMIX_USERID. A client is always checked against the user and group
the host registered it with; no setting changes that.
A tool also checks the server, from its own side: see What a tool believes, above. And the host has the last word on every tool PMIx accepts - see Accepting Connections.
2.5. Authentication
Setting |
Effect |
|---|---|
default build |
|
|
Builds |
|
Leaves |
|
Restricts the components a server loads. A server left with none fails to initialize. |
A process can use only the methods its server offers, so the server’s settings decide. See Controlling Authentication.
2.6. Other build options that change behavior
Option |
Effect |
|---|---|
|
The component used is the first of |
(no option) the shared-memory datastore, |
Built on 64-bit platforms other than macOS, and used at run time
where |
|
|
|
Each component is a separate plugin, which an administrator can remove to make it unavailable. |
|
Compiles every optional component against stand-in headers. The result does not work; never install it. |
2.7. What each person must do
Whoever builds PMIx
Decide on clients of earlier releases. If containers or applications with an older PMIx inside run on the system, configure
--enable-legacy-clients; it does nothing until turned on at run time.Decide on servers of earlier releases. If applications built against this PMIx - in a container image, say - may run under a resource manager of an earlier release, configure
--enable-legacy-servers; without it theirPMIx_Initfails there.Configure
--with-mungeif the site authenticates with MUNGE.Build every node with the same compression libraries.
The system administrator
Turn legacy clients on with
ptl_base_legacy_clients = truein the parameter file the host’s daemons read - PRRTE does not passPMIX_SERVER_LEGACY_PEERS, so for PRRTE the parameter is the only way. Do not set it, orptl_base_legacy_peers, in a build without--enable-legacy-clients.Keep the temporary directory short, and on a path only root and the server’s user can change.
Where a DVM or scheduler runs as a service account, name it in the users’
ptl_base_server_access.
The host environment’s developer
Pass
PMIX_SERVER_TOOL_SUPPORTif any tool is to connect - including from a scheduler, a system server, a session server, and a launcher, which passes it toPMIx_tool_init.To take tools of other users, pass
PMIX_SERVER_ALLOW_FOREIGN_TOOLSand widen the socket.Give a session server its ID.
Leave
PMIX_SERVER_LEGACY_PEERSunset unless the host means to override the site, and expose the choice to the site if it does.
What each of these attributes does is in PMIx_server_init(3). What the host is expected to do once it is running is in Host Environments, and Starting the Server sets out, goal by goal, what a host passes to get each behavior.