10.3.2. What a Host Must Do
This page lists what every host environment must do to support PMIx, in the order a host meets each item. The pages that follow cover security (Accepting Connections, Deciding What a Requester May Do) and the division of work for each operation (Operation by Operation).
10.3.2.1. Building and running
Build the host against the PMIx library it will run against. This is the one exception to PMIx’s forward compatibility.
PMIx_server_initreceives only a pointer to yourpmix_server_module_t. The module keeps every function pointer it has ever had, but it grows as up-calls are added, and the library cannot tell from a pointer how large your module is or which version it came from. It assumes your module matches its own headers, and that your up-calls behave as its version specifies. So when PMIx is upgraded, rebuild the host. New members are added to the module only in a new major release, so a host that runs against a library of a different major version is not supported.Applications, libraries, and tools need no rebuild. PMIx is forward compatible: every public API, macro, structure, and attribute is kept forever. A program built against an earlier PMIx builds and runs against a later one without change.
Clients and tools need not match the server’s version at run time either: a server accepts clients and tools from v2.1 on. The library handles that compatibility; the host does nothing for it. See Version Numbers and Binary Compatibility.
Check for features with capability flags, not version numbers. Each
PMIX_CAP_*flag inpmix_version.hsays a feature is present. For example,PMIX_CAP_REQUESTER_IDsays up-calls carry the requester’s identity, andPMIX_CAP_DMODEX_REQUEST2says PMIx_server_dmodex_request2(3) exists.
10.3.2.2. Starting the server
Call PMIx_server_init(3) with your module and the directives that describe your server. The directives decide, among other things:
whether tools may connect, and how they find the server (
PMIX_SERVER_TOOL_SUPPORT,PMIX_SERVER_SYSTEM_SUPPORT,PMIX_SERVER_SESSION_SUPPORT);whether tools running as other users may connect (
PMIX_SERVER_ALLOW_FOREIGN_TOOLS);where the server’s rendezvous files and session directories go (
PMIX_SERVER_TMPDIR,PMIX_SYSTEM_TMPDIR).
These are the host’s decisions. A system administrator cannot change
them through MCA parameters (see
Controlling Authentication). Whether the server takes clients
and tools of earlier releases is the host’s when it says, with
PMIX_SERVER_LEGACY_PEERS, and the ptl_base_legacy_clients (or
ptl_base_legacy_peers) parameter’s when it does not. A host that wants tools
of other users to connect must also give the server a directory those
users can search; see PMIx_server_init(3)
and Session Directories.
Important
The server decides which authentication methods PMIx uses.
PMIx’s checks on a connection are a triage, and the first of them is
the credential, checked by a security (psec) component. A server
accepts only the components it has loaded, and passes that list to
the clients it starts (through
PMIx_server_setup_fork(3)) and
to tools (in its rendezvous file). A process cannot use a method its
server does not offer, so the server’s settings are the ones that
count.
The components a server loads are chosen when it starts, by the
psec MCA parameter of the PMIx installation it runs from:
The host sets it before calling
PMIx_server_init- for example,PMIX_MCA_psec=mungein its environment - when its deployment calls for particular methods.A system administrator sets it in that installation’s
<prefix>/etc/pmix-mca-params-override.conf, whose settings take precedence over the environment’s - for every process that reads the file. A process whose environment points its library at other parameter files, or at none, does not read it (see Controlling Authentication), so the file sets the policy for servers the administrator starts, or whose environment the administrator controls. A component the installation is built without, or whose plugin is removed, is unavailable to every process of that installation.A server started by a client - a nested server - inherits
PMIX_SECURITY_MODEfrom the environment its own server gave the client, and takes it as itspsecsetting unlessPMIX_MCA_psecis set. It offers its own peers no more than its parent server offered. A server started by a process that has itself calledPMIx_InitorPMIx_tool_initinherits noPMIX_SECURITY_MODE- that init took it out of the environment - and is not narrowed.
Each PMIx installation reads only the parameter files under its own prefix, so the settings belong in the installation the host’s server runs from. See Controlling Authentication.
10.3.2.3. Accepting connections
Every client and tool connection that passes PMIx’s checks reaches the
host through client_connected2 or tool_connected2. The full
rules are on Accepting Connections; in brief:
Answer every one, once, within
PMIX_TIMEOUT. The up-call carries the number of seconds the host is asked to answer within. A host that cannot decide in time should answer no; the server refuses the process itself a little after that time.You decide on one tool of a name at a time. A slow answer delays the tools of that name waiting behind it, and their time waiting counts against their own wait.
Do not register the namespace of a tool that named itself while you decide on it: the tool is then refused with
PMIX_ERR_EXISTS, as one naming a job of yours. It needs nothing from you - the tool library sets its job values itself.Be ready to name a tool you launched. A tool you started, whose namespace and rank are already in use - several tools started by one job script, say - connects again as a tool that needs an identity, and you must give it a namespace of its own.
Handle
PMIX_ERR_LOST_CONNECTIONfor processes you accepted. A process you said yes to may never become connected - your answer came too late, or it went away first. You are then sent this event naming it, as for any process that leaves without finalizing. Register a handler for it.Expect a process that execs to be lost. A client or tool of this release closes its connection when it execs: you are sent
PMIX_ERR_LOST_CONNECTIONfor it at once, whether the process runs on or not.
Do not call PMIx_server_finalize(3)
after a failed PMIx_server_init.
10.3.2.4. Writing the up-calls
The rules below apply to every function in pmix_server_module_t(5).
Leave out what you do not support. A
NULLentry makes the library returnPMIX_ERR_NOT_SUPPORTEDto the requester. For some operations the library then does the work itself; see Operation by Operation.Return quickly. Up-calls run on the PMIx progress thread. Start the work, return, and report the result later through the callback you were given.
Return the right code.
PMIX_SUCCESS: you accepted the request and will call the callback.PMIX_OPERATION_SUCCEEDED: the work is already done and you will not call the callback. Not allowed fromspawn,lookup,direct_modex,get_credentialorvalidate_credential: their results can only travel through the callback.Any error: the request failed and you will not call the callback.
Do not make a blocking PMIx call from an up-call or from any other code running on the progress thread. It waits for itself and returns
PMIX_ERR_WOULD_BLOCK. Use the non-blocking form, or call from a thread of your own.Do not free what the library passes you, except the
datablob offence_nb, which becomes yours and which you mustfree(). What you pass back through a callback stays yours; you may release it once the callback returns, or through the release function the callback takes. See pmix_callback_functions(5).Use the requester’s identity. Every up-call made for a client or tool carries the requester’s
PMIX_USERIDandPMIX_GRPID. Use them to decide whether to perform the request; see Deciding What a Requester May Do.
10.3.2.5. Before launching a process
Do these, in this order, before you start any process that will call
PMIx_Init:
Register the job with PMIx_server_register_nspace(3).
Register every job that takes part in a collective with a local process, including jobs that have no process on this node.
Give the number of local processes. The library uses it to know when the local part of a collective is complete.
Give the job-level information applications will ask for. PMIx serves it to them without asking you again. This is what makes “instant on” startup possible.
Give the job’s owner (
PMIX_USERID,PMIX_GRPID) and the users and groups the owner allowed (PMIX_ACCESS_PERMISSIONS). Do this on every node that hosts part of the job. See Deciding What a Requester May Do.
Register each local client with PMIx_server_register_client(3), exactly once per rank. The uid and gid you give are the identity the client must prove when it connects. A client of a job you have not registered, or of a rank you have not registered, is refused.
Set up the client’s environment with PMIx_server_setup_fork(3). This is required. It gives the client what it needs to find and connect to the server.
If you used application setup, deliver its result on every node with PMIx_server_setup_local_support(3). The data comes from PMIx_server_setup_application(3), which a launcher calls once for the job.
Do not use PMIx_Store_internal to change a job’s data. To update a
registered job, register it again with a negative number of local
processes; see
PMIx_server_register_nspace(3).
10.3.2.6. While processes run
Report events about your jobs. You must report events concerning a registered job - a process or job terminating, for example - whether or not anything has registered for them.
register_eventsonly tells you which system events to forward as well.Report a connected process that dies. When a process that called PMIx_Connect(3) terminates without disconnecting, generate
PMIX_ERR_PROC_TERM_WO_SYNCfor the processes connected to it.Deliver events where they are needed. An event aimed at particular processes must reach every server where they run. Name the affected processes with
PMIX_EVENT_AFFECTED_PROCorPMIX_EVENT_AFFECTED_PROCS. See PMIx_Notify_event(3).Forward output. Deliver output to the servers that asked for it with PMIx_server_IOF_deliver(3).
Fetch data from other nodes. Answer
direct_modexup-calls by asking the server that holds the data, using PMIx_server_dmodex_request2(3) so that the requester’s identity goes with the request.Keep process sets and groups consistent. If you define process sets, every server must agree on their membership, and their names must not collide. See PMIx_server_define_process_set(3).
10.3.2.7. Shutting down
Finish each client. The
client_finalizedup-call tells you a client calledPMIx_Finalize. Release your state for it, then call the callback to let the client go. Do not deregister it there: the process may callPMIx_Initagain.Optionally, deregister each client with PMIx_server_deregister_client(3) once its process has terminated. A client that finalized is still a member of its job, and collectives that name it keep waiting for it, until you do. PRRTE deregisters each local process when it has terminated.
Deregister each job with PMIx_server_deregister_nspace(3) once nothing on this node needs it. That removes every client still registered in it.
Deregister each user you registered with PMIx_server_deregister_resources(3) once nothing you run or serve belongs to that user.
Finalize the server with PMIx_server_finalize(3), and not from within an up-call or callback.
For the full life cycle of a client, including clients that initialize and finalize more than once, see Client Init/Finalize Lifecycle and State Management.