GASNet smp-conduit documentation Dan Bonachea User Information: ----------------- smp-conduit provides a single-node loopback GASNet implementation for smp or uniprocessor systems lacking network hardware, or where all the cores to be used coexist within a hardware-shared memory. Where this conduit runs: ----------------------- smp-conduit is extremely portable and should run anywhere with a POSIX-like environment and hardware-shared memory. smp-conduit jobs can be launched directly in the same manner as normal POSIX executables - ie on the command line, no spawner script is required. However, in order to take advantage of the facilities in modern batch systems (such as control over CPU, GPU and memory affinity), the provided `gasnetrun_smp` script may also be used, as well as launchers like `mpirun` or SLURM's `srun` Optional configure options: -------------------------- * --with-smp-spawner=... controls the default value of the environment variable `GASNET_SMP_SPAWNER`, described under "Recognized environment variables". See the top-level README for information regarding process-shared memory (PSHM) support and the associated family of `--(en|dis)able-pshm*` configure options. Without PSHM enabled, smp-conduit provides only a single GASNet process. With PSHM, multiple processes can be used on a single host; these will communicate through shared memory. Job spawning: ------------- If using UPC++, Chapel, etc. the language-specific commands should be used to launch applications. Otherwise, applications can be launched using the gasnetrun_smp utility: + usage summary: gasnetrun_smp -n [options] [--] prog [program args] options: -n number of processes to run -c number of cpus per process (not always supported) -E list of environment vars to propagate -v be verbose about what is happening -t test only, don't execute anything (implies -v) -k keep any temporary files created (implies -v) -spawner= force use of a specific spawner ([supported spawners]) -- ends option parsing There are as many as four possible methods (fork, ssh, mpi and pmi) by which `gasnetrun_smp` can launch an smp-conduit application. For more information, see the description of the environment variable `GASNET_SMP_SPAWNER`, below. That variable's documentation is also applicable to launch via the same MPI or PMI launch utilities which `gasnetrun_smp` uses. As an alternative to using `gasnetrun_smp`, it is also possible to launch an smp-conduit application directly on the command line without any utilities. For information on controlling the number of processes in an smp-conduit job launched in this manner, see the description of the environment variable `GASNET_PSHM_NODES`, below. Optional compile-time settings: ------------------------------ * All the compile-time settings from extended-ref (see the extended-ref README) Recognized environment variables: --------------------------------- * GASNET_SMP_SPAWNER This environment variable determines what job spawner mechanism the application is expected to interact with. When using the `gasnetrun_smp` utility (described later in this document) this also controls the default job spawner to be used. However, when *not* using `gasnetrun_smp` is it necessary to ensure this variable is set to match the spawner in use. Supported values (case insensitive) include: + "fork" - this is the default unless another has been specified at configure time, and selects a built-in job launch mechanism. See also "GASNET_PSHM_NODES", below. + "mpi" - this selects job launch via `mpirun`, `mpiexec`, or similar as appropriate to the MPI implementation (if any) detected at configure time. This option is only available if a working MPI implementation was detected by `configure`. See also the mpi-spawner README. + "pmi" - this selects job launch using the PMI family of protocols, via such utilities as `srun` and some `mpirun` implementations. This option is only available if a PMI client library was detected by `configure`. See also the pmi-spawner README. + "ssh" - this selects job launch using GASNet-specific logic provided by the `gasnetrun_smp` utility. Relative to using `ssh directly` (such as via `ssh [host] env GASNET_PSHM_NODES=[n] ./a.out`) this option can automatically identify an allocated `[host]` with many batch systems, and can (via `-E`) provide some control over environment propagation. See also the ssh-spawner README. Note that when using `gasnetrun_smp`, the optional `-spawner=...` command line option has precedence over this envirnoment variable. * GASNET_PSHM_NODES When process-shared memory (PSHM) support is enabled at configure time and the application is launched without use of a job spawner (e.g. `gasnetrun_smp`, `mpirun`, `srun`, etc.), this variable controls the number of GASNet smp-conduit processes to fork at job launch. These processes will communicate through shared memory using PSHM. For more details PSHM support, see the top-level README. If PSHM is enabled but an explicit job spawner is used (see GASNET_SMP_SPAWNER above), this variable is ignored and the spawner-specific command line controls the number of GASNet smp-conduit processes. If PSHM is disabled, this variable is ignored and smp-conduit jobs consist of only a single process (but the client may still utilize thread-level parallelism within that process, in GASNET_PAR mode). * GASNET_BARRIER=PSHM (default when PSHM support is enabled) Enables shared-memory implementation of GASNet barriers * All the standard GASNet environment variables (see top-level README) Known problems: --------------- * Bug 3480 It has been observed on Cray systems running CLE6 that following a call to munmap(), there may be a delay before the physical page frames are available for reallocation. At startup GASNet normally tries to determine the largest available mmap() region, which is then unmapped, leading to the possibility that subsequent allocation of a large GASNet segment may fail if timing is "just right". The configure option --enable-bug3480-workaround will enable retries and an additional barrier to prevent the failure mode described above, at the cost of a small additional delay at startup. In this release, the provided cross-configure-cray-xc-* scripts AUTOMATICALLY enable the work-around if CLE6 is detected. If you do *not* desire the work-around, then you may pass the option --disable-bug3480-workaround to the cross-configure script. The most up-to-date information on this bug is maintained at: https://gasnet-bugs.lbl.gov/bugzilla/show_bug.cgi?id=3480 It is currently unknown if this problem is specific to Cray XC systems. * Bug 3727 The smp-conduit support for remote atomics is limited to certain datatypes. In general, support encompasses all datatypes on LP64 platforms but is limited to 32-bit datatypes on ILP32 platforms. However, there are exceptions on both types of platform, many of which are compiler dependent. The most up-to-date information on this bug is maintained at: https://gasnet-bugs.lbl.gov/bugzilla/show_bug.cgi?id=3727 That bug report describes the problem in more detail. * See the GASNet Bugzilla server for details on other known bugs: https://gasnet-bugs.lbl.gov/ Future work: ------------ =============================================================================== Design Overview: ---------------- All puts/gets are local accesses (with the locality check removed at compile time). When PSHM is not in use all AM handlers execute synchronously on the calling thread, and segment alignment is trivially guaranteed.