Index of /dist/mxm-conduit

[ICO]NameLast modifiedSize

[PARENTDIR]Parent Directory  -
[DIR]contrib/2018-07-19 13:23 -
[TXT]gasnet_core.c2018-07-19 12:45 98K
[TXT]gasnet_extended.c2018-07-19 12:45 37K
[TXT]gasnet_core_sndrcv.c2018-07-19 12:45 35K
[   ]Makefile.in2018-07-19 13:22 31K
[TXT]gasnet_core_internal.h2018-07-19 12:45 9.2K
[TXT]README2018-07-19 12:45 8.4K
[TXT]gasnet_core.h2018-07-19 12:45 6.1K
[   ]Makefile.am2018-07-19 12:45 4.7K
[TXT]gasnet_core_connect.c2018-07-19 12:45 4.3K
[TXT]gasnet_core_fwd.h2018-07-19 12:45 2.7K
[TXT]license.txt2018-07-19 12:45 2.0K
[TXT]gasnet_extended_fwd.h2018-07-19 12:45 2.0K
[   ]conduit.mak.in2018-07-19 12:45 1.8K
[TXT]gasnet_mxm_req.h2018-07-19 12:45 1.5K
[TXT]gasnet_core_help.h2018-07-19 12:45 489

GASNet mxm-conduit documentation


The mxm-conduit is now DEPRECATED.
No further development of this conduit is anticipated.
It may be removed in a future release.

Users with InfiniBand hardware are recommended to use ibv-conduit
by passing the following to GASNet's configure script.
  --enable-ibv --disable-mxm


User Information:

Mxm-conduit implements GASNet over the Mellanox Messaging Accelerator (MXM)
API for recent Mellanox InfiniBand Host Channel Adapters (HCAs).

See for general information on MXM,
including supported hardware and OS releases.

Where this conduit runs:

While nearly all InfiniBand HCAs can run ibv-conduit, mxm-conduit is able to
take fuller advantage of the underlying infrastructure in Mellanox's switches
and HCAs.  One significant difference is improved scalability to large node
counts, due to MXM's support for Dynamic Connection (DC) transport mode.

Mxm-conduit is expected to work with all releases of MXM.

The relative performance of mxm and ibv conduits can depend on many factors.
For this reason we cannot categorically recommend one over the other.  We
encourage users concerned with obtaining the best possible performance to
benchmark your own workload when selecting a conduit.

Optional compile-time settings:

The default spawner to be used by the gasnetrun_mxm utility can be
selected by configuring '--with-mxm-spawner=VALUE', where VALUE is one
of 'mpi', 'pmi' or 'ssh'.  If this option is not used, mpi is the
default when available, and ssh otherwise.
Here are some things to consider when selecting a default spawner:
  + mpi-spawner is the default when MPI is available precisely because it
    is so frequently present on systems where GASNet is to be installed.
    Additionally, very little (if any) configuration is required and the
    behavior is highly reliable.
  + pmi-spawner uses the same "Process Management Interface" which forms
    the basis for many mpirun implementations.  When support is available,
    this spawner can be as easy to use and as reliable as mpi-spawner, but
    without the overheads of initializing an MPI runtime.
  + ssh-spawner depends only on the availability of a remote shell command
    such as ssh.  For this reason ssh-spawner support is always compiled.
    However, it can be difficult (or impossible) to use on a cluster which
    was not setup to allow ssh to (and among) its compute nodes.
For more information on configuration and use of these spawners, see
   README-{ssh,mpi,pmi}-spawner (installed)
   other/{ssh,mpi,pmi}-spawner/README (source).

Job Spawning:

If using UPC, Titanium, etc. the language-specific commands should be used
to launch applications.  Otherwise, applications can be launched using
the gasnetrun_mxm utility:
+ usage summary:
    gasnetrun_mxm -n <n> [options] [--] prog [program args]
      -n <n>                 number of processes to run (required)
      -N <N>                 number of nodes to run on (not supported by all MPIs)
      -E <VAR1[,VAR2...]>    list of environment vars to propagate
      -v                     be verbose about what is happening
      -t                     test only, don't execute anything (implies -v)
      -k                     keep any temporary files created (implies -v)
      -spawner=(ssh|mpi|pmi) force use of a specific spawner (if available)

There are as many as three possible methods (ssh, mpi and pmi) by which one
can launch an mxm-conduit application.  Ssh-based spawning is always
available, and mpi- and pmi-based spawning are available if the respective
support was located at configure time.  The default is established at
configure time (see section "Optional compile-time settings").

To select a non-default spawner one may either use the "-spawner=" command-
line argument or set the environment variable GASNET_MXM_SPAWNER to "ssh",
"mpi" or "pmi".  If both are used, then the command line argument takes

Recognized environment variables:

* All the standard GASNet environment variables (see top-level README)

* GASNET_PHYSMEM_MAX (default is set at configure time, see below)
  If set non-zero this parameter tells mxm-conduit the maximum amount of
  memory to pin for the segment in a (default) GASNET_SEGMENT_FAST build.  The
  value may specify either a relative or absolute size.  If the value parses
  as a floating-point value less than 1.0 (including fractions such as "5/8"),
  then this is taken as a fraction of the (estimated) physical memory.
  Otherwise the value is taken as an absolute memory size, with "M", "G" and
  "T" suffixes accepted to indicate units of Megabytes, Gigabytes, and
  Terabytes, respectively.
  For backwards compatibility, "0" means 2/3 of estimated physical memory.
  The default may be set at configure time using --with-mxm-physmem-max=VALUE,
  and otherwise is "2/3" (pin up to 2/3 of the estimated physical memory).

  This is a per-node value - if there are multiple processes per shared memory
  node the value given by this variable will be divided by the number of
  processes per shared memory node to determine the maximum segment size.

* GASNET_PHYSMEM_PROBE (default is set at configure time, see below)
  This gives a boolean: "0" to disable or "1" to force a probe to verify the
  value given by GASNET_PHYSMEM_MAX.  Enabling this setting may greatly slow
  startup, but can avoid unexpected runtime failures if GASNET_PHYSMEM_MAX
  exceeds the limits imposed by the O/S and HCA.
  The default is OFF (probe disabled).  However, this can be changed to a
  default of ON by configuring using --enable-mxm-physmem-probe.

  This gives the maximum number of MXM-level operations which can be in-flight
  at any given time due to GASNet NBI Puts and Gets.

* GASNET_MXM_SPAWNER (default set at configure time)
  To override the default spawner for mxm-conduit jobs, one may set this
  environment variable as described in the section "Job Spawning", above.
  There are additional settings which control behaviors of the various
  spawners, as described in the respective READMEs (listed in section
  "Optional compile-time settings", above).

* MXM environment variables documented in the Release Notes which accompany
  your release of MXM.

Known problems:

* See the GASNet Bugzilla server for details on known bugs:

Future work:


Design Overview:

Header Files

        Definitions of basic constants and types used by the core API and
        included early in the gasnet.h file (gasnet.h provides some default
        values for various constants and types that can be overridden here).

        Provides the public interface for the GASNet core API (function
        prototypes and/or inline functions for implementing the core API).

        This is the core's entry point to provide access to private symbols
        which might be required by the core header (e.g.: node's node id,
        number of nodes in job), but which should not really be publicly
        visible to client code (i.e. not part of GASNet API).

        Provides the public interface for the GASNet extended API

        This file is NOT included by gasnet.h (and therefore not visible to
        client code) and are exclusively for use in building the conduit
        <gasnet_internal.h> is the conduit-independent version of this file
        at the top level.
        Internal GASNet implementation should include <gasnet_internal.h>
        instead of <gasnet.h>.

<< TBD >>