GASNet Extended API Notes for Conduit Writers

The GASNet Extended API is a means of leveraging high-level (and eventually collective) operations from specialized network hardware. The following document presents an overview of the requirements GASNet puts and gets impose on networks and attempts to dissambiguate the many flavours of puts/gets currently in the interface. The remainder of this document assumes familiarity with the following:

  1. GASNet spec (specifically sections on the GASNet segment, GASNet puts and of course familiarity with the GASNet core)
  2. Network-specific API documentation (LAPI, GM, Elan, SCI, ...)
  3. Pinning networks should consider using the firehose interface (link to be posted...)

The most important aspect of GASNet puts and gets, which unfortunately uncovers restrictions on many of the network hardware available today, concerns the ability for the network to complete one-sided operations without requiring programmers to set up memory available for RDMA at runtime. For this reason, the following distinction in networks is necessary:

  1. Hardware assisted RDMA. Puts and gets for any reasonable size can be completed without interrupting the remote host processor. This approach requires assistance from the NIC to handle page faults and replicate some of the data structures present in the host's Memory Management Unit. These networks are typically more expensive but are preferred for Global Address Space languages.
  2. Explicit Pinning RDMA. The network-level programmer must explicitly set up virtual memory regions that are pinned (locked from being paged out). For other programming models, pinning small portions of memory may be a sufficient approach. Since GAS languages typically read and write to remote memory regions that may be larger than the amount of physical memory, explicit pinning must be handled carefully -- mainly because of the limitations imposed on pinning all of physical memory but also because of the implications in interrupting remote host processors.

The ideal GASNet put/get operation should exhibit the following characteristics:

Explicit Pinning Idiosyncrasies

Since one-sided operation over pinning networks cannot be possibly guaranteed for every put/get, the alternatives for sending puts and gets to a remote node are the following:
  1. Rendezvous: A request to pin a remote region is sent to the target node. Once the initiator receives the reply confirming that the remote region is pinned, the one-sided operation can be initiated.
  2. Active Messages: The GASNet core provides AM messaging capability which is sufficiently general for puts/gets although discouraged for efficiency.
  3. Firehose: Firehose clients have the ability to expose onesided, zero-copy communication as a common case, while minimizing the number of host-level synchronizations (similar to the rendez-vous approach). Additionally, the cost of synchronization and pinning is amortized over multiple remote memory operations.

If the network supports more than one send mechanism, it is always useful to know what performance can be expected from each mechanism (possibly as a function of message size).

GASNet Puts/Gets

Puts non-bulk

Gets Non-bulk

Puts/Gets bulk

Memory copying

  1. 1 to word size: If the memory is aligned, a store may be sufficient. If not, see below.
  2. word size - ...: For larger sizes, various approaches may be possible.

Back to the GASNet home page