IBM Spectrum MPI

Version 10.2.0

Release Notes

IBM

IBM Spectrum MPI

Version 10.2.0

Release Notes

IBM

NoteBefore using this information and the product it supports, read the information in “Notices” on page 17.

This edition applies to version 10, release 2, modification 0 of IBM Spectrum MPI (product number 5725-G83) andto all subsequent releases and modifications until otherwise indicated in new editions.

© Copyright IBM Corporation 2018.US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contractwith IBM Corp.

Contents

About this information . . . . . . . . vConventions and terminology used in thisinformation. . . . . . . . . . . . . . . vPrerequisite and related information . . . . . . vi

Overview . . . . . . . . . . . . . . 1

Features . . . . . . . . . . . . . . 3

Limitations and restrictions . . . . . . 7GPU limitations and restrictions . . . . . . . 10NVIDIA CUDA limitations and restrictions . . . . 11

Documentation . . . . . . . . . . . 13

Accessibility features for IBMSpectrum MPI. . . . . . . . . . . . 15

Notices . . . . . . . . . . . . . . 17Programming interface information . . . . . . 19

MPI support statement . . . . . . . . . 19Open MPI license . . . . . . . . . . . 19

Trademarks . . . . . . . . . . . . . . 20Terms and conditions . . . . . . . . . . . 20Privacy policy considerations . . . . . . . . 21

Index . . . . . . . . . . . . . . . 23

© Copyright IBM Corp. 2018 iii

iv IBM Spectrum MPI Version 10.2.0: Release Notes

About this information

The release notes include late technical information that is not included in other topics, and alsohighlights new functions for IBM Spectrum™ MPI Version 10.2.0.

Disclaimer:

The functions or features found herein may not be available on all operating systems or platforms and do notindicate the availability of these functions or features within the IBM® product or future versions of the IBMproduct. The development, release, and timing of any future features or functionality is at IBM's sole discretion.IBM's plans, directions, and intent are subject to change or withdrawal without notice at IBM's sole discretion. Theinformation mentioned is not a commitment, promise, or legal obligation to deliver any material, code orfunctionality. The information may not be incorporated into any contract and it should not be relied on in making apurchasing decision.

This document explains parallel programming as it relates to IBM Spectrum, IBM's implementation ofOpen MPI 3.0. It includes information about developing, running, and optimizing parallel applicationsfor use in a distributed memory environment.

IBM Spectrum MPI is a complete MPI implementation, based on the Open MPI open source project, andis designed to comply with all the requirements of the Message Passing Interface standard, MPI: AMessage-Passing Interface Standard, Version 3.1, University of Tennessee, Knoxville, Tennessee, June 4,2015.

For information about Open MPI, and to obtain official Open MPI documentation, refer to the Open MPIwebsite.

Note: This document can be used along with information that is provided in the Open MPI website. Inmany cases, this document explains a topic at a high level, and then points you to the Open MPI websitefor more detailed information.

Conventions and terminology used in this informationTable 1 shows the conventions used in this information:

Table 1. Conventions

Convention Usage

bold Environment variables.

monospace Examples and information that the system displays, command line options, file names,pathnames.

bold monospace Command names and parameter names.

italic Italic words or characters represent variable values that you must supply.

Italics are also used for unit titles, the first use of a term, and general emphasis in text.

<key> Angle brackets (less-than and greater-than) enclose the name of a key on the keyboard. Forexample, <Enter> refers to the key on your terminal or workstation that is labeled with the wordEnter.

\ In command examples, a backslash indicates that the command or coding example continues onthe next line. For example:

mkcondition -r IBM.FileSystem -e “PercentTotUsed > 90” \-E “PercentTotUsed < 85” -m d “FileSystem space used”

{item} Braces enclose a list from which you must choose an item in format and syntax descriptions.

[item] Brackets enclose optional items in format and syntax descriptions.

© Copyright IBM Corp. 2018 v

Table 1. Conventions (continued)

Convention Usage

<Ctrl-x> The notation <Ctrl-x> indicates a control character sequence. For example, <Ctrl-c> means thatyou hold down the control key while pressing <c>.

item... Ellipses indicate that you can repeat the preceding item one or more times.

| v In synopsis statements, vertical lines separate a list of choices. In other words, a vertical linemeans Or.

v In the margin of the document, vertical lines indicate technical changes to the information.

Prerequisite and related informationIBM Spectrum MPI is a member of the IBM Spectrum Computing family (www.ibm.com/systems/spectrum-computing/).

To access the most recent IBM Spectrum MPI documentation in PDF and HTML format, refer to IBMSpectrum MPI Knowledge Center website.

IBM Spectrum LSF (Load Sharing Facility) also works in conjunction with IBM Spectrum MPI. For thelatest LSF® documentation, see the IBM Spectrum LSF Knowledge Center website.

Terminology

For terms and definitions related to IBM Spectrum MPI, see the IBM Terminology website.

vi IBM Spectrum MPI Version 10.2.0: Release Notes

Overview

IBM Spectrum MPI Version 10.2.0 is a high-performance, production-quality implementation of theMessage Passing Interface (MPI). It is widely used in the high-performance computing (HPC) industry. Itis considered one of the standards for developing scalable, parallel applications. IBM Spectrum MPIVersion 10.2.0 is based on the Open MPI Version 3.0 and implements the full MPI 3.1 standard.

IBM Spectrum MPI was previously called IBM Platform MPI. IBM Spectrum MPI was significantlyupdated and now delivers an Open MPI implementation for HPC parallel applications with improvedperformance and scalability.

IBM Spectrum MPI Version 10.2.0 incorporates advanced CPU affinity features, dynamic selection ofnetwork interface libraries, superior workload manager integrations, and improved performance. Itsupports a broad range of industry-standard platforms, interconnects, and operating systems to helpensure that parallel applications can run almost anywhere.

IBM Spectrum MPI Version 10.2.0 delivers an improved, RDMA-capable Parallel Active MessagingInterface (PAMI) using Mellanox OFED on both x86_64 and IBM Power Systems™ processor-basedservers. It also offers an improved collective MPI library that supports the seamless use of GPU memorybuffers for the application developer. The library provides advanced logic to determine the fastestalgorithm for any collective operation.

© Copyright IBM Corp. 2018 1

2 IBM Spectrum MPI Version 10.2.0: Release Notes

Features

The following features are provided by IBM Spectrum MPI Version 10.2.0:

64-bit supportIBM Spectrum MPI can be used on 64-bit architectures and operating systems in Little Endianmode (for x86) and for IBM Power Systems servers (8335-GCA and 8335-GTA), with and withoutGPUs.

Thread safetyThe MPI_THREAD_MULTIPLE (multiple threads that are run within the MPI library) option issupported.

GPU support

IBM Spectrum MPI supports the following GPU-acceleration technologies:v NVIDIA GPUDirect RDMA on POWER9™ systems and x86 systems.v CUDA-aware MPI on POWER8®, POWER9, and x86 systems.

By default, GPU support is turned off. To turn on GPU support for GPUDirect RDMA, you canrun the mpirun -gpu command. GPUDirect RDMA is turned on by default on POWER8 systems.To turn off GPUDirect RDMA on POWER8 systems and turn on CUDA-aware support, you canrun the mpirun -gpu -disable_gdr command.

Note: To use the mpirun -gpu, you must have the NVIDIA drivers 396.19, or later, installed. Tocheck the version of the NVIDIA drives that are installed, run the cat /proc/driver/nvidia/versioncommand. The output that is displayed when you run this command is similar to the following:NVRM version: NVIDIA UNIX ppc64le Kernel Module 396.19 Sun Apr 8 01:06:36 PDT 2018

GCC version: gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC)

To downloaded the latest NVIDIA drivers, see the NVIDIA Drivers website.

For Power Systems users, the CUDA Toolkit version 9.2, or later, is required. For x86 users, theCUDA Toolkit version 7.5 is required.

IBM Platform LSFIBM Spectrum MPI supports the IBM Platform Load Sharing Facility (LSF) version 9.1.3 forstarting jobs. For more information, see the Running applications with IBM Platform LSF topic.

Debugger supportIBM Spectrum MPI supports the Allinea DDT and TotalView debuggers. For more information,see the Debugging applications topic.

PMIx IBM Spectrum MPI supports the extended version of the Process Management Interface (PMI),called PMI Exascale (PMIx) version 2.0.0. PMIx extends the PMI standard, including the existingPMI-1 and PMI-2 APIs, to support clusters of exascale size. For more information about PMIx,and to obtain the PMIx library and documentation, see the PMIx Programmer's Manual.

FCA (hcoll) support

For installations that use the InfiniBand interconnect, the Mellanox Fabric Collective Accelerator(FCA), which uses Core-Direct technology, can be used to accelerate collective operations. FCA isalso known as hcoll. hcoll 4.0, or later, is required.

Portable Hardware Locality (hwloc)

© Copyright IBM Corp. 2018 3

|

|

|

|

||||

|||

|||

|

||

IBM Spectrum MPI uses hwloc (Portable Hardware Locality), which is an API that navigates thehardware topology of your server. You can view an abbreviated image of the server's hardwareby using the --report-bindings option. For example:

% mpirun -np 1 --report-bindings ./any_mpi_program.x

[ibmgpu01:27613] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket

0[core 1[hwt 0-1]], socket 0[core 2[hwt 0-1]], socket 0[core 3[hwt 0-1]], socket

0[core 4[hwt 0-1]], socket 0[core 5[hwt 0-1]], socket 0[core 6[hwt 0-1]], socket

0[core 7[hwt 0-1]]: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..]

In the following example, the end of the output line indicates that the server has two sockets,each with eight cores, and that each core has two hyper-threads. This output also shows that thestarted MPI process is bound to the first socket:[BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..]

hwloc provides IBM Spectrum MPI with details about NUMA memory nodes, sockets, sharedcaches, cores and simultaneous multithreading, system attributes, and the location of I/O devices.You can use this information to place processes, and the memory associated with them, mostefficiently, and for best performance.

IBM Spectrum MPI includes hwloc version 1.11.7. For more information about hwloc, see theOpen MPI website.

MPI-IO

MPI has a number of subroutines that enable your application program to perform efficientparallel input/output operations. These subroutines (collectively referred to as MPI-IO) allowefficient file I/O on a data structure that is distributed across several tasks for computation, but isorganized in a unified way in a single underlying file. MPI-IO presupposes a single parallel filesystem that underlies all the tasks in the parallel job. For IBM Spectrum MPI, this parallel filesystem is IBM Spectrum Scale™ version 4.2.1.

For parallel I/O, IBM Spectrum MPI supports only ROMIO version 3.1.4. To understand howROMIO was built, use the ompi_info command, with the highest level of verbosity. For example:$ MPI_ROOT/bin/ompi_info -l 9 --param io romio314

MCA io: romio314 (MCA v2.1.0, API v2.0.0, Component

v10.1.0)

MCA io romio314: ---------------------------------------------------

MCA io romio314: parameter "io_romio314_priority" (current value:

"40", data source: default, level: 9 dev/all, type:

int)

Priority of the io romio component

MCA io romio314: parameter "io_romio314_delete_priority" (current

value: "40", data source: default, level: 9

dev/all, type: int)

Delete priority of the io romio component

MCA io romio314: informational "io_romio314_version" (current value:

"from MPICH v3.1.4", data source: default, level: 9

4 IBM Spectrum MPI Version 10.2.0: Release Notes

dev/all, type: string)

Version of ROMIO

MCA io romio314: informational "io_romio314_user_configure_params"

(current value: "", data source: default, level: 9

dev/all, type: string)

User-specified command line parameters passed to

ROMIO’s configure script

MCA io romio314: informational

"io_romio314_complete_configure_params" (current

value: " FROM_OMPI=yes CC=’gcc -std=gnu99’

CFLAGS=’-DNDEBUG -m64 -O3 -Wall -Wundef

-Wno-long-long -Wsign-compare -Wmissing-prototypes

-Wstrict-prototypes -Wcomment -pedantic

-Werror-implicit-function-declaration

-finline-functions -fno-strict-aliasing -pthread

--disable-aio --disable-weak-symbols

--enable-strict", data source: default, level: 9

dev/all, type: string)

Complete set of command line parameters passed to

ROMIO’s configure scrip

Features 5

6 IBM Spectrum MPI Version 10.2.0: Release Notes

Limitations and restrictions

Adapter affinity is enabled by default, when you are running MPI jobs over PAMI. This usually results inbetter performance when CPU binding is enabled (the default setting), and instructs each rank to use theInfiniBand adapter that is physically closest to the core where the rank is bound. However, either of thefollowing circumstances can lead to SEGVs (if you are using the -verbsbypass option) or a job that hangs.v If a user runs a job across nodes that have different numbers of InfiniBand adapters per node (for

example, if some nodes have two adapters, and other nodes have only one adapter).v If a user runs a job across nodes that have one InfiniBand adapter on one fabric, and another adapter

on a different fabric.

In either of these situations, you must disable adapter affinity by specifying the following option with thempirun option.PAMI_IBV_ADAPTER_AFFINITY=0

Adapter striping is also enabled by default, when you are running MPI jobs over PAMI. However, bothadapter affinity and adapter striping must be disabled before a user runs a job across a set of nodes inwhich one node contains a single link only. In this situation, to disable both adapter affinity and adapterstriping, specify the following environment variables with the mpirun option.PAMI_ENABLE_STRIPING=0

PAMI_IBV_DEVICE_NAME=mlx5_0:1

PAMI_IBV_ADAPTER_AFFINITY=0

IBM Spectrum MPI Version 10.2.0 supports OpenSHMEM. For more information about OpenSHMEM, seethe OpenSHMEM website.

When you are running MPI jobs over PAMI, you might experience a delay when the job finishes. Theamount of time for the delay might be more pronounced for jobs with a task geometry that placesmultiple ranks on a single compute node. This delay occurs because of the InfiniBand queue pair'sresource are cleaning up what PAMI performs when the MPI_Finalize API is called.

Multithread I/O is supported, except for File I/O.

By default, PAMI creates InfiniBand connections by using Reliably Connected (RC) queue pairs for eachdestination rank. When the MPI_Finalize API is called, these queue pairs are removed by PAMI. As jobsizes grow, the number of queue pairs also grow, which require more time to clean up when the job ends.

If a job is running on a single compute node, the delay can be reduced by adding the -pami_noib flag tothe mpirun command. If a job is run on more than one compute node, you can direct PAMI to create theInfiniBand connections for the node by using Dynamically Connected (DC) queue pairs. To use the DCqueue pairs, you must add the -x PAMI_IBV_ENABLE_DCT=1 flag to the mpirun command. PAMI switchesfrom its default settings to use DC queue pairs at certain task geometries. Directing PAMI to use DCqueue pairs at other geometries might impact latency. Therefore, you must carefully evaluate and testworkloads before you use DC queue pairs. You should consider using DC queue pairs only if the jobtermination time is determined to be excessive.

The following general limitation and restrictions apply to IBM Spectrum MPI Version 10.2.0:v IBM Spectrum MPI is not Application Binary Interface (ABI) compatible with any other MPI

implementations such as Open MPI, Platform MPI, or IBM PE Runtime Edition.

© Copyright IBM Corp. 2018 7

||

||||

|

|||

||||||||

v The IBM Spectrum MPI collectives component (libcollectives) does not support intercommunicators.For intercommunicator collective support, IBM Spectrum MPI relies on Open MPI collectivecomponents.

v The SMPI health checks (daxpy, CPU dgemm, GPU dgemm, jlink), that are used to check memorybandwidth, CPU and GPU throughput, and network bandwidth on a quiet cluster, do not support coreisolation. Therefore, when you are running SMPI health checks, do not pass the -core_isolationparameter to the LSF bsub command.

v There are multiple wrong answers with the Open MPI tuned collective component. Therefore,Spectrum MPI disabled the Open MPI tuned collective component by setting theOMPI_MCA_coll_tuned_priority = -1 parameter in the /opt/ibm/spectrum_mpi/etc/smpi.conf file.There are multiple wrong answers with some of the collective implementations in Mellanox HCOLLthat is included with MOFED 4.3.2. Therefore, the following collectives are disabled in the HCOLLcollective component by setting the following parameters in the /opt/ibm/spectrum_mpi/etc/smpi.conffile.– HCOLL_ML_DISABLE_REDUCE = 1

– HCOLL_ML_DISABLE_ALLREDUCE = 1

– HCOLL_ML_DISABLE_ALLTOALL = 1

– HCOLL_ML_DISABLE_AGATHERV = 1

Note: You can override values that are specified in the /opt/ibm/spectrum_mpi/etc/smpi.conf files bysetting the values in your environment where you run the mpirun and jsrun commands. You can alsooverride values that are specified in the /opt/ibm/spectrum_mpi/etc/smpi.conf files from the mpiruncommand line by running the mpirun -x <var>=<value> command, or from the jsrun command lineby running the jsrun -E <var>=<value> command.If you want to enable these parameters, you can modify the /opt/ibm/spectrum_mpi/etc/smpi.conf file.

v To prevent data integrity problems when you are writing to a file, you must use General Parallel FileSystem (GPFS) version 4.2.3.7, or later, or version 5.0.0.0, or later.

v The MPI_Comm_spawn API has reserved keys. When you are calling the MPI_Comm_spawn API, the path keyand the soft key are ignored in Open MPI 3.0 and in IBM Spectrum MPI.

v Dynamic process management (dynamic tasking) is supported on only IBM Power® System servers. Torun applications with dynamic tasking, you must run the mpirun -pami_dt command. When you runthe mpirun -pami_dt command, a different PAMI library is used that supports dynamic tasking buthas a lower performance for non-dynamic tasking communication operations. The MPI_Comm_acceptand MPI_Comm_connect APIs are not supported by dynamic tasking.If you run applications which use dynamic tasking, you must disable the IBM collectives library byusing the -mca coll ̂ IBM option.If you launch jobs with the mpirun command without LSF, you must specify the launch node in thehost list file or in the -host parameter. If you do not want to run processes on the launch node, youcan specify slots=0 in the host list file or specify launchnodename:0 for the -host parameter. When youmake these changes to the host list file or to the -host parameter, the value of the MPI_UNIVERSE_SIZEattribute is consistent with the number of processors that are available for running jobs that are startedwith the MPI_Comm_spawn API.Dynamic tasking in IBM Spectrum MPI is supported similar to the OpenMPI implementation. For moreinformation about keys that are supported, see the MPI_Comm_spawn man page.

v If you are running the ompi_info command, you might see the following warning message:mca_base_component_repository_open: unable to open mca_coll_hcoll: libhcoll.so.1: cannot open shared object file: No such file or directory (ignored)

To avoid this warning message and view additional HCOLL related output, run the ompi_info-HCOLL command.

v If you launch many processes per node, the local daemon might exhaust the file descriptor limit on thenode. You can increase the file descriptor limit by running the ulimit -n VALUE command, whereVALUE is the upper limit of file descriptors you want to define per-process on the node. You must set

8 IBM Spectrum MPI Version 10.2.0: Release Notes

||||

|||

||||

|

|

|

|

||||||

||

||

|||||

||

||||||

||

||

||

|||

the VALUE option on all compute nodes in the system. You can estimate the number of file descriptorsthat are needed by using the following formula: 25 + (7 x PPN), where PPN is the number ofprocesses that are launched on a single node. For example, if you had eight processes that are launchedon a single node, an approximate value for the VALUE option would be 81 (25+7x8). In this example, toset the file descriptor limit to 81, you can run the ulimit -n 81 command.

v If you are using nvprof, the NVIDIA profiler tool, you must use the -restrict_libs none option whenyou run the mpirun command or the jsrun -smpiargs command to avoid unnecessary fork() systemcalls that can lead to incorrect answers from memory that is not being pinned as expected.

v The default Parallel Active Messaging Interface (PAMI) settings for multi-host support might not beoptimal for bandwidth sensitive applications. You can use the following settings to achieve optimalnode aggregate bandwidth:PAMI_IBV_DEVICE_NAME=mlx5_0:1

PAMI_IBV_DEVICE_NAME_1=mlx5_3:1

In this example, the settings configure tasks on socket 0 to use HCA port 1 and tasks on socket 1 to useHCA port 2.

v The host list file is a flat text file that contains the names of the hosts on which your applications run.Each host is included on a separate line. For example, the following example shows the contents of asimple host list file called myhosts:node1.mydomain.com

node2.mydomain.com

node3.mydomain.com

node4.mydomain.com

In a host list file, the order of the nodes in the file is not preserved when you launch processes acrossresources. In the previous example, the host list that is named myhosts and the node1.mydomain.comentry might not be the first node used, even though it is listed first in the host list file. For example,the following might be the order in which the nodes are used:1. node3.mydomain.com2. node2.mydomain.com3. node1.mydomain.com4. node4.mydomain.com

v A failure might occur when many threads (seven or more) call the MPI_Intercomm_create API.v The use of the MPI_File_set_atomicity() API call is not supported.v While creating n-dimensional topologies by using the MPI_Dims_create() API calls, the ndims value

must be greater than 0.v While running Spectrum MPI over TCP on nodes with a virtual adapter, users must specify the correct

adapter that must be used by using the -netaddr option, because Spectrum MPI does not ignore thevirbr# named devices.

v Multithreaded I/O is not supported.v At the time of this release, the MPI standard has some ambiguity about the meaning of the

MPI_Comm_get_info(), MPI_Win_get_info(), and the MPI_File_get_info() APIs. These APIs return thecurrent internal value of each info setting, which can differ from values that are provided in a previousversion of these APIs. In the future, the MPI standard is likely to be changed to clarify that the get callsshould return the same values that were set. So, the current behavior of these APIs is not compliantwith the expected clarification of the MPI standard.

v If your switch network topology consists of more than one InfiniBand network, IBM Spectrum MPIrequires a unique subnet prefix (also known as the network ID) is assigned to each network. Ifdisconnected networks are assigned the same network ID, jobs might not run.

Limitations and restrictions 9

|||||

|||

|||

|||

||

|||

|||||||

||||

|

|

|

|

|

v IBM Spectrum MPI's use of the libevent v2.0.22 library might cause a perturbation in the randomnumber stream generated by the libc rand() and random() functions. Applications that rely on behaviorshould consider using the rand_r() and random_r() APIs.

v Querying or attempting to write the value of a performance variable by using the MPI Toolinformation interface (MPI_T) is not supported.

v Fortran MPI applications require the libgfortran3 runtime library to be installed on each compute node,regardless of which Fortran compiler was used to compile and link the application. This is arequirement because the libmpi_ibm_usempi.so.3 library depends on the libgfortran.so.3 library.

v On a node with GPUs, it is recommended that you run the following commands shortly after the nodeboots and before you run any GPU workload:nvidia-smi -pm 1 # persistence mode

nvidia-modprobe -u -c=0 # pre-load uvm

v The Open MPI collectives components that are included with IBM Spectrum MPI do not support GPUbuffers. For GPU buffer collective support, you must use the libcollectives library (the default library).

v GPU-accelerated applications are supported only if you are using the IBM Spectrum MPI PAMIbackend and the IBM collective library (libcollectives). These are the default options for IBM SpectrumMPI, but if you choose to use a different messaging protocol or collective component, GPUs are notsupported.

GPU limitations and restrictionsConsider the following GPU limitations and restrictions:v The GPU-accelerated applications are supported only if you are using the IBM Spectrum MPI PAMI

backend and the IBM collective library (libcollectives). These are the default options for IBM SpectrumMPI, but if you choose to use a different messaging protocol or collective component, GPUs are notsupported.

v The Open MPI collectives components that are included with IBM Spectrum MPI do not support GPUbuffers. For GPU buffer support, you must use libcollectives library (the default option).

v The behavior of IBM Spectrum MPI is undefined when an MPI application passes a GPU deviceaddress to an MPI API that is not CUDA-aware; the application might trigger a segmentation fault anddump core, or it might give incorrect results.Most MPI APIs are CUDA-aware, however, IBM Spectrum MPI does not supply header files or Fortranmodule files that identify which MPI APIs accept GPU device addresses as arguments. Refer to thepreceding restriction for a list of MPI APIs that might not be used with GPU device addresses.

v On a node with GPUs, it is recommended that you run the following commands immediately after thenode boots and before running any GPU workload:nvidia-smi -pm 1 # persistence mode

nvidia-modprobe -u -c=0 # pre-load uvm

v The libcollectives component is the only collective component that can support CUDA buffers. As aresult, when the -gpu option is specified with the mpirun command, the libcollectives library must bethe preferred collective component. Therefore, you cannot use the mpirun -gpu command with any ofthe following options:– -mxm/-MXM

– -mxmc/-MXMC

– -tcp/-TCP

– -ibv/-IBV

– -ib/-IB

– -openib/-OPENIB

– -FCA

10 IBM Spectrum MPI Version 10.2.0: Release Notes

– -HCOLL

– -hcoll

NVIDIA CUDA limitations and restrictionsConsider the following NVIDIA CUDA limitations and restrictions:v If an application is built by using the NVIDIA CUDA Toolkit, the NVIDIA CUDA Toolkit must be

installed on the node from which it is launched, and on each compute node. For Power Systems users,the CUDA Toolkit version 8.0 is required. For x86 users, the CUDA Toolkit version 7.5 is required.

v The following MPI functions are not CUDA-aware:– MPI_Alltoallw– MPI_Ialltoallw– MPI_Ineighbor_allgather– MPI_Ineighbor_allgatherv– MPI_Ineighbor_alltoall– MPI_Ineighbor_alltoallv– MPI_Ineighbor_alltoallw– MPI_Neighbor_allgather– MPI_Neighbor_allgatherv– MPI_Neighbor_alltoall– MPI_Neighbor_alltoallv– MPI_Neighbor_alltoallw

v If you are using the IBM XL Fortran compiler to build CUDA-aware MPI programs that employ thepartitioned global address space (PGAS) model, you must use XL compiler version 15.1.5, or later.

Limitations and restrictions 11

12 IBM Spectrum MPI Version 10.2.0: Release Notes

Documentation

To view the most recent IBM Spectrum MPI documentation in the PDF and HTML format, refer to theIBM Spectrum MPI Knowledge Center website.

© Copyright IBM Corp. 2018 13

14 IBM Spectrum MPI Version 10.2.0: Release Notes

Accessibility features for IBM Spectrum MPI

Accessibility features assist users who have a disability, such as restricted mobility or limited vision, touse information technology content successfully.

Accessibility features

IBM Spectrum MPI includes the following major accessibility features:v Keyboard-only operationv Operations that use a screen reader

IBM Spectrum MPI uses the latest W3C Standard, WAI-ARIA 1.0 (www.w3.org/TR/wai-aria/), to ensurecompliance with US Section 508 (www.access-board.gov/guidelines-and-standards/communications-and-it/about-the-section-508-standards/section-508-standards) and Web Content Accessibility Guidelines(WCAG) 2.0 (www.w3.org/TR/WCAG20/). To take advantage of accessibility features, use the latestrelease of your screen reader and the latest web browser that is supported by IBM Spectrum MPI.

The IBM Spectrum MPI online product documentation in IBM Knowledge Center is enabled foraccessibility. The accessibility features of IBM Knowledge Center are described at http://www.ibm.com/support/knowledgecenter/doc/kc_help.html#accessibility.

Related accessibility information

In addition to standard IBM help desk and support websites, IBM has established a TTY telephoneservice for use by deaf or hard of hearing customers to access sales and support services:

TTY service 800-IBM-3383 (800-426-3383) (within North America)

IBM and accessibility

For more information about the commitment that IBM has to accessibility, see IBM Accessibility(www.ibm.com/able).

© Copyright IBM Corp. 2018 15

16 IBM Spectrum MPI Version 10.2.0: Release Notes

Notices

This information was developed for products and services offered in the US.

IBM may not offer the products, services, or features discussed in this document in other countries.Consult your local IBM representative for information on the products and services currently available inyour area. Any reference to an IBM product, program, or service is not intended to state or imply thatonly that IBM product, program, or service may be used. Any functionally equivalent product, program,or service that does not infringe any IBM intellectual property right may be used instead. However, it isthe user's responsibility to evaluate and verify the operation of any non-IBM product, program, orservice.

IBM may have patents or pending patent applications covering subject matter described in thisdocument. The furnishing of this document does not grant you any license to these patents. You can sendlicense inquiries, in writing, to:

IBM Director of Licensing

IBM Corporation

North Castle Drive, MD-NC119

Armonk, NY 10504-1785

US

For license inquiries regarding double-byte character set (DBCS) information, contact the IBM IntellectualProperty Department in your country or send inquiries, in writing, to:

Intellectual Property Licensing

Legal and Intellectual Property Law

IBM Japan Ltd.

19-21, Nihonbashi-Hakozakicho, Chuo-ku

Tokyo 103-8510, Japan

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS"WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOTLIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY ORFITNESS FOR A PARTICULAR PURPOSE. Some jurisdictions do not allow disclaimer of express orimplied warranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodicallymade to the information herein; these changes will be incorporated in new editions of the publication.IBM may make improvements and/or changes in the product(s) and/or the program(s) described in thispublication at any time without notice.

Any references in this information to non-IBM websites are provided for convenience only and do not inany manner serve as an endorsement of those websites. The materials at those websites are not part ofthe materials for this IBM product and use of those websites is at your own risk.

© Copyright IBM Corp. 2018 17

IBM may use or distribute any of the information you provide in any way it believes appropriate withoutincurring any obligation to you.

Licensees of this program who wish to have information about it for the purpose of enabling: (i) theexchange of information between independently created programs and other programs (including thisone) and (ii) the mutual use of the information which has been exchanged, should contact:

IBM Director of Licensing

IBM Corporation

North Castle Drive, MD-NC119

Armonk, NY 10504-1785

US

Such information may be available, subject to appropriate terms and conditions, including in some cases,payment of a fee.

The licensed program described in this document and all licensed material available for it are providedby IBM under terms of the IBM Customer Agreement, IBM International Program License Agreement orany equivalent agreement between us.

The performance data and client examples cited are presented for illustrative purposes only. Actualperformance results may vary depending on specific configurations and operating conditions.

Information concerning non-IBM products was obtained from the suppliers of those products, theirpublished announcements or other publicly available sources. IBM has not tested those products andcannot confirm the accuracy of performance, compatibility or any other claims related to non-IBMproducts. Questions on the capabilities of non-IBM products should be addressed to the suppliers ofthose products.

Statements regarding IBM's future direction or intent are subject to change or withdrawal without notice,and represent goals and objectives only.

All IBM prices shown are IBM's suggested retail prices, are current and are subject to change withoutnotice. Dealer prices may vary.

This information is for planning purposes only. The information herein is subject to change before theproducts described become available.

This information contains examples of data and reports used in daily business operations. To illustratethem as completely as possible, the examples include the names of individuals, companies, brands, andproducts. All of these names are fictitious and any similarity to actual people or business enterprises isentirely coincidental.

COPYRIGHT LICENSE:

This information contains sample application programs in source language, which illustrate programmingtechniques on various operating platforms. You may copy, modify, and distribute these sample programsin any form without payment to IBM, for the purposes of developing, using, marketing or distributingapplication programs conforming to the application programming interface for the operating platform forwhich the sample programs are written. These examples have not been thoroughly tested under allconditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these

18 IBM Spectrum MPI Version 10.2.0: Release Notes

programs. The sample programs are provided "AS IS", without warranty of any kind. IBM shall not beliable for any damages arising out of your use of the sample programs.

Each copy or any portion of these sample programs or any derivative work must include a copyrightnotice as follows:

© (your company name) (year).

Portions of this code are derived from IBM Corp. Sample Programs.

© Copyright IBM Corp. _enter the year or years_.

Programming interface information

MPI support statementThe IBM Spectrum MPI product is a complete MPI implementation, based on the Open MPI open sourceproject, designed to comply with all the requirements of the Message Passing Interface standard, MPI: AMessage-Passing Interface Standard, Version 3.1, University of Tennessee, Knoxville, Tennessee, June 4,2015. If you believe that IBM Spectrum MPI does not comply with the MPI-3.1 standard, please contactIBM Service.

Open MPI licenseMost files in this release are marked with the copyrights of theorganizations who have edited them. The copyrights below are in noparticular order and generally reflect members of the Open MPI coreteam who have contributed code to this release. The copyrights forcode used under license from other parties are included in thecorresponding files.

Copyright (c) 2004-2010 The Trustees of Indiana University and IndianaUniversity Research and TechnologyCorporation. All rights reserved.

Copyright (c) 2004-2010 The University of Tennessee and The Universityof Tennessee Research Foundation. All rightsreserved.

Copyright (c) 2004-2010 High Performance Computing Center Stuttgart,University of Stuttgart. All rights reserved.

Copyright (c) 2004-2008 The Regents of the University of California.All rights reserved.

Copyright (c) 2006-2010 Los Alamos National Security, LLC. All rightsreserved.

Copyright (c) 2006-2010 Cisco Systems, Inc. All rights reserved.Copyright (c) 2006-2010 Voltaire, Inc. All rights reserved.Copyright (c) 2006-2011 Sandia National Laboratories. All rights reserved.Copyright (c) 2006-2010 Sun Microsystems, Inc. All rights reserved.

Use is subject to license terms.Copyright (c) 2006-2010 The University of Houston. All rights reserved.Copyright (c) 2006-2009 Myricom, Inc. All rights reserved.Copyright (c) 2007-2008 UT-Battelle, LLC. All rights reserved.Copyright (c) 2007-2018 IBM Corporation. All rights reserved.Copyright (c) 1998-2005 Forschungszentrum Juelich, Juelich Supercomputing

Centre, Federal Republic of GermanyCopyright (c) 2005-2008 ZIH, TU Dresden, Federal Republic of GermanyCopyright (c) 2007 Evergrid, Inc. All rights reserved.Copyright (c) 2008 Chelsio, Inc. All rights reserved.Copyright (c) 2008-2009 Institut National de Recherche en

Informatique. All rights reserved.Copyright (c) 2007 Lawrence Livermore National Security, LLC.

All rights reserved.Copyright (c) 2007-2009 Mellanox Technologies. All rights reserved.Copyright (c) 2006-2010 QLogic Corporation. All rights reserved.

Notices 19

Copyright (c) 2008-2010 Oak Ridge National Labs. All rights reserved.Copyright (c) 2006-2010 Oracle and/or its affiliates. All rights reserved.Copyright (c) 2009 Bull SAS. All rights reserved.Copyright (c) 2010 ARM ltd. All rights reserved.Copyright (c) 2010-2011 Alex Brick . All rights reserved.Copyright (c) 2012 The University of Wisconsin-La Crosse. All rights

reserved.Copyright (c) 2013-2016 Intel, Inc. All rights reserved.Copyright (c) 2011-2014 NVIDIA Corporation. All rights reserved.

$COPYRIGHT$

Additional copyrights may follow

$HEADER$

Redistribution and use in source and binary forms, with or withoutmodification, are permitted provided that the following conditions aremet:

- Redistributions of source code must retain the above copyrightnotice, this list of conditions and the following disclaimer.

- Redistributions in binary form must reproduce the above copyrightnotice, this list of conditions and the following disclaimer listedin this license in the documentation and/or other materialsprovided with the distribution.

- Neither the name of the copyright holders nor the names of itscontributors may be used to endorse or promote products derived fromthis software without specific prior written permission.

The copyright holders provide no reassurances that the source codeprovided does not infringe any patent, copyright, or any otherintellectual property rights of third parties. The copyright holdersdisclaim any liability to any recipient for claims brought againstrecipient by any third party for infringement of that partiesintellectual property rights.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOTLIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FORA PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHTOWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOTLIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANYTHEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USEOF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

TrademarksIBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International BusinessMachines Corp., registered in many jurisdictions worldwide. Other product and service names might betrademarks of IBM or other companies. A current list of IBM trademarks is available on the web atCopyright and trademark information at www.ibm.com/legal/copytrade.shtml.

Terms and conditionsPermissions for the use of these publications are granted subject to the following terms and conditions.

20 IBM Spectrum MPI Version 10.2.0: Release Notes

Applicability

These terms and conditions are in addition to any terms of use for the IBM website.

Personal use

You may reproduce these publications for your personal, noncommercial use provided that allproprietary notices are preserved. You may not distribute, display or make derivative work of thesepublications, or any portion thereof, without the express consent of IBM.

Commercial use

You may reproduce, distribute and display these publications solely within your enterprise provided thatall proprietary notices are preserved. You may not make derivative works of these publications, orreproduce, distribute or display these publications or any portion thereof outside your enterprise, withoutthe express consent of IBM.

Rights

Except as expressly granted in this permission, no other permissions, licenses or rights are granted, eitherexpress or implied, to the publications or any information, data, software or other intellectual propertycontained therein.

IBM reserves the right to withdraw the permissions granted herein whenever, in its discretion, the use ofthe publications is detrimental to its interest or, as determined by IBM, the above instructions are notbeing properly followed.

You may not download, export or re-export this information except in full compliance with all applicablelaws and regulations, including all United States export laws and regulations.

IBM MAKES NO GUARANTEE ABOUT THE CONTENT OF THESE PUBLICATIONS. THEPUBLICATIONS ARE PROVIDED "AS-IS" AND WITHOUT WARRANTY OF ANY KIND, EITHEREXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO IMPLIED WARRANTIES OFMERCHANTABILITY, NON-INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE.

Privacy policy considerationsIBM Software products, including software as a service solutions, (“Software Offerings”) may use cookiesor other technologies to collect product usage information, to help improve the end user experience, totailor interactions with the end user or for other purposes. In many cases no personally identifiableinformation is collected by the Software Offerings. Some of our Software Offerings can help enable you tocollect personally identifiable information. If this Software Offering uses cookies to collect personallyidentifiable information, specific information about this offering’s use of cookies is set forth below.

This Software Offering does not use cookies or other technologies to collect personally identifiableinformation.

If the configurations deployed for this Software Offering provide you as the customer the ability to collectpersonally identifiable information from end users via cookies and other technologies, you should seekyour own legal advice about any laws applicable to such data collection, including any requirements fornotice and consent.

For more information about the use of various technologies, including cookies, for these purposes, seeIBM’s Privacy Policy at http://www.ibm.com/privacy and IBM’s Online Privacy Statement athttp://www.ibm.com/privacy/details the section entitled “Cookies, Web Beacons and Other

Notices 21

Technologies” and the “IBM Software Products and Software-as-a-Service Privacy Statement” athttp://www.ibm.com/software/info/product-privacy.

22 IBM Spectrum MPI Version 10.2.0: Release Notes

Index

Aaccessibility features for this product 15

Cconventions and terminology v

© Copyright IBM Corp. 2018 23

24 IBM Spectrum MPI Version 10.2.0: Release Notes

IBM®

Printed in USA