Environment Variables

I_MPI_MIC

Syntax

I_MPI_MIC=<value>

Arguments

<value>

Intel® Xeon Phi recognition

enable | yes | on | 1

Enable the Intel Xeon Phi coprocessor recognition

disable | no | off | 0

Disables the Intel Xeon Phi coprocessor recognition. This is the default value

Description

Set this environment variable to control whether the Intel Xeon processor of the Intel® MPI Library tries to detect and work with the Intel® MIC Architecture components.

If the value of environment variable I_MPI_MIC is enable, the default value of environment variable I_MPI_SSHM is enable.

Note

This is a provisional variable and is only temporarily introduced, until the architecture detection and other related matters are clarified.

I_MPI_MIC_PREFIX

Syntax

I_MPI_MIC_PREFIX=<value>

Arguments

<value>

Specify a string as the prefix of an Intel Xeon Phi coprocessor file name. The default value is an empty string

Description

Set this environment variable to add a prefix to a host executable file name to get a corresponding Intel Xeon Phi coprocessor executable file name.

For example, set different locations as the value for the I_MPI_MIC_PREFIX environment variable to distinguish Intel MIC Architecture and Intel® 64 Architecture executable files:

(host)$ mpiicc test.c -o test_hello

(host)$ mpiicc -mmic test.c -o ./MIC/test_hello

(host)$ export I_MPI_MIC=1

(host)$ export I_MPI_MIC_PREFIX=./MIC/

(host)$ mpirun -n 4 -hostfile <hostfile> test_hello

In the example, ./test_hello binary is launched on Intel® 64 Architecture nodes and ./MIC/test_hello binary is launched on Intel Xeon Phi coprocessor nodes.

I_MPI_MIC_POSTFIX

Syntax

I_MPI_MIC_POSTFIX=<value>

Arguments

<value>

Specify a string as the postfix of an Intel Xeon Phi coprocessor file name. The default value is an empty string

Description

Set this environment variable to add a postfix to a host executable name to get a corresponding Intel Xeon Phi coprocessor executable file name.

For example, set different names as the value for the I_MPI_MIC_POSTFIX environment variable to distinguish Intel Xeon Phi coprocessor and Intel 64 Architecture executable files:

(host)$ mpiicc test.c -o test_hello

(host)$ mpiicc -mmic test.c -o test_hello.mic

(host)$ export I_MPI_MIC=1

(host)$ export I_MPI_MIC_POSTFIX=.mic

(host)$ mpirun -n 4 -hostfile <hostfile> test_hello

In the example, test_hello binary is launched on Intel 64 Architecture nodes and test_hello.mic binary on Intel Xeon Phi coprocessor nodes.

I_MPI_DAPL_PROVIDER_LIST

Syntax

I_MPI_DAPL_PROVIDER_LIST=<primary provider>[,<local secondary provider> [,<remote secondary provider>]]

Arguments

<primary provider>

Provides the best latency and available on all network segments (cross box and within box)

<local secondary provider>

Provides the best bandwidth for local configurations. The distance is less than the value of the  I_MPI_DAPL_LOCALITY_THRESHOLD environment variable

<remote secondary provider>

Provides best bandwidth for remote configurations. The distance greater than the value of the  I_MPI_DAPL_LOCALITY_THRESHOLD environment variable

Description

Use this variable to define the DAPL providers to load.

With Intel® Manycore Platform Software Stack (Intel® MPSS), the arguments of I_MPI_DAPL_PROVIDER_LIST are set as the following values:

Thus, the setting is I_MPI_DAPL_PROVIDER_LIST=<CCL-direct>[,<IBSCIF>[,<CCL-proxy>]]

The following configuration is an example with the default dat.conf provided with Intel MPSS:

I_MPI_DAPL_PROVIDER_LIST=ofa-v2-mlx4_0-1u,ofa-v2-scif0,ofa-v2-mcm-1

You can adjust the threshold for secondary provider through the I_MPI_DAPL_DIRECT_COPY_THRESHOLD environment variable (<secondary provider threshold>):

I_MPI_DAPL_DIRECT_COPY_THRESHOLD=<primary provider direct copy threshold>[,<secondary provider threshold>]

<primary provider direct copy threshold> has to be lower than <secondary provider threshold>.

If the environment variable I_MPI_DAPL_PROVIDER_LIST contains a list of values, then the syntax of the following environment variables may be extended by the values related to all corresponding providers.

If you set only single value, this value applies to all providers. In case of mismatch or incorrect values, the default value is used for all providers.

For example:

export I_MPI_DAPL_PROVIDER_LIST=ofa-v2-mlx4_0-1,ofa-v2-scif0

export I_MPI_DAPL_TRANSLATION_CACHE=enable,disable

This I_MPI_DAPL_TRANSLATION_CACHE setting turns on the memory registration cache for the first provider; but turns it off for the second one.

I_MPI_DAPL_LOCALITY_THRESHOLD

Define the threshold to switch from local secondary provider to remote secondary provider.

Syntax

I_MPI_DAPL_LOCALITY_THRESHOLD=<value>

Arguments

<value>

Defines the threshold to switch from local secondary provider to remote secondary provider.

See I_MPI_DAPL_PROVIDER_LIST for the detail options descriptions

See the following description

The default value depends on Non-Uniform Memory Architecture (NUMA) configuration and DAPL*/Intel® Manycore Platform Software Stack (Intel® MPSS) version.

Description

This value from range [10; 255] is associated with NUMA distances inside the host and extrapolated for Intel Xeon Phi coprocessors. The default value relates to clusters and jobs, and is defined with the formulas shown below.

Starting from DAPL* 2.1.3, there is an auto adjustment logic which uses the formula: 255 - dmax + dmin. The following table shows the list for formulas that you should use for the I_MPI_DAPL_LOCALITY_THRESHOLD default value in difference cases:

Formula for the Default Value

Mode

d + (255 - 1 - dmax)

Two ranks are executed on different Intel Xeon Phi coprocessors of the same host

d + (255 - 1 - dmax) / 2

One rank is executed on Intel Xeon Phi coprocessor and another on the same host

d

Both ranks are executed inside the host

255

Two ranks are executed on different hosts

In the above formula:

Before DAPL 2.1.3, I_MPI_DAPL_LOCALITY_THRESHOLD is equal 255 by default, what conforms to a local secondary provider will be chosen for all ranks which are executed inside the box; otherwise a remote secondary provider will be chosen.

I_MPI_ENV_PREFIX_LIST

Define the prefixes of environment variables for the intended platforms.

Syntax

I_MPI_ENV_PREFIX_LIST=[platform:prefix][,...]

Argument

platform

The intended platform (string).

Options: htn,nhm,wsm,snb,ivb

See I_MPI_PLATFORM for the detail options descriptions

prefix

A prefix (string) for a name of an environment variable to be used for the intended platform

Description

Set this environment variable to define the prefix of environment variables to be used for the intended platform.

If you specify a prefix in I_MPI_ENV_PREFIX_LIST for an environment variable, the prefixed environment variable overrides the respective non-prefixed environment variable on the intended platform.

If you do not specify I_MPI_ENV_PREFIX_LIST, environment variables are applied to all platforms.

Note

Use the lower case when you specify the platform names.

Examples

  1. I_MPI_ENV_PREFIX_LIST=platform:prefix

    <NAME>=value is applied to all systems.

    <prefix>_<NAME>=value defines <NAME>=value for all <platform> systems.

  2. Assume that some machines are on the Intel® microarchitecture code name Sandy Bridge based platform, and the rest machines are on other architectures based platforms. The environment variable OMP_NUM_THREADS value is 3 on all platforms.

    To set OMP_NUM_THREADS=5 for the ranks on the Intel® microarchitecture code name Sandy Bridge based platform, specify the prefix in I_MPI_ENV_PREFIX_LIST for OMP_NUM_THREADS with the following configurations:

    I_MPI_ENV_PREFIX_LIST=snb:<prefix>
    OMP_NUM_THREADS=3
    <prefix>_OMP_NUM_THREADS=5