Intel® MPI Library Reference Manual for Linux* OS
I_MPI_MIC=<value>
<value> |
Intel® Xeon Phi™ recognition |
enable | yes | on | 1 |
Enable the Intel Xeon Phi coprocessor recognition |
disable | no | off | 0 |
Disables the Intel Xeon Phi coprocessor recognition. This is the default value |
Set this environment variable to control whether the Intel Xeon processor of the Intel® MPI Library tries to detect and work with the Intel® MIC Architecture components.
If the value of environment variable I_MPI_MIC is enable, the default value of environment variable I_MPI_SSHM is enable.
This is a provisional variable and is only temporarily introduced, until the architecture detection and other related matters are clarified.
I_MPI_MIC_PREFIX=<value>
<value> |
Specify a string as the prefix of an Intel Xeon Phi coprocessor file name. The default value is an empty string |
Set this environment variable to add a prefix to a host executable file name to get a corresponding Intel Xeon Phi coprocessor executable file name.
For example, set different locations as the value for the I_MPI_MIC_PREFIX environment variable to distinguish Intel MIC Architecture and Intel® 64 Architecture executable files:
(host)$ mpiicc test.c -o test_hello
(host)$ mpiicc -mmic test.c -o ./MIC/test_hello
(host)$ export I_MPI_MIC=1
(host)$ export I_MPI_MIC_PREFIX=./MIC/
(host)$ mpirun -n 4 -hostfile <hostfile> test_hello
In the example, ./test_hello binary is launched on Intel® 64 Architecture nodes and ./MIC/test_hello binary is launched on Intel Xeon Phi coprocessor nodes.
I_MPI_MIC_POSTFIX=<value>
<value> |
Specify a string as the postfix of an Intel Xeon Phi coprocessor file name. The default value is an empty string |
Set this environment variable to add a postfix to a host executable name to get a corresponding Intel Xeon Phi coprocessor executable file name.
For example, set different names as the value for the I_MPI_MIC_POSTFIX environment variable to distinguish Intel Xeon Phi coprocessor and Intel 64 Architecture executable files:
(host)$ mpiicc test.c -o test_hello
(host)$ mpiicc -mmic test.c -o test_hello.mic
(host)$ export I_MPI_MIC=1
(host)$ export I_MPI_MIC_POSTFIX=.mic
(host)$ mpirun -n 4 -hostfile <hostfile> test_hello
In the example, test_hello binary is launched on Intel 64 Architecture nodes and test_hello.mic binary on Intel Xeon Phi coprocessor nodes.
I_MPI_DAPL_PROVIDER_LIST=<primary provider>[,<local secondary provider> [,<remote secondary provider>]]
<primary provider> |
Provides the best latency and available on all network segments (cross box and within box) |
<local secondary provider> |
Provides the best bandwidth for local configurations. The distance is less than the value of the I_MPI_DAPL_LOCALITY_THRESHOLD environment variable |
<remote secondary provider> |
Provides best bandwidth for remote configurations. The distance greater than the value of the I_MPI_DAPL_LOCALITY_THRESHOLD environment variable |
Use this variable to define the DAPL providers to load.
With Intel® Manycore Platform Software Stack (Intel® MPSS), the arguments of I_MPI_DAPL_PROVIDER_LIST are set as the following values:
<primary provider>- CCL-direct
<local secondary provider>- IBSCIF
<remote secondary provider>- CCL-proxy
Thus, the setting is I_MPI_DAPL_PROVIDER_LIST=<CCL-direct>[,<IBSCIF>[,<CCL-proxy>]]
The following configuration is an example with the default dat.conf provided with Intel MPSS:
I_MPI_DAPL_PROVIDER_LIST=ofa-v2-mlx4_0-1u,ofa-v2-scif0,ofa-v2-mcm-1
You can adjust the threshold for secondary provider through the I_MPI_DAPL_DIRECT_COPY_THRESHOLD environment variable (<secondary provider threshold>):
I_MPI_DAPL_DIRECT_COPY_THRESHOLD=<primary provider direct copy threshold>[,<secondary provider threshold>]
<primary provider direct copy threshold> has to be lower than <secondary provider threshold>.
If the environment variable I_MPI_DAPL_PROVIDER_LIST contains a list of values, then the syntax of the following environment variables may be extended by the values related to all corresponding providers.
I_MPI_DAPL_DIRECT_COPY_THRESHOLD
I_MPI_DAPL_TRANSLATION_CACHE
I_MPI_DAPL_TRANSLATION_CACHE_AVL_TREE
I_MPI_DAPL_CONN_EVD_SIZE
I_MPI_DAPL_RDMA_RNDV_WRITE
If you set only single value, this value applies to all providers. In case of mismatch or incorrect values, the default value is used for all providers.
For example:
export I_MPI_DAPL_PROVIDER_LIST=ofa-v2-mlx4_0-1,ofa-v2-scif0
export I_MPI_DAPL_TRANSLATION_CACHE=enable,disable
This I_MPI_DAPL_TRANSLATION_CACHE setting turns on the memory registration cache for the first provider; but turns it off for the second one.
Define the threshold to switch from local secondary provider to remote secondary provider.
I_MPI_DAPL_LOCALITY_THRESHOLD=<value>
<value> |
Defines the threshold to switch from local secondary provider to remote secondary provider. See I_MPI_DAPL_PROVIDER_LIST for the detail options descriptions |
See the following description |
The default value depends on Non-Uniform Memory Architecture (NUMA) configuration and DAPL*/Intel® Manycore Platform Software Stack (Intel® MPSS) version. |
This value from range [10; 255] is associated with NUMA distances inside the host and extrapolated for Intel Xeon Phi coprocessors. The default value relates to clusters and jobs, and is defined with the formulas shown below.
Starting from DAPL* 2.1.3, there is an auto adjustment logic which uses the formula: 255 - dmax + dmin. The following table shows the list for formulas that you should use for the I_MPI_DAPL_LOCALITY_THRESHOLD default value in difference cases:
Formula for the Default Value |
Mode |
d + (255 - 1 - dmax) |
Two ranks are executed on different Intel Xeon Phi coprocessors of the same host |
d + (255 - 1 - dmax) / 2 |
One rank is executed on Intel Xeon Phi coprocessor and another on the same host |
d |
Both ranks are executed inside the host |
255 |
Two ranks are executed on different hosts |
In the above formula:
d - host system NUMA distance.
dmin - minimal host system NUMA distance.
dmax - maximal host system NUMA distance.
Before DAPL 2.1.3, I_MPI_DAPL_LOCALITY_THRESHOLD is equal 255 by default, what conforms to a local secondary provider will be chosen for all ranks which are executed inside the box; otherwise a remote secondary provider will be chosen.
Define the prefixes of environment variables for the intended platforms.
I_MPI_ENV_PREFIX_LIST=[platform:prefix][,...]
platform |
The intended platform (string). Options: htn,nhm,wsm,snb,ivb See I_MPI_PLATFORM for the detail options descriptions |
prefix |
A prefix (string) for a name of an environment variable to be used for the intended platform |
Set this environment variable to define the prefix of environment variables to be used for the intended platform.
If you specify a prefix in I_MPI_ENV_PREFIX_LIST for an environment variable, the prefixed environment variable overrides the respective non-prefixed environment variable on the intended platform.
If you do not specify I_MPI_ENV_PREFIX_LIST, environment variables are applied to all platforms.
Use the lower case when you specify the platform names.
I_MPI_ENV_PREFIX_LIST=platform:prefix
<NAME>=value is applied to all systems.
<prefix>_<NAME>=value defines <NAME>=value for all <platform> systems.
Assume that some machines are on the Intel® microarchitecture code name Sandy Bridge based platform, and the rest machines are on other architectures based platforms. The environment variable OMP_NUM_THREADS value is 3 on all platforms.
To set OMP_NUM_THREADS=5 for the ranks on the Intel® microarchitecture code name Sandy Bridge based platform, specify the prefix in I_MPI_ENV_PREFIX_LIST for OMP_NUM_THREADS with the following configurations:
I_MPI_ENV_PREFIX_LIST=snb:<prefix>
OMP_NUM_THREADS=3
<prefix>_OMP_NUM_THREADS=5