DAPL UD-capable Network Fabrics Control

I_MPI_DAPL_UD

Enable/disable using DAPL UD path.

Syntax

I_MPI_DAPL_UD=<arg>

Arguments

<arg>

Binary indicator

enable | yes | on | 1

Turn on using DAPL UD IB extension

disable | no | off | 0

Turn off using DAPL UD IB extension. This is the default value

Description

Set this environment variable to enable DAPL UD path for transferring data. The algorithm is enabled if you set this environment variable and a certain DAPL provider attribute indicates that UD IB extension is supported.

I_MPI_DAPL_UD_PROVIDER

Define the DAPL provider to work with IB UD transport.

Syntax

I_MPI_DAPL_UD_PROVIDER=<name>

Arguments

<name>

Define the name of DAPL provider to load

Description

Set this environment variable to define the name of DAPL provider to load. This name is also defined in the dat.conf configuration file. Make sure that specified DAPL provider supports UD IB extension.

I_MPI_DAPL_UD_DIRECT_COPY_THRESHOLD

Change the message size threshold of the DAPL UD direct-copy protocol.

Syntax

I_MPI_DAPL_UD_DIRECT_COPY_THRESHOLD=<nbytes>

Arguments

<nbytes>

Define the DAPL UD direct-copy protocol threshold

> 0

The default <nbytes> value is equal to 16456 bytes

Description

Set this environment variable to control the DAPL UD direct-copy protocol threshold. Data transfer algorithms for the DAPL-capable network fabrics are selected based on the following scheme:

This environment variable is available for both IntelĀ® and non-Intel microprocessors, but it may perform additional optimizations for Intel microprocessors than it performs for non-Intel microprocessors.

I_MPI_DAPL_UD_RECV_BUFFER_NUM

Change the number of the internal pre-registered UD buffers for receiving messages.

Syntax

I_MPI_DAPL_UD_RECV_BUFFER_NUM=<nbuf>

Arguments

<nbuf>

Define the number of buffers for receiving messages

> 0

The default value is 16 + n*4 where n is a total number of process in MPI job

Description

Set this environment variable to change the number of the internal pre-registered buffers for receiving messages. These buffers are common for all connections or process pairs.

Note

The pre-registered buffers use up memory, however they help avoid the loss of packets.

I_MPI_DAPL_UD_SEND_BUFFER_NUM

Change the number of internal pre-registered UD buffers for sending messages.

Syntax

I_MPI_DAPL_UD_SEND_BUFFER_NUM=<nbuf>

Arguments

<nbuf>

Define the number of buffers for sending messages

> 0

The default value is 16 + n*4 where n is a total number of process in MPI job

Description

Set this environment variable to change the number of the internal pre-registered buffers for sending messages. These buffers are common for all connections or process pairs.

Note

The pre-registered buffers use up memory, however they help avoid the loss of packets.

I_MPI_DAPL_UD_ACK_SEND_POOL_SIZE

Change the number of ACK UD buffers for sending messages.

Syntax

I_MPI_DAPL_UD_ACK_SEND_POOL_SIZE=<nbuf>

Arguments

<nbuf>

Define the number of ACK UD buffers for sending messages

> 0

The default value is 256

Description

Set this environment variable to change the number of the internal pre-registered ACK buffers for sending service messages. These buffers are common for all connections or process pairs.

I_MPI_DAPL_UD_ACK_RECV_POOL_SIZE

Change the number of ACK UD buffers for receiving messages.

Syntax

I_MPI_DAPL_UD_ACK_RECV_POOL_SIZE=<nbuf>

Arguments

<nbuf>

Define the number of ACK UD buffers for receiving messages

> 0

The default value is 512+n*4, where n is total number of process in MPI job

Description

Set this environment variable to change the number of the internal pre-registered ACK buffers for receiving service messages. These buffers are common for all connections or process pairs.

I_MPI_DAPL_UD_TRANSLATION_CACHE

Turn on/off the memory registration cache in the DAPL UD path.

Syntax

I_MPI_DAPL_UD_TRANSLATION_CACHE=<arg>

Arguments

<arg>

Binary indicator

enable | yes | on | 1

Turn on the memory registration cache. This is the default

disable | no | off | 0

Turn off the memory registration cache

Description

Set this environment variable to turn off the memory registration cache in the DAPL UD path.

Using the cache substantially improves performance. See product Release Notes for further details.

I_MPI_DAPL_UD_TRANSLATION_CACHE_AVL_TREE

Enable/disable the AVL* tree based implementation of RDMA translation cache in the DAPL UD path.

Syntax

I_MPI_DAPL_UD_TRANSLATION_CACHE_AVL_TREE=<arg>

Arguments

<arg>

Binary indicator

enable | yes | on | 1

Turn on the AVL tree based RDMA translation cache

disable | no | off | 0

Turn off the AVL tree based RDMA translation cache. This is the default value

Description

Set this environment variable to enable the AVL tree based implementation of RDMA translation cache in the DAPL UD path. When the search in RDMA translation cache handles over 10,000 elements, the AVL tree based RDMA translation cache is faster than the default implementation.

I_MPI_DAPL_UD_REQ_EVD_SIZE

Define the event queue size of the DAPL UD event dispatcher for sending data transfer operations.

Syntax

I_MPI_DAPL_UD_REQ_EVD_SIZE=<size>

Arguments

<size>

Define the length of the event queue

> 0

The default value is 2,000

Description

Set this environment variable to define the event queue size of the DAPL event dispatcher that handles completions of sending DAPL UD data transfer operations (DTO). If this environment variable is set, the minimum value between <size> and the value obtained from the provider is used as the size of the event queue. The provider is required to supply a queue size that is at least equal to the calculated value, but it can also provide a larger queue size.

I_MPI_DAPL_UD_CONN_EVD_SIZE

Define the event queue size of the DAPL UD event dispatcher for connections.

Syntax

I_MPI_DAPL_UD_CONN_EVD_SIZE=<size>

Arguments

<size>

Define the length of the event queue

> 0

The default value is 2*number of processes + 32

Description

Set this environment variable to define the event queue size of the DAPL event dispatcher that handles connection related events. If this environment variable is set, the minimum value between <size> and the value obtained from the provider is used as the size of the event queue. The provider is required to supply a queue size that is at least equal to the calculated value, but it can also provide a larger queue size.

I_MPI_DAPL_UD_RECV_EVD_SIZE

Define the event queue size of the DAPL UD event dispatcher for receiving data transfer operations.

Syntax

I_MPI_DAPL_UD_RECV_EVD_SIZE=<size>

Arguments

<size>

Define the length of the event queue

> 0

The default value depends on the number UD and ACK buffers

Description

Set this environment variable to define the event queue size of the DAPL event dispatcher that handles completions of receiving DAPL UD data transfer operations (DTO). If this environment variable is set, the minimum value between <size> and the value obtained from the provider is used as the size of the event queue. The provider is required to supply a queue size that is at least equal to the calculated value, but it can also provide a larger queue size.

I_MPI_DAPL_UD_RNDV_MAX_BLOCK_LEN

Define maximum size of block that is passed at one iteration of DAPL UD direct-copy protocol.

Syntax

I_MPI_DAPL_UD_RNDV_MAX_BLOCK_LEN=<nbytes>

Arguments

<arg>

Define maximum size of block that is passed at one iteration of DAPL UD direct-copy protocol

> 0  

The default value is 1,048,576

Set this environment variable to define the maximum size of memory block that is passed at one iteration of DAPL UD direct-copy protocol. If the size of message in direct-copy protocol is greater than given value, the message will be divided in several blocks and passed in several operations.

I_MPI_DAPL_UD_RNDV_BUFFER_ALIGNMENT

Define the alignment of the sending buffer for the DAPL UD direct-copy transfers.

Syntax

I_MPI_DAPL_UD_RNDV_BUFFER_ALIGNMENT=<arg>

Arguments

<arg>

Define the alignment of the sending buffer

> 0 and a power of 2

The default value is 16

Set this environment variable to define the alignment of the sending buffer for DAPL direct-copy transfers. When a buffer specified in a DAPL operation is aligned to an optimal value, this may increase data transfer bandwidth.

I_MPI_DAPL_UD_RNDV_COPY_ALIGNMENT_THRESHOLD

Define threshold where alignment is applied to send buffer for the DAPL UD direct-copy transfers.

Syntax

I_MPI_DAPL_UD_RNDV_COPY_ALIGNMENT_THRESHOLD=<nbytes>

Arguments

<nbytes>

Define send buffer alignment threshold

> 0 and a power of 2

The default value is 32,768

Set this environment variable to define the threshold where the alignment of the sending buffer is applied in DAPL direct-copy transfers. When a buffer specified in a DAPL operation is aligned to an optimal value, this may increase data transfer bandwidth.

I_MPI_DAPL_UD_RNDV_DYNAMIC_CONNECTION

Control the algorithm of dynamic connection establishment for DAPL UD endpoints used in the direct copy protocol.

Syntax

I_MPI_DAPL_UD_RNDV_DYNAMIC_CONNECTION=<arg>

Arguments

<arg>

Binary indicator

enable | yes | on | 1

Turns on the dynamic connection mode. This is the default value

disable | no | off | 0

Turns off the dynamic connections mode

Set this variable to control the dynamic connection establishment of DAPL UD endpoints used in the direct copy protocol.

If you disable the dynamic connection mode, all possible connections are established during the MPI startup phase.

If you enable the mode, the connection is established when an application calls the MPI function to pass the data from one process to another and invokes the communication between the two processes.

Note

For the RNDV dynamic connection mode, the size of the messages passed in the data is larger than the value you set in the I_MPI_DAPL_UD_DIRECT_COPY_THRESHOLD environment variable.

I_MPI_DAPL_UD_EAGER_DYNAMIC_CONNECTION

Control the algorithm of the dynamic connection establishment for DAPL UD endpoints used in eager protocol.

Syntax

I_MPI_DAPL_UD_EAGER_DYNAMIC_CONNECTION=<arg>

Arguments

<arg>

Binary indicator

enable | yes | on | 1

Turn on the dynamic connection mode. If the number of processes is over 64, this is the default value

disable | no | off | 0

Turn off the dynamic connections mode

Set this variable to control the dynamic connection establishment of DAPL UD endpoints involved in eager protocol. Eager protocol is used to transfer messages through internal pre-registered buffers.

If you disable this mode, all possible connections are established during MPI startup phase.

If you enable this mode, the connection is established when an application calls the MPI function to pass the data from one process to another and invokes the communication between the two processes.

Note

For the eager dynamic connection mode, the size of the messages passed in the data is shorter than or equal to the value you set in the I_MPI_DAPL_UD_DIRECT_COPY_THRESHOLD environment variable.

I_MPI_DAPL_UD_DESIRED_STATIC_CONNECTIONS_NUM

Define the number of processes that establish DAPL static connections at the same time.

Syntax

I_MPI_DAPL_UD_DESIRED_STATIC_CONNECTIONS_NUM=<num_procesess>

Arguments

<num_procesess>

Define the number of processes that establish DAPL UD static connections at the same time

> 0

The default value is equal to 200

Description

Set this environment variable to control the algorithm of DAPL UD static connections establishment.

If the number of processes in an MPI job is less than or equal to <num_procesess>, all MPI processes establish the static connections simultaneously. Otherwise, the processes are distributed into several groups. The number of processes in each group is calculated to be close to <num_procesess>. Then static connections are established in several iterations, including intergroup connection setup.

I_MPI_DAPL_UD_RDMA_MIXED

Control the use of the DAPL UD/RDMA mixed communication.

Syntax

I_MPI_DAPL_UD_RDMA_MIXED =<arg>

Arguments

<arg>

Binary indicator

enable | yes | on | 1

Turn on the use of DAPL UD/RDMA mixed communication

disable | no | off | 0

Turn off the use of DAPL UD/RDMA mixed communication. This is the default value

Description

Set this environment variable to enable the DAPL UD/RDMA mixed mode for transferring data. In the DAPL UD/RDMA mixed mode, small messages are passed through the UD transport and large messages are passed through the RDMA transport. If you set the I_MPI_DAPL_UD_RDMA_MIXED environment variable and a certain DAPL provider attribute indicates that UD IB extension is supported, the DAPL UD/RDMA mixed mode is enabled.

The following set of I_MPI_DAPL_UD* environment variables also controls the DAPL UD/RDMA mixed mode:

The following set of environment variables is specific for DAPL UD/RDMA mixed mode:

I_MPI_DAPL_UD_MAX_RDMA_SIZE

Control the maximum message size that can be sent though the RDMA for DAPL UD/RDMA mixed mode.

Syntax

I_MPI_DAPL_UD_MAX_RDMA_SIZE =<nbytes>

Arguments

<nbytes>

Define the maximum message size that can be sent through RDMA without fragmentation

> 0

The default <nbytes> value is 4 MB

Description

Set this environment variable to define the maximum message size that can be sent though RDMA protocol for the DAPL UD/RDMA mixed mode. If the message size is greater than this value, this message is divided into several fragments and is sent by several RDMA operations.

I_MPI_DAPL_UD_MAX_RDMA_DTOS

Control the maximum number of uncompleted RDMA operations per connection for the DAPL UD/RDMA mixed mode.

Syntax

I_MPI_DAPL_UD_MAX_RDMA_DTOS=<arg>

Arguments

<arg>

Define the maximum number of RDMA operations per connection

> 0

The default <arg> value is 8

Description

Set this environment variable to define the maximum number of RDMA operations per connection for the DAPL UD/RDMA mixed mode.