Intel® MPI Library Reference Manual for Linux* OS
-hostfile <hostfile> or -f <hostfile>
Use this option to specify host names on which to run the application. If a host name is repeated, this name is used only once.
See also the I_MPI_HYDRA_HOST_FILE environment variable for more details.
Use the -perhost, -ppn, -grr, and -rr options to change the process placement on the cluster nodes.
-machinefile <machine file> or -machine <machine file>
Use this option to control the process placement through the <machine file>. To define the total number of processes to start, use the -n option.
When you are pinning within a machine, the option -binding=map is available within the machine file for each line. For example:
$ cat ./machinefile node0:2 binding=map=0,3 node1:2 binding=map=[2,8] node0:1 binding=map=8
$ mpiexec.hydra -machinefile ./machinefile -n 5 -l numactl --show [4] policy: default [4] preferred node: current [4] physcpubind: 8 [4] cpubind: 0 [4] nodebind: 0 [4] membind: 0 1 [0] policy: default [0] preferred node: current [0] physcpubind: 0 [0] cpubind: 0 [0] nodebind: 0 [0] membind: 0 1 [1] policy: default [1] preferred node: current [1] physcpubind: 3 [1] cpubind: 1 [1] nodebind: 1 [1] membind: 0 1 [3] policy: default [3] preferred node: current [3] physcpubind: 3 [3] cpubind: 1 [3] nodebind: 1 [3] membind: 0 1 [2] policy: default [2] preferred node: current [2] physcpubind: 1 [2] cpubind: 1 [2] nodebind: 1 [2] membind: 0 1
Use this option to set the <ENVVAR> environment variable to the specified <value> for all MPI processes.
Use this option to enable propagation of all environment variables to all MPI processes.
Use this option to suppress propagation of any environment variables to any MPI processes.
-genvlist <list of genv var names>
Use this option to pass a list of environment variables with their current values. <list of genv var names> is a comma separated list of environment variables to be sent to all MPI processes.
Use this option to choose the caching mode of Process Management Interface* (PMI) message. Possible values for <mode> are:
nocache - do not cache PMI messages.
cache - cache PMI messages on the local pmi_proxy management processes to minimize PMI requests. Cached information is propagated to the child management processes.
lazy-cache - cache mode with on-request propagation of the PMI information.
alltoall - information is automatically exchanged between all pmi_proxy before any get request can be done.
The lazy-cache mode is the default mode.
See the I_MPI_HYDRA_PMI_CONNECT environment variable for more details.
-perhost <# of processes >, -ppn <# of processes >, or -grr <# of processes>
Use this option to place the indicated number of consecutive MPI processes on every host in the group using round robin scheduling. See the I_MPI_PERHOST environment variable for more details.
-rr
Use this option to place consecutive MPI processes on different hosts using the round robin scheduling. This option is equivalent to -perhost 1. See the I_MPI_PERHOST environment variable for more details.
(SDK only) -trace [<profiling_library>] or -t [<profiling_library>]
Use this option to profile your MPI application using the indicated <profiling_library>. If you do not specify <profiling_library>, the default profiling library libVT.so is used.
Set the I_MPI_JOB_TRACE_LIBS environment variable to override the default profiling library.
Use this option to collect statistics from your MPI application using internal Intel MPI statistics and additional collector, which can collect hardware counters, memory consumption, internal MPI imbalance and OpenMP* imbalance if the application runs with Intel OpenMP implementation. When you use this option, two text files are generated: stats.txt and app_stat.txt. The stats.txt file contains the Intel® MPI Library native statistics and the app_stat.txt file contains the statistics information for your application provided by MPI Performance Snapshot. These files can be analyzed by mps utility. By using the mps utility, you can simplify the analysis of the Intel MPI statistics.
For example, to collect the statistics, use the following command:
$ mpirun -mps –n 2 ./myApp
For more information, see the Native Statistics Format.
Before running an application with this option, set up the environment by using mpsvars.sh from the Intel® Trace Analyzer and Collector installation directory. For detailed description, see MPI Performance Snapshot for Linux* OS User's Guide at <installdir>/itac_latest/doc/MPI_Perf_Snapshot_User_Guide.pdf in Intel® Parallel Studio XE Cluster Edition.
If you have used the options -trace or -check_mpi in the command, the -mps option is ignored.
(SDK only) -check_mpi [<checking_library>]
Use this option to check your MPI application using the indicated <checking_library>. If you do not specify <checking_library>, the default checking library libVTmc.so is used.
Set the I_MPI_JOB_CHECK_LIBS environment variable to override the default checking library.
(SDK only) -trace-pt2pt
Use this option to collect the information about point-to-point operations.
(SDK only) -trace-collectives
Use this option to collect the information about collective operations.
Use the -trace-pt2pt and -trace-collectives to reduce the size of the resulting trace file or the number of message checker reports. These options work with both statically and dynamically linked applications.
Use this option to specify the file <filename> that contains the command-line options. Blank lines and lines that start with '#' as the first character are ignored.
Use this option to restrict the number of child management processes launched by the mpiexec.hydra command, or by each pmi_proxy management process.
See the I_MPI_HYDRA_BRANCH_COUNT environment variable for more details.
-pmi-aggregate or -pmi-noaggregate
Use this option to switch on or off, respectively, the aggregation of the PMI requests. The default value is -pmi-aggregate, which means the aggregation is enabled by default.
See the I_MPI_HYDRA_PMI_AGGREGATE environment variable for more details.
Use this option to run <executable> under the TotalView* debugger. For example:
$ mpiexec.hydra -tv -n <# of processes> <executable>
See Environment Variables for information on how to select the TotalView* executable file.
TotalView* uses rsh by default. If you want to use ssh, set the value of the TVDSVRLAUNCHCMD environment variable to ssh.
The TotalView* debugger can display message queue state of your MPI program. To enable this feature, do the following steps:
To display the internal state of the MPI library textually, select the Tools > Message Queue command. If you select the Process Window Tools > Message Queue Graph command, the TotalView* environment variable displays a window that shows a graph of the current message queue state. For more information, see the TotalView* environment variable.
Use this option to attach the TotalView* debugger to an existing Intel® MPI Library job. Use the mpiexec.hydra process id as <pid>. You can use the following command:
$ mpiexec.hydra -tva <pid>
Use this option to run <executable> under the GNU* debugger. You can use the following command:
$ mpiexe.hydra -gdb -n <# of processes> <executable>
Use this option to attach the GNU* debugger to the existing Intel® MPI Library job. You can use the following command:
$ mpiexec.hydra -gdba <pid>Use this option to launch tools such as Intel® VTune™ Amplifier XE, Valgrind*, and GNU* Debugger on specified ranks through the mpiexec.hydra command.
You can not use the -gdb option and -gtool option simultaneously except that you have not specified any of debugger to be launched with the -gtool option.
-gtool “<command line for a tool 1>:<ranks set 1>[=lanuch mode 1][@arch 1]; <command line for a tool 2>:<ranks set 2>[=exclusive][@arch 2]; … ;<command line for a tool n>:<ranks set n>[=exclusive][@arch n]” <executable>
Or
$ mpiexec.hydra -n <# of processes> -gtool “<command line for a tool 1>:<ranks set 1>[=launch mode 1][@arch 1]” -gtool “<command line for a tool 2>:<ranks set 2>[=launch mode 2][@arch 2]” … -gtool “<command line for a tool n>:<ranks set n>[=launch mode 3][@arch n]” <executable>
In the syntax, the separator “;” and the -gtool option can be replaced with each other.
<arg> |
Parameters |
<rank set> |
Specify the ranks range which are involved in the tool execution. Ranks are separated with a comma or “-” symbol can be used for a set of contiguous ranks. NoteIf you specify incorrect rank index, the corresponding warning is printed and a tool continues working for valid ranks. |
[=launch mode] |
Specify the launch mode. |
[@arch] |
Specify the architecture on which a tool applies. For a given <rank set>, if you specify this argument, the tool is only applied for those ranks which have been allocated on hosts with the specific architecture. This parameter is optional. For the values of[@arch], see the argument table of I_MPI_PLATFORM for the detail value descriptions. If you launch the debugger on Intel® Xeon Phi™ coprocessor, setting [@arch] is required. See examples for details. |
Rank sets cannot intersect; for example, the -gtool option with missed parameter or the same [@arch] parameter. However, you can specify the same rank sets with clearly specified different [@arch] parameters so that these two sets are not intersect. You should specify which tools to apply within a single mpiexec.hydra launch. It may happen that some tools cannot work at the same time or such usages may lead to incorrect results.
The following table shows the parameter values for [=launch mode]
Value |
Description |
exclusive |
Specify this value to prevent launching a tool for more than one rank per host. The value is optional. |
attach |
Specify this value to attach the tool from a -gtool option to an executable file. If you use debuggers or other tools which can attach to a process in a debugger manner, you need to specify this value in [=launch mode]. The current implementation has been tested for debuggers only. |
node-wide |
Specify this value to apply the tool from a -gtool option to a higher level than an executable file (to pmi_proxy daemon). This parameter will be implemented in future release. |
You can specify several values for each tool. In this case, separate the tools with a comma sign “,”. For example:
$ mpiexec.hydra -gtool "gdb-ia:all=attach,exclusive; /usr/bin/gdb:all=exclusive, attach@knc" -host <hostname> -n 2 <app> : -host <hostname-mic0> -n 2 <mic-app>
In this example, the Intel version of GNU* GDB (gdb-ia) and the native GNU* GDB for Intel® Xeon Phi™ coprocessor are launched only for one rank on a host as well as for one rank on a coprocessor.
The following command examples demonstrate different scenarios of using the -gtool option:
Launch Intel® VTune™ Amplifier XE and Valgrind* through the mpiexec.hydra command:
$ mpiexec.hydra -n 16 -gtool “amplxe-cl -collect advanced-hotspots -analyze-system -r result1:5,3,7-9=exclusive@nhm;valgrind -log-file=log_%p :0,1,10-12@wsm” a.out
Use this command to apply amplxe-cl to a rank with a minimal index allocated on the hosts in Intel® microarchitecture code name Nehalem from the given rank set. At the same time Valgrind* is applied for all ranks allocated on the hosts in Intel® microarchitecture code name Westmere from the specified rank set. Valgrind results are written to files with names log_<process ID>.
Launch GNU* Debugger (GDB*) through the mpiexec.hydra command:
$ mpiexec.hydra -n 16 -gtool “gdb:3,5,7-9=attach” a.out
Use this command to apply gdb to the given rank set.
Set up an environment variable for a specific rank set:
$ mpiexec.hydra -n 16 -gtool “env VARIABLE1=value1 VARIABLE2=value2:3,5,7-9; env VARIABLE3=value3:0,11” a.out
Use this command to set VARIABLE1 and VARIABLE2 for 3,5,7,8,9 ranks and establish VARIABLE3 for ranks with numbers 0 and 11.
Debug an application on selected ranks through the mpiexec.hydra command.
When configure the debugger through the mpiexec.hydra command, you can use the following options for corresponding debuggers. You can also launch any other debuggers by using the -gtool option.
standard GNU* Debugger (GDB*): gdb
Intel version of GNU* Debugger: gdb-ia
native GNU* Debugger for Intel® Xeon Phi™ coprocessor: gdb
The -gtool option can simultaneously support debuggers for the host with Intel® Xeon Phi™ coprocessor. In this case, you must specify @arch parameter for ranks allocated on Intel® Xeon Phi™ machines. Set @arch=knc in the command for architectures with Intel® Xeon Phi™ coprocessor.
When debugging on hosts with Intel® Xeon Phi™ Coprocessor or on heterogeneous clusters, you need to enable the Intel® Xeon Phi™ coprocessor recognition by setting the I_MPI_MIC environment variable. See I_MPI_MIC on how to specify this environment variable.
If you have specified @arch, you can specify rank sets with intersections.
If the debugging session has been launched on a binary compatible cluster, the parameter @arch can be omitted. The value generic comprises all platforms except for the platform with Intel® Xeon Phi™ Coprocessor.
If the debugging session has been launched on a heterogeneous cluster and the debugger for Intel® Xeon Phi™ coprocessor has not been specified, /usr/bin/gdb is launched on the coprocessor by default.
For example:
$ mpiexec.hydra -n 16 -gtool “gdb:3,5=attach;gdb-ia:7-9=attach” a.out
In this case the standard GNU* Debugger (gdb) is launched for ranks with numbers 3, 5, 7, 8, 9.
$ mpiexec.hydra -gtool “gdb-ia:all=attach@generic; /tmp/gdb:all=attach@knc” -host <hostname> -n 8 <host-app> : -host <hostname-mic0> -n 8 <mic-app>
In this case the Intel version of GNU* GDB (gdb-ia) is launched for all hostname ranks. While native GNU* GDB for Intel® Xeon Phi™ coprocessor is applied to all ranks allocated on a coprocessor. This example demonstrates the fact that rank sets can intersect if you have specified different architectures for those rank sets.
$ mpiexec.hydra -gtool “gdb-ia:3,5,7-9” -host <hostname> -n 8 <host-app> : -host <hostname-mic0> -n 8 <mic-app>
In this case, the Intel version of GNU* GDB (gdb-ia) is launched for all ranks from the rank set which are allocated on a hostname machine. Any rank from the rank set that is assigned to the coprocessor uses the native GNU* GDB for Intel® Xeon Phi™ coprocessor (/usr/bin/gdb) automatically.
Apply a tool for a certain rank through the <machine file> option.
In this example, suppose the m_file has the following contents:
hostname_1:2
hostname_2:3
hostname_3:1
The following command line demonstrates how to use the -machinefile option to apply a tool:
$ mpiexec.hydra -n 6 -machinefile m_file -gtool “amplxe-cl -collect advanced-hotspots -analyze-system -r result1:5,3=exclusive@nhm;valgrind:0,1@wsm” a.out
In this example, the use of -machinefie option means that ranks with indices 0 and 1 is allocated on hostname_1 machine; rank with index 3 is allocated on hostname_2 machine and rank number 5 - on hostname_3 machine. After that, amplxe-cl is applied only ranks 3 and 5 (since these ranks belong to different machines, exclusive option passes both of them) in case if hostname_2 and hostname_3 machines have Intel® microarchitecture code name Nehalem architecture. At the same time the Valgrind* tool is applied to both ranks allocated on hostname_1 machine in case if it has Intel® microarchitecture code name Westmere architecture.
Show how the ranks are distributed across the cluster nodes. Use the z show map command with -gtool option. To see the full function of z commands, use the z help command.
$ mpiexec.hydra -gtool "gdb-ia:0,2,3,9,10=attach;/tmp/gdb:5,6=attach@knc" -host <hostname_1> -n 4 <host-app> : -host <hostname_1-mic0> -n 4 <mic-app> : -host <hostname_2> -n 4 <host-app>
[0,2,3,5,6,9,10] (mpigdb) z show map
[0,2,3]: hostname_1
[5,6]: hostname_1-mic0
[9,10]: hostname_2
[0,2,3,5,6,9,10] (mpigdb) z help
z <positive number(s) up to 11 or all> - Sets ranks for debugging
z show map - Shows ranks distribution across the cluster nodes
z help - Shows help information
[0,2,3,5,6,9,10] (mpigdb)
-gtoolfile <gtool_config_file>
Use this option to launch tools such as Intel® VTune™ Amplifier XE, Valgrind*, and GNU* Debugger on specified ranks through the mpiexec.hydra command.
If gtool_config_file contains the following settings:
env VARIABLE1=value1 VARIABLE2=value2:3,5,7-9; env VARIABLE3=value3:0,11
env VARIABLE4=value4:1,12
then the following command sets VARIABLE1 and VARIABLE2 for 3,5,7,8,9 ranks and establishes VARIABLE3 for ranks with numbers 0 and 11, while VARIABLE4 is stated for the first and the twelfth ranks.
$ mpiexec.hydra -n 16 -gtoolfile gtool_config_file a.out
The options and the environment variable -gtool, -gtoolfile and I_MPI_GTOOL are mutually exclusive. The options -gtool and -gtoolfile are of the same priority. The first specified option in a command line is effective and the second specified option is ignored. Both -gtool and -gtoolfile options have higher priority than the I_MPI_GTOOL environment variable. Thus, use the I_MPI_GTOOL environment variable if you have not specified neither -gtool or -gtoolfile options in the mpiexec.hydra command.
Use this option to avoid running the <executable> on the host where the mpiexec.hydra is launched. You can use this option on clusters that deploy a dedicated master node for starting the MPI jobs and a set of dedicated compute nodes for running the actual MPI processes.
Use this option to specify a particular <nodelist> on which to run the MPI processes. For example, the following command runs the executable a.out on hosts host1 and host2:
$ mpiexec.hydra -n 2 -ppn 1 -hosts host1,host2 ./a.out
If <nodelist> consists of only one node, this option is interpreted as a local option. See Local Options for details.
Use this option to choose the appropriate network interface. For example, if the IP emulation of your InfiniBand* network is configured to ib0, you can use the following command.
$ mpiexec.hydra -n 2 -iface ib0 ./a.out
See theI_MPI_HYDRA_IFACE environment variable for more details.
Use this option to set the polling mode for multiple I/O. The default is poll.
<spec> |
Define the polling mode for multiple I/O |
poll |
Set poll as the polling mode. This is the default value. |
select |
Set select as the polling mode. |
See the I_MPI_HYDRA_DEMUX environment variable for more details.
Use this option to control the Xlib* traffic forwarding. The default value is -disable-x, which means the Xlib traffic is not forwarded.
Use this option to insert the MPI process rank at the beginning of all lines written to the standard output.
where:
<arg> = {<dir_name>, <configuration_file>}.
Use this option to optimize the Intel® MPI Library performance by using the data collected by the mpitune utility.
Use the mpitune utility to collect the performance tuning data before using this option.
If <arg> is not specified, the best-fit tune options are selected for the given configuration. The default location of the configuration file is <installdir>/<arch>/etc directory.
To specify a different location for the configuration file, set <arg>=<dir_name> .
To specify a different configuration file, set <arg>=<configuration_file> .
-ilp64
Use this option to preload the ILP64 interface. See ILP64 Support for more details.
Use this option to direct standard input to the specified MPI processes.
<spec> |
Define MPI process ranks |
all |
Use all processes |
<l>,<m>,<n> |
Specify an exact list and use processes <l>, <m> and <n> only. The default value is zero |
<k>,<l>-<m>,<n> |
Specify a range and use processes <k>, <l> through <m>, and <n> |
Use this option to disable processing of the mpiexec.hydra configuration files described in Configuration Files.
Use this option to avoid intermingling of data output from the MPI processes. This option affects both the standard output and the standard error streams.
When using this option, end the last output line of each process with the end-of-line (\n) character. Otherwise the application may stop responding.
Use this option to specify the path to the <executable> file.
Use this option to create a temporary file containing information about the launched processes. The file name is mpiexec_${username}_$PPID.log, where PPID is a parent process PID. This file is created in the temporary directory selected by the -tmpdir option. This file is used by the mpicleanup utility. If a job terminates successfully, the mpiexec.hydra command automatically removes this file.
See the I_MPI_HYDRA_CLEANUP environment variable for more details.
Use this option to set a directory for temporary files.
See the I_MPI_TMPDIR environment variable for more details.
Use this option to display the version of the Intel® MPI Library.
-info
Use this option to display build information of the Intel® MPI Library. When this option is used, the other command line arguments are ignored.
Use this option to performs rank placement on the assumption of transferred data from the stats file and cluster topology. You can use the following command:
$ mpiexec.hydra –use-app-topology ./stats.txt <…> ./my_app
<value> |
The path to the native Intel MPI statistics file level 1 or higher |
The hydra PM uses API of libmpitune.so the same way as mpitune_rank_placement in the static method and uses the resulting host list for rank assignment.
See I_MPI_HYDRA_USE_APP_TOPOLOGY and Topology Awareness Application Tuning for more details.
Use this option to explicitly specify the local host name for the launching node.
Example:
$ mpiexec.hydra -localhost <localhost_ip> -machinefile <file> -n 2 ./a.out