Intel® MPI Library Reference Manual for Linux* OS
-hostfile <hostfile> or -f <hostfile>
Use this option to specify host names on which to run the application. If a host name is repeated, this name is used only once.
See also the I_MPI_HYDRA_HOST_FILE environment variable for more details.
Note:
Use the -perhost, -ppn, -grr, and -rr options to change the process placement on the cluster nodes.
-machinefile <machine file> or -machine <machine file>
Use this option to control the process placement through the <machine file>. The total number of processes to start is defined by the -n option.
When you are pinning within a machine, the option -binding=map is available within the machine file for each line. For example:
$ cat ./machinefile
node0:2 binding=map=0,3
node1:2 binding=map=[2,8]
node0:1 binding=map=8
$ mpiexec.hydra -machinefile ./machinefile -n 5 -l numactl --show
[4] policy: default
[4] preferred node: current
[4] physcpubind: 8
[4] cpubind: 0
[4] nodebind: 0
[4] membind: 0 1
[0] policy: default
[0] preferred node: current
[0] physcpubind: 0
[0] cpubind: 0
[0] nodebind: 0
[0] membind: 0 1
[1] policy: default
[1] preferred node: current
[1] physcpubind: 3
[1] cpubind: 1
[1] nodebind: 1
[1] membind: 0 1
[3] policy: default
[3] preferred node: current
[3] physcpubind: 3
[3] cpubind: 1
[3] nodebind: 1
[3] membind: 0 1
[2] policy: default
[2] preferred node: current
[2] physcpubind: 1
[2] cpubind: 1
[2] nodebind: 1
[2] membind: 0 1
Use this option to set the <ENVVAR> environment variable to the specified <value> for all MPI processes.
Use this option to enable propagation of all environment variables to all MPI processes.
Use this option to suppress propagation of any environment variables to any MPI processes.
-genvlist <list of genv var names>
Use this option to pass a list of environment variables with their current values. <list of genv var names> is a comma separated list of environment variables to be sent to all MPI processes.
Use this option to choose the Process Management Interface* (PMI) message caching mode. Possible values for <mode> are:
nocache - do not cache PMI messages.
cache - cache PMI messages on the local pmi_proxy management processes to minimize PMI requests. Cached information is propagated to the child management processes.
lazy-cache - cache mode with on-request propagation of the PMI information.
alltoall - information is automatically exchanged between all pmi_proxy before any get request can be done.
The lazy-cache mode is the default mode.
See the I_MPI_HYDRA_PMI_CONNECT environment variable for more details.
-perhost <# of processes >, -ppn <# of processes >, or -grr <# of processes>
Use this option to place the indicated number of consecutive MPI processes on every host in the group using round robin scheduling. See the I_MPI_PERHOST environment variable for more details.
-rr
Use this option to place consecutive MPI processes on different hosts using the round robin scheduling. This option is equivalent to -perhost 1. See the I_MPI_PERHOST environment variable for more details.
(SDK only) -trace [<profiling_library>] or -t [<profiling_library>]
Use this option to profile your MPI application using the indicated <profiling_library>. If the <profiling_library> is not mentioned, the default profiling library libVT.so is used.
Set the I_MPI_JOB_TRACE_LIBS environment variable to override the default profiling library.
(SDK only) -check_mpi [<checking_library>]
Use this option to check your MPI application using the indicated <checking_library>. If <checking_library> is not mentioned, the default checking library libVTmc.so is used.
Set the I_MPI_JOB_CHECK_LIBS environment variable to override the default checking library.
(SDK only) –trace-pt2pt
Use this option to collect the information about point-to-point operations.
(SDK only) –trace-collectives
Use this option to collect the information about collective operations.
Note:
Use the –trace-pt2pt and –trace-collectives to reduce the size of the resulting trace file or the number of message checker reports. These options work with both statically and dynamically linked applications.
Use this option to specify the file <filename> that contains the command-line options. Blank lines and lines that start with '#' as the first character are ignored.
Use this option to restrict the number of child management processes launched by the mpiexec.hydra command, or by each pmi_proxy management process.
See the I_MPI_HYDRA_BRANCH_COUNT environment variable for more details.
-pmi-aggregate or -pmi-noaggregate
Use this option to switch on or off, respectively, the aggregation of the PMI requests. The default value is -pmi-aggregate, which means the aggregation is enabled by default.
See the I_MPI_HYDRA_PMI_AGGREGATE environment variable for more details.
Use this option to run <executable> under the TotalView* debugger. For example:
$ mpiexec.hydra -tv -n <# of processes><executable>
See Environment Variables for information on how to select the TotalView* executable file.
Note:
TotalView* uses rsh by default. If you want to use ssh, set the value of the TVDSVRLAUNCHCMD environment variable to ssh.
Note:
The TotalView* debugger can display message queue state of your MPI program. To enable this feature, do the following steps:
Run your <executable> with the -tv option.
$ mpiexec.hydra -tv -n <# of processes> <executable>
To display the internal state of the MPI library textually, select the Tools > Message Queue command. If you select the Process Window Tools > Message Queue Graph command, the TotalView* environment variable displays a window that shows a graph of the current message queue state. For more information, see the TotalView* environment variable.
Use this option to attach the TotalView* debugger to an existing Intel® MPI job. Use the mpiexec.hydra process id as <pid>. For example:
$ mpiexec.hydra -tva <pid>
Use this option to run <executable> under the GNU* debugger. For example:
$ mpiexe.hydra -gdb -n <# of processes> <executable>
Use this option to attach the GNU* debugger to the existing Intel® MPI job. For example:
$ mpiexec.hydra -gdba <pid>Use this option to launch tools such as Intel® VTune™ Amplifier XE, Valgrind*, and GNU* Debugger on specified ranks through the mpiexec.hydra command.
Note:
You can not use the –gdb option and –gtool option simultaneously except that you have not specified any of debugger to be launched with the –gtool option.
Syntax
-gtool “<command line for a tool 1>:<ranks set 1>[=lanuch mode 1][@arch 1]; <command line for a tool 2>:<ranks set 2>[=exclusive][@arch 2]; … ;<command line for a tool n>:<ranks set n>[=exclusive][@arch n]” <executable>
Or
$ mpiexec.hydra –n <# of processes> -gtool “<command line for a tool 1>:<ranks set 1>[=launch mode 1][@arch 1]” -gtool “<command line for a tool 2>:<ranks set 2>[=launch mode 2][@arch 2]” … -gtool “<command line for a tool n>:<ranks set n>[=launch mode 3][@arch n]” <executable>
Note:
In the syntax, the separator “;” and the –gtool option can be replaced with each other.
Arguments
<arg> |
Parameters |
<rank set> |
Specify the ranks range which will be involved in the tool execution. Ranks are separated with a comma or “-” symbol can be used for a set of contiguous ranks. Note: If you specify incorrect rank index, the corresponding warning is printed and a tool continues working for valid ranks. |
[=launch mode] |
Specify the launch mode. |
[@arch] |
Specify the architecture on which a tool will apply. That means that for a given <rank set>, the tool is only applied for those ranks which have been allocated on hosts with the specific architecture. This parameter is optional. For the values of[@arch], see the argument table of I_MPI_PLATFORM for the detail value descriptions. If you launch the debugger on Intel® Xeon Phi™ coprocessor, setting [@arch] is required. See examples for details. |
Note:
Rank sets cannot intersect; for example, the –gtool option with missed parameter or the same [@arch] parameter. However, you can specify the same rank sets with clearly specified different [@arch] parameters so that these two sets will not intersect. You should specify which tools to apply within a single mpiexec.hydra launch. It may happen that some tools cannot work at the same time or such usages my lead to incorrect results.
The following table shows the parameter values for [=launch mode]
Arguments
Value |
Description |
exclusive |
Specify this value to prevent launching a tool for more than one rank per host. The value is optional. |
attach |
Specify this value to attach the tool from a –gtool option to an executable file. This value of [=launch mode] parameter is required for debuggers or other tools which can attach to a process in a debugger manner. The current implementation has been tested for debuggers only. |
node-wide |
Specify this value to apply the tool from a –gtool option to a higher level than an executable file (to pmi_proxy daemon). This parameter will be implemented in future release. |
You can specify several values for each tool. In this case, separate the tools with a comma sign “,”. For example:
$ mpiexec.hydra -gtool "gdb-ia:all=attach,exclusive; /usr/bin/gdb:all=exclusive, attach@knc" -host <hostname> -n 2 <app> : -host <hostname-mic0> -n 2 <mic-app>
In this example, the Intel version of GNU* GDB (gdb-ia) and the native GNU* GDB for Intel® Xeon Phi™ coprocessor will be launched only for one rank on a host as well as for one rank on a coprocessor.
Examples:
The following command examples demonstrate different scenarios of using the –gtool option:
Launch Intel® VTune™ Amplifier XE and Valgrind* through the mpiexec.hydra command:
$ mpiexec.hydra –n 16 -gtool “amplxe-cl –collect advanced-hotspots –analyze-system –r result1:5,3,7-9=exclusive@nhm;valgrind –log-file=log_%p :0,1,10-12@wsm” a.out
Use this command to apply amplxe-cl to a rank with a minimal index allocated on the hosts in Intel® microarchitecture code name Nehalem from the given rank set. At the same time Valgrind* is applied for all ranks allocated on the hosts in Intel® microarchitecture code name Westmere from the specified rank set. Valgrind results are written to files with names log_<process ID>.
Launch GNU* Debugger (GDB*) through the mpiexec.hydra command:
$ mpiexec.hydra –n 16 -gtool “gdb:3,5,7-9” a.out
Use this command to apply gdb to the given rank set.
Set up an environment variable for a specific rank set:
$ mpiexec.hydra –n 16 -gtool “env VARIABLE1=value1 VARIABLE2=value2:3,5,7-9; env VARIABLE3=value3:0,11” a.out
Use this command to set VARIABLE1 and VARIABLE2 for 3,5,7,8,9 ranks and establish VARIABLE3 for ranks with numbers 0 and 11.
Debug an application on selected ranks through the mpiexec.hydra command.
When configure the debugger through the mpiexec.hydra command, you can use the following options for corresponding debuggers. You can also launch any other debuggers by using the -gtool option.
standard GNU* Debugger (GDB*): gdb
Intel version of GNU* Debugger: gdb-ia
native GNU* Debugger for Intel® Xeon Phi™ coprocessor: gdb
The -gtool option can simultaneously support debuggers for the host with Intel® Xeon Phi™ coprocessor. In this case, you must specify @arch parameter for ranks allocated on Intel® Xeon Phi™ machines. Set @arch=knc in the command for architectures with Intel® Xeon Phi™ coprocessor.
Note:
When debugging on hosts with Intel® Xeon Phi™ Coprocessor or on heterogeneous clusters, you need to enable the Intel® Xeon Phi™ coprocessor recognition by setting the I_MPI_MIC environment variable. See I_MPI_MIC on how to specify this environment variable.
If you have specified @arch, you can specify rank sets with intersections.
If the debugging session has been launched on a binary compatible cluster, the parameter @arch can be omitted. The value generic comprises all platforms except for the platform with Intel® Xeon Phi™ Coprocessor.
If the debugging session has been launched on a heterogeneous cluster and the debugger for Intel® Xeon Phi™ coprocessor has not been specified, /usr/bin/gdb will be launched on the coprocessor by default.
For example:
$ mpiexec.hydra –n 16 -gtool “gdb:3,5=attach;gdb-ia:7-9=attach” a.out
In this case the standard GNU* Debugger (gdb) is launched for ranks with numbers 3, 5, 7, 8, 9.
$ mpiexec.hydra -gtool “gdb-ia:all=attach@generic; /tmp/gdb:all=attach@knc” –host <hostname> -n 8 <host-app> : -host <hostname-mic0> -n 8 <mic-app>
In this case the Intel version of GNU* GDB (gdb-ia) is launched for all hostname ranks. While native GNU* GDB for Intel® Xeon Phi™ coprocessor is applied to all ranks allocated on a coprocessor. This example demonstrates the fact that rank sets can intersect if you have specified different architectures for those rank sets.
$ mpiexec.hydra -gtool “gdb-ia:3,5,7-9” –host <hostname> -n 8 <host-app> : -host <hostname-mic0> -n 8 <mic-app>
In this case, the Intel version of GNU* GDB (gdb-ia) is launched for all ranks from the rank set which are allocated on a hostname machine. Any rank from the rank set that is assigned to the coprocessor uses the native GNU* GDB for Intel® Xeon Phi™ coprocessor (/usr/bin/gdb) automatically.
Apply a tool for a certain rank through the <machine file> option.
In this example, suppose the m_file has the following contents:
hostname_1:2
hostname_2:3
hostname_3:1
The following command line demonstrates how to use the –machinefile option to apply a tool:
$ mpiexec.hydra –n 6 –machinefile m_file -gtool “amplxe-cl –collect advanced-hotspots –analyze-system –r result1:5,3=exclusive@nhm;valgrind:0,1@wsm” a.out
In this example, the use of –machinefie option means that ranks with indices 0 and 1 is allocated on hostname_1 machine; rank with index 3 is allocated on hostname_2 machine and rank number 5 – on hostname_3 machine. After that, amplxe-cl is applied only ranks 3 and 5 (since these ranks belong to different machines, exclusive option passes both of them) in case if hostname_2 and hostname_3 machines have Intel® microarchitecture code name Nehalem architecture. At the same time the Valgrind* tool is applied to both ranks allocated on hostname_1 machine in case if it has Intel® microarchitecture code name Westmere architecture.
Show how the ranks are distributed across the cluster nodes. Use the z show map command with –gtool option. To see the full function of z commands, use the z help command.
mpiexec.hydra -gtool "gdb-ia:0,2,3,9,10=attach;/tmp/gdb:5,6=attach@knc" –host <hostname_1> -n 4 <host-app> : –host <hostname_1-mic0> -n 4 <mic-app> : –host <hostname_2> -n 4 <host-app>
[0,2,3,5,6,9,10] (mpigdb) z show map
[0,2,3]: hostname_1
[5,6]: hostname_1-mic0
[9,10]: hostname_2
[0,2,3,5,6,9,10] (mpigdb) z help
z <positive number(s) up to 11 or all> - Sets ranks for debugging
z show map - Shows ranks distribution across the cluster nodes
z help - Shows help information
[0,2,3,5,6,9,10] (mpigdb)
Use this option to launch tools such as Intel® VTune™ Amplifier XE, Valgrind*, and GNU* Debugger on specified ranks through the mpiexec.hydra command.
Example:
If gtool_config_file contains the following settings:
env VARIABLE1=value1 VARIABLE2=value2:3,5,7-9; env VARIABLE3=value3:0,11
env VARIABLE4=value4:1,12
then the following command sets VARIABLE1 and VARIABLE2 for 3,5,7,8,9 ranks and establishes VARIABLE3 for ranks with numbers 0 and 11, while VARIABLE4 is stated for the first and the twelfth ranks.
$ mpiexec.hydra –n 16 -gtoolfile gtool_config_file a.out
Note:
The options and the environment variable –gtool, -gtoolfile and I_MPI_GTOOL are mutually exclusive. The options –gtool and –gtoolfile are of the same priority. The first specified option in a command line is effective and the second specified option is ignored. Both –gtool and -gtoolfile options have higher priority than the I_MPI_GTOOL environment variable. Thus, use the I_MPI_GTOOL environment variable if you have not specified neither –gtool or –gtoolfile options in the mpiexec.hydra command.
Use this option to avoid running the <executable> on the host where the mpiexec.hydra is launched. You can use this option on clusters that deploy a dedicated master node for starting the MPI jobs and a set of dedicated compute nodes for running the actual MPI processes.
Use this option to specify a particular <nodelist> on which to run the MPI processes. For example, the following command runs the executable a.out on hosts host1 and host2:
$ mpiexec.hydra -n 2 -hosts host1,host2 ./a.out
Note:
If <nodelist> consists of only one node, this option is interpreted as a local option. See Local Options for details.
Use this option to choose the appropriate network interface. For example, if the IP emulation of your InfiniBand* network is configured to ib0, you can use the following command.
$ mpiexec.hydra -n 2 -iface ib0 ./a.out
See the I_MPI_HYDRA_IFACE environment variable for more details.
Use this option to set polling mode for multiple I/O. The default is poll.
Arguments
<spec> |
Define the polling mode for multiple I/O |
poll |
Set poll as the polling mode. This is the default value. |
select |
Set select as the polling mode. |
See the I_MPI_HYDRA_DEMUX environment variable for more details.
Use this option to control the Xlib* traffic forwarding. The default value is -disable-x, which means the Xlib traffic will not be forwarded.
Use this option to insert the MPI process rank at the beginning of all lines written to the standard output.
where:
<arg> = {<dir_name>, <configuration_file>}.
Use this option to optimize the Intel® MPI Library performance by using the data collected by the mpitune utility.
Note:
Use the mpitune utility to collect the performance tuning data before using this option.
If <arg> is not specified, the best-fit tune options are selected for the given configuration. The default location of the configuration file is <installdir>/<arch>/etc directory.
To specify a different location for the configuration file, set <arg>=<dir_name> .
To specify a different configuration file, set <arg>=<configuration_file> .
-ilp64
Use this option to preload the ILP64 interface. See ILP64 Support for more details.,
Use this option to direct standard input to the specified MPI processes.
Arguments
<spec> |
Define MPI process ranks |
all |
Use all processes |
<l>,<m>,<n> |
Specify an exact list and use processes <l>, <m> and <n> only. The default value is zero |
<k>,<l>-<m>,<n> |
Specify a range and use processes <k>, <l> through <m>, and <n> |
Use this option to disable processing of the mpiexec.hydra configuration files described in Configuration Files.
Use this option to avoid intermingling of data output from the MPI processes. This option affects both the standard output and the standard error streams.
Note:
When using this option, end the last output line of each process with the end-of-line (\n) character. Otherwise the application may stop responding.
Use this option to specify the path to the <executable> file.
Use this option to create a temporary file containing information about the launched processes. The file name is mpiexec_${username}_$PPID.log, where PPID is a parent process PID. This file is created in the temporary directory selected by the -tmpdir option. This file is used by the mpicleanup utility. If a job terminates successfully, the mpiexec.hydra command automatically removes this file.
See the I_MPI_HYDRA_CLEANUP environment variable for more details.
Use this option to set a directory for temporary files.See the I_MPI_TMPDIR environment variable for more details.
Use this option to display the version of the Intel® MPI Library.
-info
Use this option to display build information of the Intel® MPI Library. When this option is used, the other command line arguments are ignored.
Use this option to select a built-in bootstrap server to use. A bootstrap server is the basic remote node access mechanism that is provided by the system. Hydra supports multiple runtime bootstrap servers such as ssh, rsh, pdsh, fork, persist, slurm, ll, lsf, sge, or jmi to launch the MPI processes. The default bootstrap server is ssh. By selecting slurm, ll, lsf, or sge, you use the corresponding srun, llspawn.stdio, blaunch, or qrsh internal job scheduler utility to launch service processes under the respective selected job scheduler (SLURM*, LoadLeveler*, LSF*, and SGE*).
Arguments
<arg> |
String parameter |
ssh |
Use secure shell. This is the default value |
rsh |
Use remote shell |
pdsh |
Use parallel distributed shell |
fork |
Use fork call |
persist |
Use Hydra persist server |
slurm |
Use SLURM* srun command |
ll |
Use LoadLeveler* llspawn.stdio command |
lsf |
Use LSF blaunch command |
sge |
Use Oracle Grid Engine* qrsh command |
jmi |
Use Job Manager Interface (tighter integration) |
To enable tighter integration with the SLURM* or PBS Pro* job manager, use the jmi bootstrap server. Tighter integration includes registration of the process identifiers by the respective job managers. This configuration enables better resource accounting by the respective job manager, and better node cleanup upon job termination.
See the -bootstrap jmi description and the I_MPI_HYDRA_BOOTSTRAP environment variable for details.
-bootstrap-exec <bootstrap server>
Use this option to set the executable to be used as a bootstrap server. The default bootstrap server is ssh. For example:
$ mpiexec.hydra -bootstrap-exec <bootstrap_server_executable> \
-f hosts.file -env <VAR1> <VAL1> -n 2 ./a.out
See the I_MPI_HYDRA_BOOTSTRAP environment variable for more details.
-bootstrap persist
Use this option to launch MPI processes using Hydra persist server. Before running a job, start these servers on each host:
$ hydra_persist&
Note:
Do not start the services under the root account. A server can be shutdown using the Linux shell kill command.
Use this option to enable tight integration with the SLURM* or PBS Pro* job schedulers. Tighter integration is implemented using a particular job scheduler application programming interface or utility. If you specify this option, the default libjmi.so library is loaded. You can overwrite the default library name through the I_MPI_HYDRA_JMI_LIBRARY environment variable.
See the I_MPI_HYDRA_JMI_LIBRARY environment variable for more details.
Use this option to pin or bind MPI processes to a particular processor and avoid undesired process migration. In the following syntax, the quotes may be omitted for a one-member list. Each parameter corresponds to a single pinning property.
This option is supported on both Intel® and non-Intel microprocessors, but it may perform additional optimizations for Intel microprocessors than it performs for non-Intel microprocessors.
Syntax
-binding "<parameter>=<value>[;<parameter>=<value> ...]"
Parameters
pin |
Pinning switch |
enable | yes | on | 1 |
Turn on the pinning property. This is the default value |
disable | no | off | 0 |
Turn off the pinning property |
cell |
Pinning resolution |
unit |
Basic processor unit (logical CPU) |
core |
Processor core in multi-core system |
map |
Process mapping |
spread |
The processes are mapped consecutively to separate processor cells. Thus, the processes do not share the common resources of the adjacent cells. |
scatter |
The processes are mapped to separate processor cells. Adjacent processes are mapped upon the cells that are the most remote in the multi-core topology. |
bunch |
The processes are mapped to separate processor cells by #processes/#sockets processes per socket. Each socket processor portion is a set of the cells that are the closest in the multi-core topology. |
p0,p1,...,pn |
The processes are mapped upon the separate processors according to the processor specification on the p0,p1,...,pn list: theith process is mapped upon the processor pi, where pi takes one of the following values:
|
[m0,m1,...,mn] |
The ith process is mapped upon the processor subset defined by mi hexadecimal mask using the following rule: The jth processor is included into the subset mi if the jth bit of mi equals 1. |
domain |
Processor domain set on a node |
cell |
Each domain of the set is a single processor cell (unit or core). |
core |
Each domain of the set consists of the processor cells that share a particular core. |
cache1 |
Each domain of the set consists of the processor cells that share a particular level 1 cache. |
cache2 |
Each domain of the set consists of the processor cells that share a particular level 2 cache. |
cache3 |
Each domain of the set consists of the processor cells that share a particular level 3 cache. |
cache |
The set elements of which are the largest domains among cache1, cache2, and cache3 |
socket |
Each domain of the set consists of the processor cells that are located on a particular socket. |
node |
All processor cells on a node are arranged into a single domain. |
<size>[:<layout>] |
Each domain of the set consists of <size> processor cells. <size> may have the following values:
Note: Domain size is limited by the number of processor cores on the node. Each member location inside the domain is defined by the optional <layout> parameter value:
If <layout> parameter is omitted, compact is assumed as the value of <layout> |
order |
Linear ordering of the domains |
compact |
Order the domain set so that adjacent domains are the closest in the multi-core topology |
scatter |
Order the domain set so that adjacent domains are the most remote in the multi-core topology |
range |
Order the domain set according to the BIOS processor numbering |
offset |
Domain list offset |
<n> |
Integer number of the starting domain among the linear ordered domains. This domain gets number zero. The numbers of other domains will be cyclically shifted. |