|
LAM / MPI Parallel Computing |
|
This file contains the release notes for LAM/MPI version 6.5.3. Here's a brief table of contents (### indicates updates for this version):
Installation instructionsPlease see the installation guide for instructions on installing LAM 6.5.3 in the INSTALL file.
Supported systemsLAM 6.5.3 has been tested on the following systems:
Even if your operating system is not listed here, LAM will likely compile/run properly. The source code is fairly portable -- it should work properly under most POSIX-like systems. We request that you also download the "lamtest" package from the
LAM web site and run it on your system. If any errors are
encountered, please send us all the output (see the README file in
that package).
LAM is fully MPI-1 complaint with the exception of MPI_CANCEL.
MPI_CANCEL works properly for receives, but will almost never work on
sends. MPI_CANCEL is most frequently used with unmatched MPI_IRECV's
that were made "in case" a matching message arrived. This simply
entails removing the receive request from the local queue, and is
fairly straightforward to implement.
Actually canceling a send operation is much more difficult because
some meta information about a message is usually sent immediately. As
such, the message is usually at least partially sent before an
MPI_CANCEL is issued. Trying to chase down all the particular cases
is a nightmare, to say the least.
As such, the LAM Team decided not to implement MPI_CANCEL on
sends, and instead concentrate on other features.
But in true MPI Forum tradition, we would be happy to discuss any
code that someone would like to submit that fully implements
MPI_CANCEL. :-)
LAM provides full source code backward compatibility with previous
versions of LAM. Old applications that compiled properly with older
versions of LAM can simply be recompiled with this version of LAM.
Binary compatibility, however, is not provided -- applications
that have been compiled with previous versions of LAM will need to be
recompiled in order to run properly with this version of LAM. If
applications are not re-compiled with this LAM, their behavior will be
unpredictable (read: they just won't work).
Specifically -- users need to ensure that they use the new
versions of all the LAM commands in order to run properly. For
example, mixing LAM 6.3.2 and 6.5.3 commands (e.g.,
running the "wrong" version of lamd) will result in unpredictable
behavior.
LAM is frequently used on Linux-based machines (iX86 and
otherwise). It works correctly under 2.0.36 (we didn't test under
2.0.37, but we have no reason to believe that it wouldn't work under
that version as well, since it is really only minor changes from
2.0.36) and 2.2.x. We haven't explicitly tested under 2.4, but we
don't expect any problems with it.
However, versions 2.2.0 through 2.2.9 had some TCP/IP performance
problems. It seems that version 2.2.10 fixed these problems; if you
are using a Linux version between 2.2.0 and 2.2.9, LAM may exhibit
poor C2C performance due to the Linux TCP/IP kernel bugs. We
recommend that you upgrade to 2.2.10 (or the latest version).
See http://www.lam-mpi.org/linux/
for a full discussion of the problem.
LAM 6.5
LAM 6.3 series
LAM 6.2 series
LAM is now "somewhat aware" of the Portable Batch System (PBS). That
is, the following problems have been solved:
This "multiple LAM" behavior is *only* exhibited under a batch
environment (i.e., PBS). Other batch systems can easily be supported
-- let the LAM Team know if you'd like to see support for others
included. Additionally, manually setting the environment variable
LAM_MPI_SOCKET_SUFFIX on the node where lamboot is run achieves the
same ends.
One major feature that is still lacking with respect to PBS is to
use PBS's implementation of the Task Management (TM) extensions. Once
LAM uses the TM extensions:
The following LAM binaries have had their help messages greatly
expanded:
The messages should be much more helpful in trying to diagnose
problems, especially for first-time users. The help messages
generally try to identify the problem and suggest solutions. It is
possible for multiple cascading error messages to be printed; one
failure may cause other failures. As such, the first error message is
generally (but not always) the most relevant message -- solving that
error may solve the rest.
Additionally, much more information is now output when the "-d" switch
is used on all of these program (which enables debugging output).
The help messages are all contained in a single ASCII file which is
initially installed into the following file (where $prefix is the
option supplied to --prefix in the ./configure script):
The format of the file is simple; simple delimiter lines separate help
topic messages. It should be quite obvious which message corresponds
to which program/topic name combination.
This file allows system administrators to customize help messages for
their users according to the local environment. When LAM tries to
find the helpfile to print out a help message, it actually searches
for the file in the following order:
This seemingly-over complicated scheme will allow for maximum
flexibility for system administrators and/or users to define the
location of customized help files.
LAM has several structures that are used in many different
situations. One example is LAM's internal "struct nmsg" -- one of the
underlying message constructs used to pass data between LAM entities.
But since the "struct nmsg" is used in so many places, it is a
generalized structure and contains fields that are not used in every
situation.
By default, LAM only initializes relevant struct members before
using a structure. Using a structure may involve sending the entire
structure (including uninitialized members) to a remote host. This is
not a problem, because the remote host will also ignore the irrelevant
struct members (depending on the specific function being invoked).
More to the point -- LAM was designed this way to avoid setting
variables that will not be used; this is a slight optimization in
run-time performance.
Memory-checking debuggers are quite popular (such as purify and
the Solaris Workshop/Forte bcheck program), and quite useful to find
memory leaks, indexing past the end of arrays, and other types of
Heisenbugs. Since LAM "uses" uninitialized memory, it tends to
generate many warnings with these types of debuggers.
The --with-purify option has been added to the ./configure script
that will force LAM to zero out *all* memory before it is used. This
will eliminate the "read from uninitialized" types of warnings that
memory-checking debuggers will identify deep inside LAM. However,
this option invokes a slight overhead penalty in the run-time
performance of LAM, so it is not the default.
In prior versions of LAM/MPI, several additional Trillium
executables were built during the default build (Trillium is the
built-in lower layer of the LAM/MPI run-time system). Additionally,
several Trillium-specific header files and man pages were installed by
default.
These executables, header files, and man pages are actually
intended for LAM/Trillium developers, and are not necessary for normal
MPI use. That is -- these executables, header files, and man pages
are not necessary to develop, compile, and run normal MPI programs
under LAM/MPI. Third party packages may require the Trillium header
files and/or executables, however. For example, compiling XMPI will
require that the Trillium header files were previously installed.
As such, these executables have been removed from the default
build. Correspondingly, the header files and man pages have been
removed from the default install. There is an option to configure to
enable both the Trillium executables to be built as well as the header
files and man pages to be installed. See the INSTALL file for
details.
Since LAM/MPI is now build with automake/libtool, the two
libraries necessary for MPI programs (liblam and libmpi) can be built
as shared or static libraries. See the INSTALL file for details.
However, the two third party packages that LAM/MPI includes -- the
MPI 2 C++ bindings and ROMIO -- do *not* support shared library
builds. If you select to build either of these packages (both are
enabled by default; see the INSTALL file for how to disable them),
building LAM/MPI with shared library support is disallowed.
Additionally, GNU libtool also does not appear to build shared
libraries correctly on all systems. For example, libtool fails to
build the LAM shared libraries properly under AIX 4.3.3. Hence, LAM
can only be built statically on such platforms. This is a limitation
of libtool, not LAM. Since LAM uses libtool to build itself, libtool
limitations can constrain the build options for LAM/MPI. libtool
cannot also reliably build C++ libraries (it *usually* works for
static linking of C++ libraries that have no templates, but shared
libraries typically don't work with non-GNU compilers).
It is hoped that future versions of the GNU libtool package will
allow buildings C++ libraries and correct building of shared libraries
on all platforms. Write to your congressman/local government
representative! ;-)
Several new features have been added to mpirun.
New "C" syntax has been added to mpirun (note that this section
also applies to the "lamexec" command). When running on SMP machines,
it is frequently desirable to group as many adjoining ranks as
possible on a single node in order to maximize shared memory message
passing. When used in conjunction with the extended bootschema syntax
(that allows the specification of number of CPUs available on each
host), the mpirun "C" syntax will run one executable on each available
CPU, and will group adjoining MPI_COMM_WORLD ranks on the same nodes.
For example, when running on two SMPs, the first having four CPUs, and
the second having two CPUs, the following command:
will run four copies of "foo" on the four-way SMP (MPI_COMM_WORLD
ranks 0 through 3), and will run two copies of "foo" on the two-way
SMP (MPI_COMM_WORLD ranks 4 and 5).
Just like the "N" syntax in mpirun, the "C" syntax can also be
used to indicate specific CPUs. For example:
runs "foo" on the fourth and fifth CPUs (i.e., the two-way SMP
from the previous examples). "C" and "cX" syntax can also be
combined:
could be used to launch a "master" process (i.e., rank 0 in
MPI_COMM_WORLD) on CPU zero, a slave on each CPU (including rank 0.
This may be desirable, for example, in situations where the master
rank does very little computation).
The behavior of "-np" has been altered to match the "C" semantics.
"-np" now schedules across CPUs, not nodes. Using "-np 6" in the
previous example would be the same as "C"; using "-np 4" would run one
four copies of "foo" on the four-way SMP.
Also note that "N", "nX", C", and "cX" syntax can all be used
simultaneously, although it is not clear that this is really useful.
All environment variables named LAM_MPI_* are now automatically
passed to remote notes (unless disabled via the "-nx" option to
mpirun). The "-x" option enabled exporting of specific environment
variables to the remote nodes:
This will launch the "samIam" application on all available CPUs.
The LAM_MPI_FOO, DISPLAY, and SEUSS variables will be created on all
nodes before the user's program is invoked.
Note that the parser for the "-x" option is currently not very
sophisticated -- it cannot even handle quoted values when defining new
environment variables. Users are advised to set variables in the
environment prior to invoking mpirun, and only use "-x" to export the
variables to the remote nodes (not to define new variables), if
possible.
The "-pty" option to mpirun enabled pseudo tty support. Among
other things, this gives line-buffered output from the remote nodes
(which is probably what you want). "-pty" is now the default mode for
mpirun. Use the "-npty" option to disable pseudo tty support.
The default behavior of mpirun has changed with respect to the
directory that processes will be started in.
The "-wd" option to mpirun allows the user to change to an
arbitrary directory before their program is invoked. It can also be
used in application schema files to specify working directories on
specific nodes and/or for specific applications.
If the "-wd" option appears both in a schema file and on the
command line, the schema file directory will override the command line
value. "-wd" is mutually exclusive with "-D".
If neither "-wd" nor "-D" are specified, the local node will send
the directory name where mpirun was invoked from to each of the remote
nodes. The remote nodes will then try to change to that directory.
If they fail (e.g., if the directory does not exist on that node),
they will start from the user's home directory.
All directory changing occurs before the user's program is
invoked; it does not wait until MPI_INIT is called.
mpirun can now also run non-LAM/MPI programs. That is, one can
mpirun a shell script, debugger, or any other program that will
eventually either exec a LAM/MPI program or spawn a LAM/MPI program as
a child.
This is extremely helpful for batch systems and debugging
environments. For example:
However, one must be careful when running programs on remote nodes
that expect the use of stdin. Recall that stdin on remote nodes is
redirected to /dev/null. It is probably more advantageous to export
the DISPLAY environment variable, and run a shell script that invokes
an xterm with "gdb" running in it on each node. For example:
The default options now correspond to "-w -c2c -nger". That is:
To get the old behavior, use the options "-lamd -ger -nw".
mpirun now recognizes command lines of the form
For example,
runs 4 copies of the /bin/foobar program on nodes n0 and n1 (2
copies on each node), passing the arguments, 12 a b c, to the program.
The new syntax is equivalent to the following in the "-c" syntax which
is still supported.
The lamexec command has been added to LAM/MPI's reporitore. It is
an "mpirun clone", but is specifically for running non-MPI programs.
That is, one can do the following:
which will run "hostname" on all nodes in the multicomputer. It
can take most of the same command line arguments as mpirun; it does
not support the flags that do not make sense for non-MPI programs
(e.g., -c2c, -lamd, etc.). See lamexec(1) for more details.
Note that the "C" syntax for mpirun also applies to lamexec.
The hcc, hcp, and hf77 wrapper compilers have previously not
automatically passed the "-lmpi" option to the underlying compiler.
The rationale behind this decision was that the "mpicc" and "mpif77"
wrapper compilers added this functionality; the "h" wrappers were
intended as Trollius compilers, not LAM/MPI compilers.
But hcc, hcp, and hf77 have become the de facto wrapper compilers
(vs. mpicc and mpif77). Indeed, some users have been confused about
why -lmpi is not implicit to the "h" wrapper compilers.
Hence, "-lmpi" is now automatically passed to the underlying
compiler in the hcc, hcp, and hf77 wrapper compilers. The mpicc and
mpif77 compilers are now symbolic links to hcc and hf77, respectively.
For symmetry, mpiCC has been created as a symbolic link to hcp.
The help messages in the lamboot, lamhalt, and wipe commands have
been greatly enhanced. It is hoped that this will be very useful to
new users of LAM.
Since many users do not use the Bourne or Korn shells, an
alternative algorithm that does not invoke the extra remote shell can
be utilized with the "-b" switch, resulting in a nice speedup. Bourne
and Korn shell users cannot use the "-b" switch. It should be noted
that Bash, even though it is a Bourne shell derivative, *can* use the
"-b" switch for faster execution.
lamhalt is considerably faster and does not require a hostfile.
wipe should only be used if something happened to "damage" the
currently running LAM such that lamhalt fails to shut it down
properly.
indicates that there are four CPUs available on the "blinky" host,
and that there are two CPUs available on the "pinky" host. Note that
this works nicely in a PBS environment, because PBS will list a host
multiple times when multiple vnodes on a single node have been
allocated by the scheduler.
Note that different usernames can be specified for specific hosts
as well. For example:
specifies that the username "lamguest" should be used to login to
the machine "blinky". This is different than previous syntax for
specifying usernames for remote nodes; although its use is deprecated,
the "-o" option to recon, lamboot, and wipe can be used to obtain
previous parsing behavior.
still works as expected, by redirecting the output of the parallel
run of "foo" to the file foo.out. The "-s" option does allow the
lamboot command to be executed remotely by a remote agent such as rsh:
rsh will wait for stdout/stderr to close before returning. So
even if lamboot completes, rsh will hang while waiting for
stdout/stderr of all launched processes to close. The "-s" option
will force lamboot to close stdout/stderr and allow rsh to complete
normally. While most programs will not be affected, be sure to note
that "-s" will break tstdio functionality.
Previous versions of wipe would ask for a filename if none were
specified on the command line; this option has been removed.
The LAMRSH environment variable can contain a binary name and any
number of arguments. For example (for the C shell and its
derivatives):
LAM was specifically designed to abstract away hostnames once
lamboot has completed successfully. However, for various reasons
(usually related to system-administration concerns, and/or for
creating human-readable reports), it can be desirable to retrieve the
hostnames of LAM nodes long after lamboot.
A new command, "lamnodes", can be used for this purpose. It
accepts both the "N" and "C" syntax from mpirun, and will return the
corresponding names of the specified nodes. For example:
will return the node that each CPU is located on, the hostname of
that node, and the total number of CPUs on each (lamnodes' command
line arguments can be used to customize its output).
will return the node, hostname, and number of CPUs on n0 and n3.
It is a Very Bad Idea to run the LAM executables as root.
LAM was designed to be run by individual users; it was *not*
designed to be run as a root-level service where multiple users use
the same LAM daemons in a client-server fashion (see "Typical Usage"
in the INSTALL file). LAM should be booted by each individual user
who wishes to run MPI programs. There are a wide array of security
issues when root runs a service-level daemon; LAM does not even
attempt to address any of these issues.
Especially with today's propensity for hackers to scan for
root-owned network daemons, it could be tragic to run this program as
root. While LAM is known to be quite stable, and LAM does not leave
network sockets open for random connections after the initial setup,
several factors should strike fear into system administrator's hearts
if LAM were to be constantly running for all users to utilize:
Hence, it's a Very Bad Idea to run LAM as root. LAM binaries will
quit immediately if root runs them. Login as a different user to run
LAM.
LAM currently provides three client-to-client transport layers
which implement the request progression interface (RPI). As in LAM
6.1, the LAM daemon RPI transport (lamd) is always available. It is
no longer the default transport and must be explicitly invoked via the
-lamd option to mpirun.
The three client-to-client transports are:
The tcp transport uses TCP sockets for all interprocess
communication.
The usysv transport is multi-protocol. Processes on the same node
communicate via SYSV shared memory and processes on different nodes
communicate via TCP sockets. It uses spin-locks for shared memory
synchronization as well as a SYSV semaphore or pthread mutex for
synchronizing access to a per node global shared memory pool.
The spin-locks require that the architecture has strongly ordered
writes and this transport is only supported on such platforms. It
should be relatively easy to modify this transport to work on systems
with weakly ordered writes by adding memory barriers in appropriate
places.
The usysv transport should give the best performance on
SMPs. Please refer to the tuning notes for more on the performance of
the various transports and on tuning them.
The sysv transport is the same as the usysv transport except that
SYSV semaphores are used for message synchronization rather than
spin-locks. On some uniprocessor systems (e.g. Linux), the blocking
nature of semaphores can lead to better performance than when using
spin-locks.
LAM MPI now catches the signals SEGV, BUS, FPE, and ILL. The
signal handler terminates the application. This is useful in batch
jobs to help ensure that mpirun returns if an application process
dies. To disable the catching of signals use the -nsigs option to
mpirun.
The signal used internally by LAM has been changed from SIGUSR1 to
SIGUSR2 to reduce the chance of conflicts with the Linux pthreads
library. The signal used is configurable. See the installation guide
for the specific ./configure flag that can be used to change the
internal signal.
Support has been added for the MPI_LONG_LONG_INT,
MPI_UNSIGNED_LONG_LONG and MPI_WCHAR basic datatypes.
LAM 6.5.3 has support for many MPI-2 features.
MPI-IO support has been added by including the ROMIO package from
Argonne National Labs (http://www.mcs.anl.gov/romio/), version 1.0.3.
The MPI-IO functions are described in chapter 9 of the MPI-2 standard,
which can be found at http://www.mpi-forum.org/.
The ROMIO package can be compiled with LAM, and the LAM wrapper
compilers (mpicc/mpiCC/mpif77) will automatically do "the right
things" to compile/link user programs that use ROMIO function calls.
The ROMIO group at Argonne has now included support for LAM in
their package. However, there are still "one or two" things that the
LAM Team had to do to make ROMIO behave nicely with LAM/MPI. As such,
if you try to install the ROMIO package manually with LAM/MPI, you
will experience some difficulties.
ROMIO support can be disabled via the LAM ./configure script; see
the INSTALL file for specific instructions.
There are some important limitations to ROMIO that are discussed
in the romio/README file.
One limitation that is not currently listed in the ROMIO README
file is that atomic file access will not work with AFS. This is
because of file locking problems with AFS (i.e., AFS does not support
file locking). The ROMIO test program "atomicity" will fail if you
specify an output file on AFS.
Since building ROMIO is now the default for LAM/MPI, building
LAM's profiling layer is also the default. You cannot build ROMIO
without profiling support (i.e., if you attempt to configure LAM to
not build the profiling layer and still build ROMIO, the configure
script will complain).
Additionally, ROMIO does not [yet] support shared libraries. Even
though LAM can now support shared libraries, if you select to build
ROMIO (the default), shared library support is disabled.
Additionally, ROMIO does not support the following LAM
functionality:
Please see the sections "ROMIO Users Mailing List" and "Reporting
Bugs" in romio/README for how to submit questions and bug reports
about ROMIO (that do not specifically pertain to LAM).
C++ bindings for MPI-1 are provided from the MPI-2 C++ bindings
package from the University of Notre Dame
(http://www.mpi.nd.edu/research/mpi2c++/), version 1.0.6. The MPI-1
C++ bindings are described in Chapter 10 and Appendix B of the MPI-2
standard, which can be found at http://www.mpi-forum.org/.
The C++ bindings package is compiled, by default, with LAM, and
the LAM wrapper compilers (mpicc/mpiCC/mpif77) will automatically do
"the right things" to compile/link user programs that use MPI C++
bindings function calls.
Note that the C++ bindings have requirements on the degree of
conformance that your C++ compiler supports; see the file
mpi2c++/README for more details. If your C++ compiler cannot support
the requirements of the C++ bindings package, it is safest just to
disable MPI C++ bindings support in LAM (however, most modern C++
compilers now support all the required functionality).
MPI C++ bindings support can be disabled via the LAM ./configure
script; see the INSTALL file for specific instructions.
Please see the "Contact Information" section of the mpi2c++/README
file for how to submit questions and bug reports about the MPI 2 C++
bindings package (that do not specifically pertain to LAM).
Inter-language interoperability is supported. It is now possible
to initialize LAM MPI from either C or Fortran and mix MPI calls from
both languages.
Support is provided for get/put/accumulate data transfer
operations and for the post/wait/start/complete and fence
synchronization operations. No support is provided for window locking.
The datatypes used in the get/put/accumulate operations are
restricted to being basic datatypes or single level contiguous/vectors
of basic datatypes.
The implementation of the one-sided operations is layered on top
of the point-to-point functions, and will thus perform no better than
them. Nevertheless it is hoped that providing this support will aid
developers in developing and debugging codes using one-sided
communication.
The following functions related to one-sided communication have
been implemented.
The dynamic process support provided in LAM 6.2b has been extended
and the function names changed to conform to the final MPI-2
standard. The following functions related to dynamic process support
are provided.
As requested by LAM users, MPI_Comm_spawn and
MPI_Comm_spawn_multiple can now take an MPI_Info argument specifying
an appschema file for MPMD spawns. The key name is "file", and the
value should be the filename of the appschema (relative to the present
working directory on the root, or an absolute pathname accessible on
the root).
These functions supersede the MPIL_Comm_spawn function that
LAM/MPI introduced in version 6.2b. MPIL_Comm_spawn is considered
depreciated and should not be used (although it is still available for
backwards compatibility).
Full support for info objects is provided.
The new communicator error handler functions are supported, and
window error handlers are also supported.
Handle conversions for inter-language interoperability are
supported.
Attributes may now be set on and retrieved from datatypes and
windows. The new communicator attribute handling functions are also
supported.
Support has been added for the following new derived type
constructors.
Support has also been added for the new type inquiry functions.
Implementations of the following functions are provided. LAM 6.5.3 reports its MPI version as 1.2.
The MPI_Alloc_mem and MPI_Free_mem functions (and their two
corresponding RPI functions) have been added in anticipation of RPIs
that can take advantage of "special" memory for fast message passing.
They are simply wrappers to malloc() and free() (respectively) in the
currently provided RPIs, but can be used for portability and future
performance gains.
|
|
Questions? Comments? Feedback? Please click here |
This site is located in: Notre Dame, IN, USA |
Copyright ©1996-2001
LAM Team / UND 2-Apr-2001 / 14:27:02 EST |