DMTCP(1)

NAME
dmtcp - Distributed MultiThreaded Checkpointing

SYNOPSIS
dmtcp_coordinator [port]
dmtcp_checkpoint command [args...]
dmtcp_restart ckpt_FILE1.dmtcp [ckpt_FILE2.dmtcp...] dmtcp_command coordinatorCommand dmtcp_inspector [-o<ofile>] [-t<tool>] [-cdaznh] <ckpt1.dmtcp> [ckpt2.dmtcp...] mtcp_restart FILE.mtcp

DESCRIPTION
DMTCP is a tool to transparently checkpointing the state of an arbitrary group of programs spread across many machines and connected by sockets. It does not modify the user's program nor the operating system. MTCP is a standalone component of DMTCP available as a checkpointing library for a single process.

OPTIONS
For each command, the --help or -h flag will show the command-line options. Most command line options can also be controlled through environment variables. These can be set in bash with "export NAME=value" or in tcsh with "setenv NAME value".

DMTCP_CHECKPOINT_INTERVAL=integer
    Time in seconds between automatic checkpoints. Checkpoints can also be initiated manually by typing 'c' into the coordinator. (default: 0, disabled; dmtcp_coordinator only)

DMTCP_HOST=string
    Hostname where the cluster-wide coordinator is running. (default: localhost; dmtcp_checkpoint, dmtcp_restart only)

DMTCP_PORT=integer
    The port the cluster-wide coordinator listens on. (default: 7779)

DMTCP_GZIP=(1|0)
    Set to "0" to disable compression of checkpoint images. (default: 1, compression enabled; dmtcp_checkpoint only) WARNING: gzip adds seconds. Without gzip, ckpt/restart is often less than 1 s

DMTCP_CHECKPOINT_DIR=path
    Directory to store checkpoint images in. (default: ./)

DMTCP_SIGCKPT=integer
    Internal signal number to use for checkpointing. Must not be used by the user program. (default: SIGUSR2; dmtcp_checkpoint only)

DMTCP_COORDINATOR
Each computation to be checkpointed must include a DMTCP coordinator process. One can explicitly start a coordinator through dmtcp_coordinator, or allow one to be started implicitly in background by either dmtcp_checkpoint or dmtcp_restart to operate. The address of the unique coordinator should be specified by dmtcp_checkpoint, dmtcp_restart, and dmtcp_command either through the --host and --port command-line flags or through the the DMTCP_HOST and DMTCP_PORT environment variables. If neither is given, the host-port pair defaults to localhost-7779. The host-port pair associated with a particular coordinator is given by the command-line flags used in the dmtcp_coordinator command, or the environment variables then in effect, or the default of localhost-7779. The coordinator is stateless and is not checkpointed. On restart, one can use an existing or a new coordinator. Multiple computations under DMTCP control can coexist by providing a unique coordinator (with a unique host-port pair) for each such computation. The coordinator initiates a checkpoint for all processes in its computation group. Checkpoints can be: performed automatically on an interval (see DMTCP_CHECKPOINT_INTERVAL above); or initiated manually on the standard input of the coordinator (see next paragraph); or initiated directly under program control by the comptuation through the dmtcpaware API (see below). The coordinator accepts the following commands on its standard input. Each command should be followed by the <return> key. The commands are:
  l : List connected nodes
  s : Print status message
  c : Checkpoint all nodes
  f : Force a restart even if there are missing nodes (debugging)
  k : Kill all nodes
  q : Kill all nodes and quit
  ? : Show this message Coordinator commands can also be issued remotely using dmtcp_command.

DMTCP_INSPECTOR
dmtcp_inspector is a tool for offline checkpoint analysis. It provides information about socket connections and parent-child relations between processes of a distributed program. The output is in graphviz package format and can be rendered using graphviz tools like dot, neato, twopi, circo, fdp and sfdp. Command line options are following:

-o, --out <file>
    Write output to <file>

-t, --tool
    Graphviz tool to use. By default no graphviz and output is in dot-like format

-c, --cred
    Add information about parent-child relations on the graph

-d, --no-sock
    Remove information about socket connections

-a, --sock-all
    Add verbose information about socket connections

-z, --sock-half
    Also represent half-connections (when some *.dmtcp files are missed

-n, --node
    Verbose node names indication

-h, --help
    Display this help

EXAMPLE USAGE

1. In a separate terminal window, start the dmtcp_coodinator.
    (See previous section.)
     dmtcp_coordinator

2. In separate terminal(s), replace each command(s) with "dmtcp_checkpoint
    [command]". The checkpointed program will connect to the coordinator specified by DMTCP_HOST and DMTCP_PORT. New threads will be checkpointed as part of the process. Child processes will automatically be checkpointed. Remote processes started via ssh will automatically checkpointed. (Internally, DMTCP modifies the ssh command line to call dmtcp_checkpoint on the remote host.)
     dmtcp_checkpoint ./myprogram

3. To manually initiate a checkpoint, either run the command below
    or type "c" followed by <return> into the coordinator. Checkpoint files for each process will be written to DMTCP_CHECKPOINT_DIR. The dmtcp_coordinator will write "dmtcp_restart_script.sh" to its working directory. This script contains the necessary calls to dmtcp_restart to restart the entire computation, including remote processes created via ssh.
         dmtcp_command -c
    OR: dmtcp_command --checkpoint

4. To restart, one should execute dmtcp_restart_script.sh, which is
    created by the dmtcp_coordinator in its working directory at the time of checkpoint. One can optionally edit this script to migrate processes to different hosts. By default, only one restarted process will be restarted in the foreground and receive the standard input. The script may be edited to choose which process will be restarted in the foreground.
     ./dmtcp_restart_script.sh

DMTCPAWARE API
DMTCP provides a programming interface to allow checkpointed applications to interact with dmtcp. In the source distribution, see dmtcpaware/dmtcpaware.h for the functions available. See test/dmtcpaware[123].c for three example applications. For an example of its usage, try:
 cd test; rm dmtcpaware1; make dmtcpaware1; ./autotest -v dmtcpaware1 The user application should link with libdmtcpaware.so (-ldmtcpaware) and use the header file dmtcp/dmtcpaware.h. The file utils/dmtcp.py in the source distribution provides an example python binding for the dmtcpaware interface.

DMTCP PLUGIN MODULES
The source distribution includes a top-level plugin directory, with examples of how to write a plugin module for DMTCP. Further examples are in the test/plugin directory. The plugin feature adds three new user-programmable capabilities. A plugin may: add wrappers around system calls; take special actions at during certain events (e.g. pre-checkpoint, resume/post-checkpoint, restart); and may insert key-value pairs into a database at restart time that is then available to be queried by the restarted processes of a computation. (The events available to the plugin feature form a superset of the events available with the dmtcpaware interface.) One or more plugins are invoked via a list of colon-separated absolute pathnames.
  dmtcp_checkpoint --with-plugin PLUGIN1[:PLUGIN2]...

RETURN CODE
A target program under DMTCP control normally returns the same return code as if executed without DMTCP. However, if DMTCP fails (as opposed to the target program failing), DMTCP returns a DMTCP-specific return code, rc (or rc+1, rc+2 for two special cases), where rc is the integer value of the environment variable DMTCP_FAIL_RC if set, or else the default value, 99.

SEE ALSO
Full documentation is available from http://dmtcp.sourceforge.net/

AUTHORS
DMTCP and its standalone single-process compontent MTCP (MultiThreaded CheckPointing) were created and are maintained by Jason Ansel, Kapil Arya, Gene Cooperman, Artem Y. Polyakov, Mike Rieker, Ana-Maria Visan, and a series of newer contributors including Alex Brick, Tyler Denniston, Rohan Garg, Gregory Kerr, and others.