Backup and Restore
===================

Backup of juju's state is a critical feature, not only for juju users
but for use inside juju itself.  This is likewise the case for the
ability to restore previous backups.  This doc is intended as an
overview of both since changes in juju are prone to break both.

Backup
-------------------

Backing up juju state involves dumping the state database (currently
from mongo) and copying all files that are critical to juju's operation.
All the files are bundled up into an archive file.  Effectively the
archive represents a snapshot of juju state.

That snapshot is stored by the controller in such a way that only a
backup ID is needed for restore (no need to upload an archive).  Note
that if the controller is not available, restoring with just the ID is
not an option.  While that situation will need to be addressed in the
short-term, it should not require much additional effort.

We make reasonable efforts to ensure that the archive is consistent with
the snapshot.  This includes stopping the database for the length of
time it takes to dump it.  There is, however, room for improvement with
regard to ensuring the consistency of the archive.

First of all, running juju commands will fail while the DB is
unavailable (already running services should not be affected).  While
this period of time is rather short, we expect that it will grow over
time.  Furthermore, the larger a model's state, the larger the
impact of this downtime.  In the long term this makes it less than ideal
to run backup as often as it should be.

Secondly, state currently does not block for the backup process as a
whole.  This means that if we dump the DB first, it may be out of date
by the time we finish gathering the state-related files.  In practice
this isn't a big concern since we do not expect the files to change
during the interval that backup is running.  However, we do backup some
log files, so there is a small chance they will differ from when backups
started.

Restore
-------------------

Restore involves reviving the juju state in a new model by
reversing the steps taken by backup.  However, the process is a bit more
complicated than just gathering files and dumping the DB.

If no controller is present restore will do the following:

1. bootstrap a new node in safe mode (ProvisionerSafeMode reports
   whether the provisioner should not destroy machines it does not know
   about), 
2. stop juju-db,
3. stop jujud-machine,
4. load the backed-up db in place of the recently created one,
5. un-tar the fs files onto the root dir of the current machine,
6. run a set of bash scripts that replace the dns/instance names of the
   old machine with those of the new machine in the relevant config
   files and also in the db (if this step is not performed peergrouper
   will kick our machine out of the vote list and fill it with the old
   dead ones),
7. restart all services.

As noted above, restoring via an uploaded archive (rather than by using
an ID) will need to be addressed in the short term, since the existing
restore functionality works this way.  However, it shouldn't involve
more than bootstrapping a new model, uploading the archive to it,
and then requesting restore of that backup.  The design in this document
already accommodates doing this.

HA
-------------------

HA is a work in progress, for the moment we have a basic support which is an
extension of the regular backup functionality.
Read carefully before attempting backup/restore on an HA model.

In the case of HA, the backup process will backup files/db for machine 0,
support for "any working controller" is plans for the near future.
We assume, for now, that if you are running restore is because you have
lost all controller machines. Out of this restore you will get one
functioning controller that you can use to start you other state machines.
BEWARE, only run restore in the case where you no longer have working
controllers since otherwise this will take them offline and possibly
cripple your model.

Previous Implementation
-------------------

Backup and restore were both implemented as plugins (in cmd/plugins/) to
the juju CLI.  The plugins were essentially scripts we sent over SSH to
the state machine and ran there.  However, they were definitely distinct
pieces of software.


Implementation
===================

Key Points
-------------------

* Backups are created and then stored on the state machine.
* Each backup archive has an associated metadata document (stored in
  mongo).
* Each backup archive is stored relative to the state machine (currently
  env storage).
* Restore will have access to the state machine where the archive and
  metadata are stored.
* In the common case there is no need to upload or download a backup.
* The backups machinery has its own facade in state/apiserver/backups.
* The choice of mechanism for uploading and downloading backups has not
  been decided yet.
* The backups machinery is divided into 4 layers:
  - state-dependent functionality,
  - state-independent functionality,
  - the state API facade for backups,
  - the juju CLI sub-command for backups.
* The state-independent functionality can be broken down further:
  - a high-level backups interface/implementation,
  - low-level backup/restore functionality,
  - components of the backups machinery.
* Backups depend on the github.com/juju/utils/filestorage package.
* Backups have a special relationship with state (see note at
  beginning of state/backups.go).

Backup Archives
-------------------

Each backup archive is a gzipped tar file (.tar.gz).  It has the
following structure.

juju-backup/
    metadata.json - the backup metadata for the archive.
    root.tar      - the bundle of state-related files (exluding mongo).
    dump/         - all the files dumped from the DB (using mongodump).

At present we do not include any sort of manifest/index file in the
archive.

For more information, see:
  - state/backups/db/dump.go     - how the DB is dumped;
  - state/backups/files/files.go - which files are included in root.tar.

File Layout
--------------------

The layering of the backups machinery and divisions of the state-
independent functionality map almost directly to the following
structure in the filesystem.  The state API facade for backups is spread
between state/apiserver and state/api.

state/
    backups.go - state-dependent functionality (basically the
                 interaction with mongo and with env storage)
    backups/   - state-independent functionality
        backups.go - high-level/public backups interface/implementation
        create.go  - low-level implementation of backing up juju state
        restore.go - low-level implementation of restoring juju state
        archive/   - an abstraction of a backups archive
        db/        - all stuff related to external interaction with the
                     DB (internal interactions live in state/backups.go)
        files/     - all stuff related to files we back up and restore
        metadata/  - the backups metadata implementation
    apiserver/
        backups/ - the state API facade for backups
            backups.go - facade implementation (not including methods
                         for end-points)
            create.go  - implementation of the Create() end-point
            info.go    - (wraps state/backups/backups.go:Backups.Get)
            list.go
            remove.go
            restore.go - implementation of the Restore() end-point
    api/
        backups.go - the juju state API client implementation for the
                     backups facade
        params/
            backups.go - the backups-related API arg/result types
    cmd/
        juju/
            backups.go - the juju CLI sub-command implementation

Note that upload/download aren't accommodated in apiserver/backups/ yet.

Layers of Abstraction
--------------------

As noted above, the backups machinery is divided in 4 layers and the
state-independent portion into 3 parts.  Here is an example (using
"create") of how those layers interact.

* The juju CLI for backups wraps:
  - the backups facade's Create() method.
* The state API facade wraps:
  - the high-level backups implementation (state/backups/backups.go),
  - the state-backups interactions (state/backups.go).
* the backups implementation wraps:
  - a "filestorage" implementation (../utils/filestorage:FileStorage),
  - the low-level "create" implementation,
  - DB connection info (state/backups/db/info.go),
  - the backup's metadata.
* the "create" implementation makes use of:
  - the code in state/backups/{archive,db,files}.

Backups Interface
--------------------

Backups
  Add(meta Metadata, archive io.ReadCloser) error
  Create() (id string, err error)
  Get(id string) (Metadata, io.ReadCloser, error)
  List() ([]Metadata, error)
  Remove(id string) error
  Restore(id string) error

Note: Restore() makes use of Get().

State API Facade
--------------------

BackupsAPI
  Create(BackupsCreateArgs) BackupsCreateResult
  Info(BackupsInfoArgs) BackupsMetadataResult
  List() BackupsMetadataListResult
  Remove(BackupsRemoveArgs)
  Restore(BackupsRestoreArgs)

Again note that upload and download are not yet included here.

CLI sub-command
--------------------

The juju CLI sub-command for backups is called "backups".  Its own
sub-commands have basically a 1-to-1 equivalence with the API client
methods of the same respective names.  The essential sub-commands are
exposed via the following options:

- juju backups [--create] [--quiet] [<notes>]
- juju backups --info <ID>
- juju backups --list [--brief]
- juju backups --remove <ID>
- juju backups --restore <ID>

Note: further options may be appropriate for later addition
      (e.g. [<filename>] on --restore).

Other anticipated subcommands:

- juju backups --download <ID> [<filename>]
- juju backups --upload <filename>

Note that download and upload are only hypothetical, pending support in
the API facade.  When we add the ability to restore from an archive
(rather than an ID), --download and --upload (or a --filename option on
restore) will become essential.
