Backing Up and Restoring Data¶

Database backups and registry data¶

The store-gateway component stores database backups and is used to store data for the registry. On our local machine, we can use s3cmd sync to copy the objects locally:

$ s3cmd sync s3://db_wal .
$ s3cmd sync s3://registry .

Log data¶

The store-volume service mounts a filesystem which is used by the controller and logger components to store and retrieve application and component logs.

Since this is just a POSIX filesystem, you can simply tarball the contents of this directory and rsync it to a local machine:

$ ssh core@<hostname> 'cd /var/lib/deis/store && sudo tar cpzf ~/store_file_backup.tar.gz .'
tar: /var/lib/deis/store/logs/deis-registry.log: file changed as we read it
$ rsync -avhe ssh core@<hostname>:~/store_file_backup.tar.gz .

Note that you’ll need to specify the SSH port when using Vagrant:

$ rsync -avhe 'ssh -p 2222' core@127.0.0.1:~/store_file_backup.tar.gz .

Note the warning - in a running cluster the log files are constantly being written to, so we are preserving a specific moment in time.

Database data¶

While backing up the Ceph data is sufficient (as database ships backups and WAL logs to store), we can also back up the PostgreSQL data using pg_dumpall so we have a text dump of the database.

We can identify the machine running database with deisctl list, and from that machine:

core@deis-1 ~ $ docker exec deis-database sudo -u postgres pg_dumpall > dump_all.sql
core@deis-1 ~ $ docker cp deis-database:/app/dump_all.sql .

Database backups and registry data¶

Because neither the database nor registry have started, the bucket we need to restore to will not yet exist. So, we’ll need to create those buckets:

$ s3cmd mb s3://db_wal
$ s3cmd mb s3://registry

Now we can restore the data:

$ s3cmd sync basebackups_005 s3://db_wal
$ s3cmd sync wal_005 s3://db_wal
$ s3cmd sync registry s3://registry

Log data¶

Once we copy the tarball back to one of the CoreOS machines, we can extract it:

$ rsync -avhe ssh store_file_backup.tar.gz core@<hostname>:~/store_file_backup.tar.gz
$ ssh core@<hostname> 'cd /var/lib/deis/store && sudo tar -xzpf ~/store_file_backup.tar.gz --same-owner'

Note that you’ll need to specify the SSH port when using Vagrant:

$ rsync -avhe 'ssh -p 2222' store_file_backup.tar.gz core@127.0.0.1:~/store_file_backup.tar.gz

Finishing up¶

Now that the data is restored, the rest of the cluster should come up normally with a deisctl start platform.

The last task is to instruct the controller to re-write user keys, application data, and domains to etcd. Log into the machine which runs deis-controller and run the following. Note that the IP address to use in the export command should correspond to the IP of the host machine which runs this container.

$ nse deis-controller
$ cd /app
$ export ETCD=172.17.8.100:4001
./manage.py shell <<EOF
from api.models import *
[k.save() for k in Key.objects.all()]
[a.save() for a in App.objects.all()]
[d.save() for d in Domain.objects.all()]
[c.save() for c in Certificate.objects.all()]
[c.save() for c in Config.objects.all()]
EOF
$ exit

That’s it! The cluster should be fully restored.