An example on how to use duplicity to perform encrypted incremental backups on Amazon S3.
I'm using duplicity version 0.6.20 (released October 28, 2012).
Getting started
If you've never heard about duplicity before, you should check the documentation.
Install duplicity
First, you need to install duplicity, I always install it from source since the duplicity package is not often updated.
$ sudo apt-get install python-dev librsync-dev
$ cd /opt
$ sudo wget https://code.launchpad.net/duplicity/0.6-series/0.6.20/+download/duplicity-0.6.20.tar.gz
$ sudo tar xvzf duplicity-0.6.20.tar.gz
$ cd duplicity-0.6.20
$ python sudo setup.py install
But you can install it with apt-get
$ sudo apt-get install duplicity
Next you can also install s3cmd from S3 Tools, it's a command line tool for managing your S3 buckets, but it's not required.
$ sudo apt-get install s3cmd
$ s3cmd --configure
Encrypted Backups
Before backing up the data, you need to think about encryption, duplicity makes use of gpg and handles both private/public key pair (a gpg key) and symmetric encryption (a passphrase).
I use passsphrases since I'll never lose it and I don't have to backup a gpg key.
My backup script
Since you need to specify many args to perform the differents actions, I crafted a bash script that make working with duplicity easier, duptools.
Features
- Backup multiple directories
- Send email report on backup
- Quickly list file and show bucket status
- Restore your files easily
Duptools
#!/bin/bash
export AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_ACCESS_KEY
export PASSPHRASE=YOU_PASSHRASE
# directories, space separated
SOURCE="/home/thomas/backup /home/thomas/bin /home/thomas/documents"
BUCKET=s3+http://mybucket
LOGFILE=/home/thomas/tmp/duplicity.log
# set email to receive a backup report
EMAIL=""
backup() {
INCLUDE=""
for CDIR in $SOURCE
do
TMP=" --include ${CDIR}"
INCLUDE=${INCLUDE}${TMP}
done
# perform an incremental backup to root, include directories, exclude everything else, / as reference.
duplicity --full-if-older-than 30D $INCLUDE --exclude '**' / $BUCKET > $LOGFILE
if [ -n "$EMAIL" ]; then
mail -s "backup report" $EMAIL < $LOGFILE
fi
}
list() {
duplicity list-current-files $BUCKET
}
restore() {
if [ $# = 2 ]; then
duplicity restore --file-to-restore $1 $BUCKET $2
else
duplicity restore --file-to-restore $1 --time $2 $BUCKET $3
fi
}
status() {
duplicity collection-status $BUCKET
}
if [ "$1" = "backup" ]; then
backup
elif [ "$1" = "list" ]; then
list
elif [ "$1" = "restore" ]; then
if [ $# = 3 ]; then
restore $2 $3
else
restore $2 $3 $4
fi
elif [ "$1" = "status" ]; then
status
else
echo "
duptools - manage duplicity backup
USAGE:
./duptools.sh backup
./duptools.sh list
./duptools.sh status
./duptools.sh restore file [time] dest
"
fi
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export PASSPHRASE=
Installation
Set up config vars at the top of the script and make the script executable.
Backup
$ ./duptools.sh backup
List/Status
$ ./duptools.sh list
$ ./duptools.sh status
Restore
Be careful while restoring not to preprend a slash to the path.
Restoring a single file to tmp
$ ./duptools.sh restore home/thomas/bin/setupscreen tmp/setupscreen
Restoring an older version of a directory to tmp (interval or full date)
$ ./duptools.sh restore home/thomas/bin 1D3h5s tmp/bin
$ ./duptools.sh restore home/thomas/bin 2012/7/5 tmp/bin
Bakthat, a lightweight solution
You may also want to check out bakthat, a python command-line tool I wrote, that allow you to compress, encrypt (symmetric encryption) and upload files directly to Amazon S3/Glacier in a single command.
And you ?
If you have any suggestions, feel free to share it !
Tip with Bitcoin
Tip me with Bitcoin and vote for this post!
Leave a comment