As the backup user, configure rsnapshot (vi /ffp/etc/rsnapshot.conf) by modifying the lines shown below.  Note that fields should be separated by a ‘\t’-characters.

The exclude file should be in rsync format.  Refer to “backup-using-rsync” for an example.

snapshot_root   /mnt/HD_a2/rsnapshot/
cmd_cp          /ffp/bin/cp
cmd_du          /ffp/bin/du
cmd_rsnapshot_diff      /ffp/bin/rsnapshot-diff

du_args -csh
lockfile        /ffp/home/backup/.rsnapshot/rsnapshot.pid
#interval       hourly  0
interval        daily   7
interval        weekly  4
interval        monthly 3

#backup  /home/          localhost/
#backup  /etc/           localhost/
#backup  /usr/local/     localhost/
backup  rsync://ws.vonk:/users/        ws.vonk/users/     +rsync_long_args=–exclude-file=/ffp/etc/backup/ws.vonk/users,+rsync_long_args=–passwordfile=/ffp/home/backup/.rsync.passwd

#backup  ws.vonk:users/        ws.vonk/users/     +rsync_long_args=–exclude-file=/ffp/etc/backup/ws.vonk/users

Install on the client

Install rsyncd and rsync/ssh on the client system.  For instruction refer to “Preparing the client::rsyncd” and “Preparing the client::rsync/ssh” further down in this document.

Run some tests, and start a backup

[backup@backup]$ ssh backup@ws.vonk hostname

[backup@backup]$ rsync rsync://ws.vonk/users/     # test rsync (requires passwd)

[backup@backup]$ rsync -e ssh ws.vonk:/           # test rsync/ssh

[backup@backup]$ rsnapshot configtest             # verify configuration file syntax

[backup@backup]$ rsnapshot -v daily               # try a backup

rdiff-backup

This is also a nice package, requiring little tweaking or configuration to make it run on the DNS-321.  It uses the rsync library over SSH for transport and needs the same program and SSH to be run on the clients.  It appears to be widely deployed.

Install on the server

As root, install the tool and its dependencies on the backup server  (details at here, here and here)

mkdir /mnt/HD_a2/src/rdiff-backup
cd /mnt/HD_a2/src/rdiff-backup

for pp in librsync-0.9.7-1-ffp0.5.tgz Python-2.5.2-2.tgz ; do
wget http://www.drak0.com/files/dns323/$pp
funpkg -i $pp
done

for pp in rdiff-backup-1.2.8 ; do
wget http://savannah.nongnu.org/download/rdiff-backup/${pp}.tar.gz
tar -xvzf `basename $pp`.tar.gz
( cd `basename` $pp
python setup.py install –prefix=/ffp )
done

mkdir /ffp/etc/rdiff-backup
mkdir /mnt/HD_a2/rdiff-backup/ws.vonk/users
chown -R backup:backup /ffp/etc/rdiff-backup /mnt/HD_a2/rdiff-backup

Create an exclude file (vi /mnt/HD_a2/rdiff-backup/ws.vonk/users)

- U:/Users/User Name/Virtual Machines/
- U:/Users/User Name/Documents/.emacs.d/auto-save-list/
- U:/Users/User Name/Documents/.bash_history
- U:/Users/User Name/Documents/Other/Not Backed Up/
- U:/Users/User Name/Videos/Layout/releases/*/*.iso

Install on the client

Install rdiff-backup/ssh on the client system.  For instruction refer to “Preparing the client::rdiff-backup/ssh” further down in this document.

Run some tests, and start a backup

[backup@backup]$ ssh backup@ws.vonk hostname            # try SSH’ing to ws.vonk

[backup@backup]$ ssh ws.vonk ‘rdiff-backup –version’   # verify that rdiff-backup can run on ws.vonk
[backup@backup]$ rdiff-backup –test-server ws.vonk::/ # verify that rdiff-backup thinks everything is OK with ws.vonk

[backup@backup]$ rdiff-backup -v5 –print-statistics –exclude-symbolic-links \
–ssh-no-compression –no-compression \
–exclude-filelist /ffp/etc/rdiff-backup/ws.vonk/users \
ws.vonk::U:/Users /mnt/HD_a2/rdiff-backup/ws.vonk/users

Snapback2

This is another perl incarnation of Mike Rubel’s rsync_snapshots .  I found it less polished than rsnapshot that provides similar functionality.

Install on the server

Install the support package (Perl)

[user@linux]$ ssh root@backup

mkdir -p /mnt/HD_a2/src/snapback2

cd /mnt/HD_a2/src/snapback2
rsync -avP inreto.de::dns323/fun-plug/0.5/extra-packages/All/perl-5.10-2.tgz .

funpkg -i perl-5.10-2.tgz

Install Config::ApacheFormat from source (there is no binary ffp package)

for pp in gcc-4.1-2.tgz uclibc-0.9.29-7.tgz kernel-headers-2.6.9.1-2.tgz binutils-2.18.50.0.1-4.tgz make-3.81-3.tgz distcc-3.0-1.tgz ; do

rsync -avP inreto.de::dns323/fun-plug/0.5/packages/$pp .

funpkg -i $pp

done

# doesn’t work .. /usr/bin/perl -MCPAN -e ‘install Config::ApacheFormat’

wget http://search.cpan.org/CPAN/authors/id/S/SC/SCHWIGON/class-methodmaker/Class-MethodMaker-2.15.tar.gz

tar -xvzf Class-MethodMaker-2.15.tar.gz

cd Class-MethodMaker-2.15/
perl Makefile.PL
make && make test && make install

cd ..

wget http://search.cpan.org/CPAN/authors/id/S/SA/SAMTREGAR/Config-ApacheFormat-1.2.tar.gz

tar -xvzf Config-ApacheFormat-1.2.tar.gz

cd Config-ApacheFormat-1.2

perl Makefile.PL
make && make test && make install

cd ..

Finally, it is time to install snapback2

wget http://www.perusion.com/misc/Snapback2/Snapback2-0.913.tar.gz
tar -xvzf Snapback2-0.913.tar.gzcd Snapback2-0.913/

perl Makefile.PL
make && make install

cd ..

sudo mkdir /mnt/HD_a2/snapback

sudo -R chown backup:backup /mnt/HD_a2/snapback
mkdir ~backup/.snapback

As “backup”, create ~backup/.snapback/snapback2.conf

Hourlies    0
Dailies     7
Weeklies    4
Monthlies   12
AutoTime    Yes

AdminEmail user.name@emailprovider.com

LogFile /ffp/home/backup/.snapback/snapback.log
IgnoreVanished yes
SnapbackRoot /ffp/home/backup/.snapback
RsyncVerbose yes

Cp /ffp/bin/cp
Mv /ffp/bin/mv
Rm /ffp/bin/rm

DestinationList /mnt/HD_a2/snapback

<Backup ws.vonk>
Directory /Settings
Exclude “/Public/”
Exclude “/Default/”
Exclude “/Default User/”
Exclude “/HelpAssistant/”
Exclude “/LocalService/”
Exclude “/NetworkService/”
Exclude “/*/ntuser.dat*”
</Backup>

<Backup ws.vonk>
Directory /Users
Exclude “/User Name/Virtual Machines/”
Exclude “/User Name/Documents/.emacs.d/auto-save-list/”
Exclude “/User Name/Documents/.bash_history”
Exclude “/User Name/Documents/Other/Not Backed Up/”
Exclude “/User Name/Videos/Layout/releases/*/*.iso”
</Backup>

Install on the client

Install rsync/ssh on the client system.  For instruction refer to “Preparing the client::rsync/ssh” further down in this document.

Run some tests, and start a backup

[backup@backup]$ ssh backup@ws.vonk hostname

[backup@backup]$ rsync -e ‘ssh’ ws.vonk:users

[backup@backup]$ snapback2 -d -c ~/.snapback/snapback2.conf

BackupPC

BackupPC is a different beast.  It detects identical files in different backups.  It supports a variety of transport mechanism such as rsyncd, SMB, ssh/rsync and ssh/tar.  Update: I just learned that there is a fun package for BackupPC.

Install on the server

Install Perl and the Perl modules Compress::Zlib, Archive::Zip and File::RsyncP from source (details in the howto)

[user@linux]$ ssh root@backup

mkdir -p /mnt/HD_a2/src/backuppc

cd /mnt/HD_a2/src/backuppc
for pp in perl-5.10-2.tgz ; do
rsync -avP inreto.de::dns323/fun-plug/0.5/extra-packages/All/$pp .
funpkg -i $pp
done

# alternate: cpan YAML Compress::Zlib Archive::Zip File::RsyncP  # get a drink, because this takes a while

distccd –allow 127.0.0.1

for pp in P/PM/PMQS/Compress-Raw-Zlib-2.021 P/PM/PMQS/Compress-Raw-Bzip2-2.021 P/PM/PMQS/IO-Compress-2.022 P/PM/PMQS/Compress-Zlib-2.015 A/AD/ADAMK/Archive-Zip-1.30 C/CB/CBARRATT/File-RsyncP-0.68 ; do

wget http://search.cpan.org/CPAN/authors/id/$pp.tar.gz

tar -xvzf `basename $pp`.tar.gz
( cd `basename $pp`

perl Makefile.PL

make && make test && make install )

done

Install BackupPC from source (detailed instructions)

wget http://downloads.sourceforge.net/project/backuppc/backuppc/3.1.0/BackupPC-3.1.0.tar.gz

tar zxf BackupPC-3.1.0.tar.gz
( cd BackupPC-3.1.0

( # fix compile error – Bareword “compareLOGName” not allowed while “strict subs” in use at lib/BackupPC/Lib.pm line 1466

cd lib/BackupPC

cp Lib.pm Lib.pm~

sed ‘s/sort(compareLOGName @files)/sort compareLOGName @files/’ < Lib.pm~ > Lib.pm )
perl configure.pl )

Answer the configuration questions

–> Full path to existing main config.pl []? press enter
–> Are these paths correct? [y]? press enter
–> BackupPC will run on host [backup]? press enter
–> BackupPC should run as user [backuppc]? backup
–> Install directory (full path) [/usr/local/BackupPC]? /ffp/home/backup/backuppc
–> Data directory (full path) [/data/BackupPC]? /mnt/HD_a2/backuppc
–> Compression level [3]? 0
–> CGI bin directory (full path) []? /mnt/HD_a2/www/pages/cgi-bin
–> Apache image directory (full path) []? /mnt/HD_a2/www/pages/BackupPC
–> URL for image directory (omit http://host; starts with ‘/’) []? /BackupPC
–> Do you want to continue? [y]? press enter

Move the configuration file from the ramdisk to the hard disk

mv /etc/BackupPC /ffp/etc/

ln -s /mnt/HD_a2/ffp/etc/BackupPC /etc/

mkdir /mnt/HD_a2/www/logs  # create the log directory while we’re at it

mkdir -p /mnt/HD_a2/backuppc/pc /mnt/HD_a2/backuppc/cpool

chown -R backup:backup /mnt/HD_a2/www/logs /mnt/HD_a2/backuppc

Create a startup script (vi /ffp/start/backuppc.sh), and give it execute permissions (chmod 755)

#!/bin/sh
# From http://wiki.dns323.info/howto:backuppc

# PROVIDE: backuppc
# REQUIRE: DAEMON
# BEFORE: LOGIN
# KEYWORD: shutdown

. /ffp/etc/ffp.subr

name=”backuppc”
start_cmd=”backuppc_start”
restart_cmd=”backuppc_restart”
stop_cmd=”backuppc_stop”
status_cmd=”backuppc_status”

backuppc_start()
{
ln -s /mnt/HD_a2/ffp/etc/BackupPC /etc/
ln -s /mnt/HD_a2/ffp/var/log/BackupPC /var/log/
su – backup -s /bin/sh -c ‘/mnt/HD_a2/ffp/home/backup/backuppc/bin/BackupPC -d’
echo “${name} started”
}

backuppc_restart()
{
backuppc_stop
sleep 1
backuppc_start
}

backuppc_stop()
{
/ffp/bin/pkill -f “/mnt/HD_a2/ffp/usr/BackupPC/bin/BackupPC -d”
echo “${name} stopped”
}

backuppc_status()
{
if [ "`ps ax | grep "BackupPC -d" | grep perl`" = "" ] ; then
echo “${name} not running”
else
echo “${name} running”
fi
}

extra_commands=”status”
run_rc_command “$1″

Fix the cgi script (vi BackupPC_Admin)  to work with lighttpd.

( cd /mnt/HD_a2/www/pages/cgi-bin
ln -s BackupPC_Admin BackupPC_Admin.pl )

( cp BackupPC_Admin BackupPC_Admin~
sed ‘s,#!/ffp/bin/perl,#!/mnt/HD_a2/ffp/bin/perl,’ < BackupPC_Admin~ > BackupPC_Admin )

Set the XferMethod to used rsyncd as the transport method

sed -i~ “s/$Conf{XferMethod} = ‘smb’;/$Conf{XferMethod} = ‘rsyncd’;/” /ffp/etc/BackupPC/config.pl

Some more housekeeping

mkdir /ffp/etc/BackupPC/pc
chown -R backup:backup /ffp/etc/BackupPC /var/log/BackupPC
chown -R backup:backup /mnt/HD_a2/www

Create a configuration file for the client (vi /ffp/etc/BackupPC/pc/ws.vonk.pl)

$Conf{XferMethod} = ‘rsyncd’;
$Conf{RsyncShareName} = ‘users’; #this setting must match a later file
$Conf{RsyncdUserName} = ‘backup’; #remember this username and password for a later file
$Conf{RsyncdPasswd} = ‘rsyncpassword‘;

# this line is optional but helps get rid of transfer errors in your log files while backing up
# in addition to not backing up some unnecessary temp and cache files
$Conf{BackupFilesExclude} = [
'/User Name/Virtual Machines',
'/User Name/Documents/.emacs.d/auto-save-list',
'/User Name/Documents/.bash_history',
'/User Name/Documents/Other/Not Backed Up/',
'/User Name/Videos/Layout/releases/*/*.iso'
];

Add the following line to /ffp/etc/BackupPC/hosts

ws.vonk 0 backup

Install on the client

Install rsyncd and rsync/ssh on the client system.  For instruction refer to “Preparing the client::rsyncd” and “Preparing the client::rsync/ssh” further down in this document.

Run some tests, and start a backup

killall BackupPC
/ffp/home/backup/backuppc/bin/BackupPC -d

Start a backup through the web page “backup:8080/cgi-bin/BackupPC_Admin.pl”

backup-using-rsync, my old but proven script

About 8 years ago, I stumbled across Art Muler’s snapback script.  At the time, I did not have a /bin/bash shell, so I massaged it to run under /bin/ash, extended the snapshot rotation mechanism, and made it more robust to interrupted backups.  Like so many other tools, it uses  “snapshot”-style backups with  hard links to create the illusion of multiple, full backups without much of the space or processing overhead.

Configure the server

Create some place holders for the scripts and configuration files

[user@linux]$ ssh root@backup
    mkdir /ffp/etc/backup
    touch /ffp/bin/backup
    touch /ffp/bin/backup-using-rsync
    mkdir /mnt/HD_a2/backup
    chown backup:backup /ffp/etc/backup /ffp/bin/backup /ffp/bin/backup-using-rsync /mnt/HD_a2/backup
    exit

From the backup account (ssh backup@backup), create /ffp/bin/backup, and give it execute-permissions (chmod 755)

#!/ffp/bin/bash
# GPL $Id$

LOGGER=/ffp/bin/logger
CONFIG_DIR=/ffp/etc/backup
RSYNC_BACKUP=/ffp/bin/backup-using-rsync
BACKUP_DIR=/mnt/HD_a2/backup
AWK=/ffp/bin/awk
DF=/ffp/bin/df
PASSWD_FILE=/ffp/home/backup/.rsync.passwd
LS=/ffp/bin/ls

echo "Starting $0 ..." | $LOGGER -s -t backup

FILES=`( cd $CONFIG_DIR ; pwd ; $LS */* )`

for node in $FILES ; do
    $RSYNC_BACKUP \
        --password-file=${PASSWD_FILE} \
        --exclude-from=${CONFIG_DIR}/${node} \
        $* \
    rsync://${node} \
    ${BACKUP_DIR}/${node} 2>&1 | $LOGGER -s -t backup
done

$DF -h ${BACKUP_DIR} 2>&1 | $LOGGER -s -t backup

( cd ${BACKUP_DIR} ; $LS -dl --time-style=long-iso */*/* | $AWK '{ printf("stored backups: %08s %s\n", $6, $8) }' | $LOGGER -s -t backup )

From the backup account, create /ffp/bin/backup-using-rsync, and give it execute-permissions (chmod 755)

#!/ffp/bin/bash
# GPL $Id$
# ----------------------------------------------------------------------
# rotating-filesystem-snapshot utility using 'rsync'
#
# inspired by http://www.mikerubel.org/computers/rsync_snapshots
# ----------------------------------------------------------------------
# probably still runs under /ffp/bin/ash if you want ..

#set -o nounset  # do not allow uninitialized variables
#set -o errexit  # exit if any statement returns an error value

# ------------- file locations -----------------------------------------

#SNAPSHOT_DEV="/dev/sda2"
SNAPSHOT_DIR=/mnt/HD_a2/backup
LOCKFILE=/mnt/HD_a2/ffp/home/backup/`basename $0`.pid

# ------------- system commands used by this script --------------------

ECHO=/ffp/bin/echo
CUT=/ffp/bin/cut
PING=/ffp/bin/ping
GREP=/ffp/bin/grep
SED=/ffp/bin/sed
AWK=/ffp/bin/awk
PS=/ffp/bin/ps
DIRNAME=/ffp/bin/dirname
DATE=/ffp/bin/date

# after parsing the command line parameters, these the following commands
# will be prefixed with $DRY

#MOUNT=/ffp/bin/mount
MKDIR=/ffp/bin/mkdir
CHMOD=/ffp/bin/chmod
RM=/ffp/bin/rm
MV=/ffp/bin/mv
CP=/ffp/bin/cp
TOUCH=/ffp/bin/touch
RSYNC=/ffp/bin/rsync

# ------------- other local variables ----------------------------------

PROGRAM=`basename $0`
USAGE="
Usage: $PROGRAM [--parameters] SRC DST
--verbose              - increase verbosity
--quiet                - decrease verbosity
--exclude=PATTERN      - exclude files matching PATTERN
--exclude-from=FILE    - patterns listed in FILE
--include-from=FILE    - don't exclude patterns listed in FILE
--dry-run              - do not start any file transfers
just report the actions it would have taken
--remove-last-daily    - remove the last backup
--version              - shows revision number
Example:
$PROGRAM --verbose --exclude-from=/ffp/etc/backup/hostname/module rsync://hostname/module $SNAPSHOT_DIR/hostname/module
"

# ------------- the script itself --------------------------------------

usage() {
$ECHO "$USAGE"
}

case "$1" in
--help|"")
usage
exit 0
;;
--version)
REVISION=`$ECHO '$Revision 0.1$'|tr -cd '0-9.'`
$ECHO "$PROGRAM version $REVISION"
exit 0
;;
--help)
usage
exit 0
;;
esac

# ------ print the error message to stderr, and remount r/o-------------

die() {
$ECHO "$PROGRAM: $*"
$ECHO "use '$PROGRAM --help' for help"
#$MOUNT -t ext3 -o remount,ro $SNAPSHOT_DEV $SNAPSHOT_DIR
exit 1
}

# ------ execute a command, and exit on error --------------------------

checkExit() {
$* || die "ERROR: $*"
}

# ----- returns 0 if $LOCKFILE exists, 1 otherwise ---------------------

removeOldLock() {
if [ -e ${LOCKFILE} ] ; then
a=`cat ${LOCKFILE}`
if ! `$PS | $AWK "\\$1 == \"$a\" { exit 1 }"` ; then
$ECHO "$PROGRAM:isLocked: WARNING cleaning old lockfile"
rm -f $LOCKFILE
fi
fi
}

isLockedOBSOLETE() {
if [ ! -e $LOCKFILE ] ; then
return 0
fi

# if the process that created the lock is dead, then cleanup its lockfile
a=`cat ${LOCKFILE}`
if ! `$PS | $AWK "\\$1 == \"$a\" { exit 1 }"` ; then
$ECHO "$PROGRAM:isLocked: WARNING cleaning old lockfile"
rm -f $LOCKFILE
return 0;
fi

return 1;
}

# ------- cleanup TERM, EXIT and INT traps -----------------------------

cleanup() {
trap - INT TERM EXIT

if [ -e $LOCKFILE ] ; then
LOCKFILE_PROCID=`cat $LOCKFILE`
if [ "$$" = "$LOCKFILE_PROCID" ] ; then
$RM -f $LOCKFILE
else
$ECHO "$PROGRAM: Can't remove lockfile ($LOCKFILE)"
$ECHO "process $LOCKFILE_PROCID created the lock, while this process is $$)"
fi
fi
exit $1
}

# ----- print to stdout when the debug level $VERB >= $1 ---------------

verbose() {
local LEVEL="$1"
[ ! -z "$LEVEL" ] || die "verbose: unspecified LEVEL"

if [ $VERB -ge $LEVEL ] ; then
shift
echo "$PROGRAM: $*"
fi
}

# ------ prints directory, if debug level $VERB >= $1 ------------------

verbose_ls() {
[ $VERB -lt $1 ] || ( shift ; ls -l $*/ )
}

# --- returns 0 if rsyncd is running on host $1, 1 otherwise -----------

rsyncRunningOnRemote() {
local SOURCE="$1"
local HOSTNAME

[ ! -z "$SOURCE" ] || die "rsyncRunningOnRemote: unspecified source"

if $ECHO $SOURCE | grep '^rsync://'  2>/dev/null >/dev/null ; then
HOSTNAME=`$ECHO $SOURCE | $CUT -d/ -f3`
if $RSYNC $HOSTNAME::  2>/dev/null >/dev/null ; then
return 0
else
return 1
fi
else
return 1
fi
}

# ------ returns the name of the oldest daily/weekly backup directory --

findOldest() {
local TYPE="$1"
local ALL_DAILY
local OLDEST_DAILY

[ ! -z "$TYPE" ] || die "findOldest: unspecified duration {daily|weekly}"

ALL_DAILY=`ls -d -r $DST/$TYPE.* 2>/dev/null`
OLDEST_DAILY=`$ECHO $ALL_DAILY | $SED "s,^$DST/,," | $CUT -d' ' -f1`

echo $OLDEST_DAILY
}

# ----- returns 0 if weekly backup should be made, 1 otherwise ---------

shouldMakeWeeklyBackup() {
local OLDEST_DAILY
local TODAY_DAY TODAY_YEAR
local OLDEST_DAILY_DAY OLDEST_DAILY_YEAR

OLDEST_DAILY=`findOldest daily`

# no point in making a weekly backup, if there is no daily one
if [ -z $OLDEST_DAILY ] ; then
return 1
fi

# only make a weekly backup if the oldest daily backup is at least 7 days old

TODAY_DAY=`$DATE +%j | $SED 's/^0*//g'` # leading 0 would represent Octal
TODAY_YEAR=`$DATE +%Y`

OLDEST_DAILY_DAY=`$DATE -r $DST/$OLDEST_DAILY +%j | $SED 's/^0*//g'`
OLDEST_DAILY_YEAR=`$DATE -r $DST/$OLDEST_DAILY +%Y`

let DAY_OF_FIRST_WEEKLY=$OLDEST_DAILY_DAY+7

if [ $TODAY_YEAR -ne $OLDEST_DAILY_YEAR ] ; then
let TODAY_DAY+="356 * ($TODAY_YEAR - $OLDEST_DAILY_YEAR)"
fi

if [ $TODAY_DAY -lt $DAY_OF_FIRST_WEEKLY ] ; then
verbose 2 "No weekly backup, $TODAY_DAY -lt $DAY_OF_FIRST_WEEKLY"
return 1
fi

# make a weekly backup, if the last weekly backup was >= 14 days ago, or
# there was no last weekly backup.

TODAY_DAY=`$DATE +%j | $SED 's/^0*//g'`
TODAY_YEAR=`$DATE +%Y`

if [ -d $DST/weekly.0 ] ; then
LAST_WEEKLY_DAY=`$DATE -r $DST/weekly.0 +%j | $SED 's/^0*//g'`
LAST_WEEKLY_YEAR=`$DATE -r $DST/weekly.0 +%Y`
else
LAST_WEEKLY_DAY=0
LAST_WEEKLY_YEAR=0
fi

let DAY_OF_NEXT_WEEKLY=$LAST_WEEKLY_DAY+14
if [ $TODAY_YEAR -ne $LAST_WEEKLY_YEAR ] ; then
let TODAY_DAY=$TODAY_DAY+365
fi

if [ $TODAY_DAY -ge $DAY_OF_NEXT_WEEKLY ] ; then
verbose 2 "Weekly backup, today($TODAY_DAY) -ge next($DAY_OF_NEXT_WEEKLY)"
return 0
else
verbose 2 "No weekly backup, today($TODAY_DAY) -ge next($DAY_OF_NEXT_WEEKLY)"
return 1
fi
}

# ----- renumber the $1 {daily,weekly} backups, starting at $2 ---------

renumber() {
local TYPE="$1"
local START="$2"

[ ! -z "$TYPE" ] || die "renumber: missing TYPE"
[ ! -z "$START" ] || die "renumber: missing START"

[ "$TYPE" = "daily" ] || [ "$TYPE" = "weekly" ] || die "renumber: incorrect TYPE"

LIST=`ls -d $DST/$TYPE.* 2>/dev/null`
for ITEM in $LIST ; do
$MV $ITEM $ITEM.tmp
done

COUNT=$START
for ITEM in $LIST ; do
ITEM_NEW=`$DIRNAME $ITEM`/$TYPE.$COUNT
$MV $ITEM.tmp $ITEM_NEW
let COUNT++
done
}

# ----- create the backup ------------------------------------ ---------

backup() {
local OLDEST_DAILY

$MKDIR -p $DST || die "backup: $MKDIR -p $DST"

verbose 2 "STEP 0: the status quo"
verbose_ls 2 $DST

if shouldMakeWeeklyBackup ; then

verbose 2 "STEP 1: delete weekly.2 backup, if it exists"

if [ -d $DST/weekly.2 ] ; then
$RM -rf $DST/weekly.2
fi ;

verbose_ls 2 $DST

verbose 2 "STEP 2: shift the middle weekly backups(s) back by one,"\
"if they exist"

renumber weekly 1

verbose_ls 2 $DST

OLDEST_DAILY=`findOldest daily`

verbose 2 "STEP 3: make a hard-link-only (except for dirs) copy of"\
"$OLDEST_DAILY, into weekly.0"

if [ -d $DST/$OLDEST_DAILY ] ; then
$CP -al $DST/$OLDEST_DAILY $DST/weekly.0
fi

verbose_ls 2 $DST

# note: do *not* update the mtime of weekly.0; it will reflect
# when daily.7 was made, which should be correct.
else
verbose 2 "STEP 1: no weekly backup needed, skipping STEP 2 and 3"
fi

verbose 2 "STEP 4: delete daily.7 backup, if it exists"

if [ -d $DST/daily.7 ] ; then
$RM -rf $DST/daily.7
fi

verbose_ls 2 $DST

verbose 2 "STEP 5: shift the middle backups(s) back by one, if they exist"

renumber daily 1

verbose_ls 2 $DST

verbose 2 "STEP 6: make a hard-link-only (except for dirs) copy of the"\
"latest backup, if that exists"

if [ -d $DST/daily.1 ] ; then
$CP -al $DST/daily.1 $DST/daily.0
else
$MKDIR -p $DST/daily.0
$CHMOD 755 $DST/daily.0
fi;

verbose_ls 2 $DST

verbose 2 "STEP 7: rsync from $SRC to $DST/daily.0"

# (notice that rsync behaves like cp --remove-destination by default, so
# the destination is unlinked first.  If it were not so, this would copy
# over the other backup(s) too!

verbose 1 "$RSYNC --archive --delete --delete-excluded --chmod=u=rwx $PARAM $SRC $DST/daily.0"
verbose 0 "$SRC"

# --compress
$DRY $RSYNC --archive --delete --delete-excluded --chmod=u=rwx $PARAM $SRC $DST/daily.0

verbose 1 "$RSYNC done"
verbose 2 "STEP 8: update the mtime of daily.0 to reflect the backup time"

$TOUCH $DST/daily.0

# at the end of the week, the oldest daily backup, becomes last weeks
# backup

verbose_ls 2 $DST

verbose 1 "STEP 9: done"
}

# ----- remove the last daily backup -----------------------------------

removeLastDaily() {
verbose 2 "STEP 1: renumbering daily backups starting at ($DST/daily.0)"

renumber daily 0

verbose 2 "STEP 2: deleting the newest backup, if it exists "\
"($DST/daily.0)"

if [ -d $DST/daily.0 ] ; then
$RM -rf $DST/daily.0

verbose 2 "STEP 3: renumbering daily backups starting at "\
"($DST/daily.0)"

renumber daily 0
fi
}

# ----- remount the file system ----------------------------------------

remount() {
local MOUNT_MODE="$1"
[ ! -z "$MOUNT_MODE" ] || die "remount, missing MOUNT_MODE"

#$MOUNT -t ext3 -o remount,$MOUNT_MODE $SNAPSHOT_DEV $SNAPSHOT_DIR
}

# ------------- main ---------------------------------------------------

PARAM=
VERB=0
DRY=
REMOVE_LAST_DAILY=

while [ -n "$1" ] ; do
case $1 in
--verbose)
shift
let VERB++
;;
--quiet)
PARAM="$PARAM $1"
shift
[ $VERB -eq 0 ] || let VERB--
;;
--help | -h)
shift;
usage
exit 1;
;;
--dry-run)
PARAM="$PARAM $1"
shift;
DRY="$ECHO"
;;
--remove-last-daily)
shift;
REMOVE_LAST_DAILY=y
;;
-*)
PARAM="$PARAM $1"
shift
;;
*)
if [ -z $SRC ] ; then
SRC=$1
else
if [ -z $DST ] ; then
DST=$1
else
die "ignoring parameter '$1'"
fi
fi
shift
esac
done

RSYNC_VERS=`$RSYNC --version | $AWK '$1 == "rsync" && $2 == "version" { print $3 }'`

[ ! -z $SRC ] || die "source not specified"
[ ! -z $DST ] || die "destination not specified"

# [ `id -u` = 0 ] || die "only root can do that"

trap 'cleanup' TERM EXIT INT  # catch kill, script exit and int

echo testing for lock
if [ -z $DRY ] ; then
mkdir -p /var/lock

echo removing old lock
removeOldLock

echo creating new lock
if ( set -o noclobber ; echo "$$" > $LOCKFILE ) 2> /dev/null ; then
trap 'cleanup' TERM EXIT INT  # clean up lockfile at kill, script exit or ^c
else
die "Failed to acquire lock: $LOCKFILE held by $(cat $LOCKFILE)"
fi
echo got the lock
fi

MOUNT="checkExit $DRY /ffp/bin/mount"
MKDIR="checkExit $DRY /ffp/bin/mkdir"
CHMOD="checkExit $DRY /ffp/bin/chmod"
RM="checkExit $DRY /ffp/bin/rm"
MV="checkExit $DRY /ffp/bin/mv"
CP="checkExit $DRY /ffp/bin/cp"
TOUCH="checkExit $DRY /ffp/bin/touch"

verbose 2 "Backup '$SRC' -> '$DST'"
verbose 2 "parameters: '$PARAM'"

if [ ! -z $REMOVE_LAST_DAILY ] ; then
removeLastDaily
exit 0
fi

if rsyncRunningOnRemote $SRC ; then
remount rw
backup
RET=$?
remount ro
else
$ECHO "RSYNC daemon not running on '$SRC'"
RET=1
fi

exit $RET

The configuration file name indicate what machines should be backed up.  The contents of the file dictate what files should be excluded from the backup.  For example, the file /ffp/etc/backup/ws.vonk/users, tells the /ffp/bin/backup script that module “users” on client “ws.vonk” should be backed up.

Create a configuration file: /ffp/etc/backup-using-rsync/ws.vonk/users

- /User Name/Virtual Machines/
- /User Name/Documents/.emacs.d/auto-save-list/
- /User Name/Documents/.bash_history
- /User Name/Documents/Other/Not Backed Up/
- /User Name/Videos/Layout/releases/*/*.iso

Create an other configuration file: /ffp/etc/backup-using-rsync/ws.vonk/settings

- /Default/
- /Default User/
- /HelpAssistant/
- /LocalService/
- /NetworkService/
- /All Users/Microsoft/Windows Defender/
- /All Users/Microsoft/Search/
- /All Users/Microsoft/RAC/
- /All Users/Microsoft/eHome/
- /All Users/Microsoft/Windows NT/MSFax/
- /*/ntuser.dat*
- /*/AppData/
- /*/Application Data/
- /*/Local Settings/
- /*/My Documents/
- /*/NetHood/
- /*/PrintHood/
- /*/Recent/
- /*/Searches/

The bash script relies on syslog to communicate with the outside world.  Enable syslog deamon.  The syslog server configuration is described further down.

cd /mnt/HD_a2/ffp/start/
chmod a+x syslogd.sh
vi syslogd.sh
# add the line: syslogd_flags="-m 0 -R linux"
# change the line: klogd_flags="-c 8" # was -c 3

./syslogd.sh start

Install on the client

Install rsyncd on the client system as described in “Prepare the client::rsyncd” earlier in this document.

Run some tests, and start a backup

[backup@backup]$ rsync rsync://ws.vonk/users/     # test rsync (requires passwd)

    backup --verbose --verbose
#meanwhile ..
    tail -f /var/log/messages

Schedule a daily backup

Create the file /ffp/start/backup.sh and give it execute permission (chmod 755).  Add the backup to the crontab when the DNS-321 reboots.

#!/bin/sh
# set the time zone, while we are at it
echo "PST8PDT" > /etc/TZ

# schedule backup to run every day at 12:15
CRON=/ffp/tmp/backup.cron
/bin/crontab -u backup -l > ${CRON}
/bin/echo "15 12 * * * /ffp/bin/backup --verbose" >> ${CRON}
/bin/crontab ${CRON} -u backup
/bin/rm $CRON

This time add the backup to crontab by hand

[backup@backup]$ /ffp/start/backup.sh

Configure the Linux Syslog server

We will use logwatch as included in the Fedora 10 distribution.  Logwatch is set up to run once a day and generates a single email gathering the backup log analysis. To allow logwatch to check backup-using-rsync logs running on a Fedora (or other Linux system) you need to install this script & conf file.

On the syslog server, define which log files should be analyzed  (vi /etc/logwatch/conf/logfiles/backup.conf)

# GPL $Id$
# defines which log files should be analyzed for service backup

LogFile = messages
Archive = messages-*

On the syslog server, define the service (vi /etc/logwatch/conf/services/backup.conf

# GPL $Id$
# defines the service logwatch

Title = "BACKUP script (backup)"

LogFile = messages
*OnlyService = backup
*OnlyHost = backup
*RemoveHeaders =

On the syslog server, create the logwatch script (vi /etc/logwatch/scripts/backup), and give it execute permissions (chmod 755)

#!/usr/bin/perl
# GPL $Id$
# script for BACK logwatch for service backup

# example:
#  export show_only_server=ws.vonk
#  logwatch --archives --range yesterday \
#           --hostname backup.vonk --service backup --mailto root

$ShowOnlyServer    = $ENV{'show_only_server'}    || "";
$ShowSuccess       = $ENV{'show_successful'}     || 1;
$ShowFailed        = $ENV{'show_failed'}         || 1;
$ShowIOerror       = $ENV{'show_io_error'}       || 1;
$ShowVanishedFiles = $ENV{'show_vanished_files'} || 1;
$ShowFailedFiles   = $ENV{'show_failed_files'}   || 1;
$ShowDiskFree      = $ENV{'show_disk_free'}      || 1;
$ShowStored        = $ENV{'show_stored'}         || 1;
$ShowUnmatched     = $ENV{'show_unmatched'}      || ( $ShowOnlyServer eq "" );

sub showServer {
my($server) = @_;
return ( length($ShowOnlyServer) == 0 or ( $ShowOnlyServer eq $server ) );
}

while (defined($ThisLine = <STDIN>)) {

if ( ($Server,$Service) =
($ThisLine =~ /RSYNC daemon not running on \'rsync:\/\/(.*?)\/(.*?)\'/i ) ) {

$CurrServer="";
$CurrService="";
if ( showServer($Server) ) {
$Failed->{$Server}->{$Service}++;
}

} elsif ( ($Server,$Service) =
($ThisLine =~ /backup-using-rsync: rsync:\/\/(.*?)\/(.*?)$/i ) ) {

$CurrServer=$Server;
$CurrService=$Service;
if ( showServer($Server) ) {
$Success->{$Server}->{$Service}++;
}

} elsif ( ($FileName,$Service) = ($ThisLine =~ /file has vanished: \"(.*?)\" \(in (.*?)\).*$/i ) ) {

if ( showServer($Server) ) {
$VanishedFiles->{$CurrServer}->{$Service}->{$FileName}++;
}

} elsif ( ($FileName,$Service) = ($ThisLine =~ /rsync: read errors mapping \"(.*?)\" \(in (.*?)\):.*$/i ) ) {

if ( showServer($Server) ) {
$FailedFiles->{$CurrServer}->{$Service}->{$FileName}++;
}

} elsif ( ($ThisLine =~ /IO error encountered -- skipping file deletion/ ) ) {
if ( showServer($Server) ) {
$IOerror->{$CurrServer}->{$CurrService}++;
}

} elsif ( ($Date,$Server,$Service,$Period) =
($ThisLine =~ /stored backups: (.*?) (.*?)\/(.*?)\/(.*?)$/i )) {

if ( showServer($Server) ) {
$StoredBackup->{$Server}->{$Service}->{$Period} = $Date;
}

} elsif ( ($ThisLine =~ /ERROR: file corruption in/ ) or
($ThisLine =~ /rsync error: some files could not be transferred/ ) or
($ThisLine =~ /rsync: failed to connect to nis.vonk/ ) or
($ThisLine =~ /rsync error: error in socket IO \(code 10\) at clientserver.c/ ) or
($ThisLine =~ /--help/ ) or
($ThisLine =~ /backup-using-rsync: ERROR:/ ) ) {
# ignore

} elsif ( ($ThisLine =~ /Filesystem/ ) or
($ThisLine =~ /\/dev\/md0/ ) ) {
push @DiskFreeList,$ThisLine;

} else {
# Report any unmatched entries...
push @OtherList,$ThisLine;
}
}

if ($ShowSuccess) {
if (keys %{$Success}) {
print "\nSuccessful Backups:\n";
foreach    $Server (sort {$a cmp $b} keys %{$Success}) {
foreach $Service (sort {$a cmp $b} keys %{$Success->{$Server}}) {
print "\t" . $Server . "/" . $Service;
$count = $Success->{$Server}->{$Service};
if ( $count > 1 ) {
print " (" . $count . " times)";
}
print "\n";
}
}
}
}

if ($ShowFailed) {
if (keys %{$Failed}) {
print "\nFailed Backups:\n";
foreach    $Server (sort {$a cmp $b} keys %{$Failed}) {
foreach $Service (sort {$a cmp $b} keys %{$Failed->{$Server}}) {
print "\t" . $Server . "/" . $Service;
$count = $Failed->{$Server}->{$Service};
if ( $count > 1 ) {
print " (" . $count . " times)";
}
print "\n";
}
}
}
}

if ($ShowFailedFiles) {
if (keys %{$FailedFiles}) {
print "\nFiles skipped due to file locking:\n";
foreach    $Server (sort {$a cmp $b} keys %{$FailedFiles}) {
foreach $Service (sort {$a cmp $b} keys %{$FailedFiles->{$Server}}) {
print "\t" . $Server . "/" . $Service . "\n";
foreach $FileName (sort {$a cmp $b} keys %{$FailedFiles->{$Server}->{$Service}}) {
print "\t\t";
my $len=length($FileName);
if ( $len > 40 ) {
print ".." . substr( $FileName, $len - 38, 38);
} else {
print $Filename;
}
$count = $FailedFiles->{$Server}->{$Service}->{$FileName};
if ( $count > 1 ) {
print " (" . $count . " times)";
}
print "\n";
}
}
}
}
}

if ($ShowIOerror) {
if (keys %{$IOerror}) {
print "\nOld files not deleted as a precaution for an IO error:\n";
foreach    $Server (sort {$a cmp $b} keys %{$IOerror}) {
foreach $Service (sort {$a cmp $b} keys %{$IOerror->{$Server}}) {
print "\t" . $Server . "/" . $Service;
$count = $IOerror->{$Server}->{$Service};
if ( $count > 1 ) {
print " (" . $count . " times)";
}
print "\n";
}
}
}
}

if ($ShowVanishedFiles) {
if (keys %{$VanishedFiles}) {
print "\nFiles that vanished:\n";
foreach    $Server (sort {$a cmp $b} keys %{$VanishedFiles}) {
foreach $Service (sort {$a cmp $b} keys %{$VanishedFiles->{$Server}}) {
print "\t" . $Server . "/" . $Service . "\n";
foreach $FileName (sort {$a cmp $b} keys %{$VanishedFiles->{$Server}->{$Service}}) {
print "\t\t";
my $len=length($FileName);
if ( $len > 40 ) {
print ".." . substr( $FileName, $len - 38, 38);
} else {
print $Filename;
}
$count = $VanishedFiles->{$Server}->{$Service}->{$FileName};
if ( $count > 1 ) {
print " (" . $count . " times)";
}
print "\n";
}
}
}
}
}

if ($ShowStored) {
if (keys %{$StoredBackup}) {
print "\nStored Backups:\n";
foreach    $Server (sort {$a cmp $b} keys %{$StoredBackup}) {
foreach $Service (sort {$a cmp $b} keys %{$StoredBackup->{$Server}}) {
print "\t" . $Server . "/" . $Service . "\n";
foreach $Period (sort {$a cmp $b} keys %{$StoredBackup->{$Server}->{$Service}}) {
print "\t\t" . $StoredBackup->{$Server}->{$Service}->{$Period} .
" (" . $Period . ")\n";
}
}
}
}
}

if (($ShowDiskFree) and ($#DiskFreeList >= 0)) {
print "\nDisk Space:\n\n";
print @DiskFreeList;
}

if (($#OtherList >= 0) and ($ShowUnmatched)) {
print "\n**Unmatched Entries**\n";
print @OtherList;
}

exit(0);

On the syslog server, try the logwatch report

logwatch --archives --hostname backup.vonk --service backup --print | less

On the syslog server, schedule a cronjob to execute logwatch for this service (as root, crontab -e)

0 2 * * * /usr/sbin/logwatch --archives --range yesterday --hostname backup.vonk --service backup --mailto you@hostname.com
0 1 * * * export show_only_server=ws.vonk ; /usr/sbin/logwatch --archives --range yesterday --service backup --hostname backup.vonk --mailto you@hostname.com

Prepare the client

The installation and configuration examples in this section are focussed on Windows 7 x64, but can easily be adopted to other windows flavors such as 32-bit, Windows/XP or Windows/Vista.  Most Linux users will already have the programs installed, and can just use the configuration examples listed below.  I assume that the same is true for OS X users.

MS Windows does not include native versions of the popular rsync, rsyncd or ssh.  Depending on the backup tool used, we will have to install or or more of these programs.  The backup tool installation notes (further down) will tell you what programs are needed on the client computer

rsyncd

Rsyncd deamon runs as a service on the client and waits for the backup server to connect.  I like to use it for the initial backup, because it is about three times as fast as rsync-over-ssh.  The weak authentication and lack of encryption however makes it unsuitable for backing up remote systems.

Remember to disable the rsyncd when you switch over to the secure rsync/ssh (right-click Computer > manage > Services and Applications > Services > RsyncServer, stop and disable”.

To install:

1. download and install cwRsyncServer 4.0.2 to “C:\Program Files (x86)\rsyncd”
2. create a new service account “backup”

Create a secrets file (notepad “C:\Program Files (x86)\rsyncd\rsyncd.secrets”).  Not exactly a germ in authentication, but beats a total lack of authentication.

backup:rsyncpassword

Configure rsyncd (notepad “C:\Program Files (x86)\rsyncd\rsyncd.conf”).  For full restores, I like to have the option of a writable directory on the client.  When using cwRsyncServer, you will need to prepare to allow the service to write to this directory (Start > cwRsyncServer > Prepare a Dir for Upload).

use chroot = false
strict modes = false
hosts allow = backup.vonk
auth users = backup
secrets file = rsyncd.secrets

[users]
path = /cygdrive/u/Users/
read only = yes
list = yes

[settings]
path = /cygdrive/c/Users/
read only = yes
list = yes

#[users-rw]
#  path = /cygdrive/u/Users/recovered/
#  read only = no
#  list = no

rsync/ssh

A far more secure approach is to run rsync over a ssh-encrypted tunnel.  The sshd deamon runs as a service on the client and awaits connections from the backup server.  Authentication will use public key authentication.  From there on, rsync will communicate over the ssh tunnel.

1. Install copSSH
   • download copSSH 3.0.1 and install to “C:\Program Files (x86)\sshd\”
   • create a “backup” account, run as a service, do not generate user keys
2. Copy rsync.exe from cwRsync 4.0.1 to “C:\Program Files (x86)\sshd\bin”

Copy the public key to the client

[backup@backup]$ scp ~/.ssh/id_rsa.pub backup@ws.vonk:.ssh/authorized_keys2

Create a short cut to often used directories

[backup@backup]$ ssh ws.vonk

ln -s /cygdrive/c/Users settings

ln -s /cygdrive/u/Users users

exit

rdiff-backup/ssh

Another approach is to run rdiff-backup over a ssh-encrypted tunnel.  Again, the sshd deamon waits for the server to connect, and will authenticate using a public key.  But in this case rdiff-backup communicate with its peer over the ssh tunnel.

1. Install copSSH
   • download copSSH 3.0.1 and install to “C:\Program Files (x86)\sshd\”
   • create a “backup” account, run as a service, do not generate user keys
2. Download rdiff-backup-1.2.8-win32.zip and copy the .exe to “C:\Program Files (x86)\sshd\bin\”

Copy the public key to the client

[backup@backup]$ scp ~/.ssh/id_rsa.pub backup@ws.vonk:.ssh/authorized_keys2

Notes / feedback / questions

• Feedback and question can be addressed through the DNS-323 forum.
• The build-in fdisk dumps core.  The forum discussion points to an alternate fdisk
• My Windows 7 system was leaking Nonpaged Kernel Memory. The typical error message was “rsync: writefd_unbuffered failed to write 4092 bytes to socket [sender]: No buffer space available (105)”. In the end this turned out to be caused by Sun/Oracle’s VirtualBox. An upgrade of this software addressed the problem.