dirvish - linux "rsync hardlink" backup to disk

Dirvish - Linux Backup to Disk

http://www.dirvish.org/

Nice tutorial and tipps: ~~http://wiki.edseek.com/howto:dirvish~~
http://www.ullright.org/ullWiki/show/howto-dirvish-copy-of-http-wiki-edseek-com-howto-dirvish

Dirvish & btrfs: https://www.mendix.com/blog/btrfs-dirvish-perfect-match/

Config Options

xdev: 0/1
- Whether to stay in the filesystem or cross over to others
- 1 = stay, do not cross, 0 = allowed to cross
- Corresponds to "rsync -x" -> Stay in the filesystem - do not cross over to another filesystem
- Set to "1" in most cases
zxfer:
- 1 = Compress transfer. Corresponds to "rsync -z"
exclude
- Use leading slash to exclude an "anchored" top level directory e.g. for vault /home.
  The exclude pattern is relative to the vault directory.
  - exclude:
    /myusername/

Execution

dirvish-runall

Master configuration file

/etc/dirvish/master.conf

## Example dirvish master configuration file:
xdev: 1
index: gzip
image-default: %Y-%m-%d_%H-%M

bank:
        /srv/backup
exclude:
        lost+found/
        *~
        .nfs*
Runall:
        home
        etc
        var
expire-default: +5 days
expire-rule:
#       MIN HR    DOM MON       DOW  STRFTIME_FMT
        *   *     *   *         1    +1 months
        *   *     1   *         *    +1 year

Note: If you have multiple backups per day, set the "HR" part of the expire-rule to the same hour as the cronjob.

Vault configuration

cd /srv/backup
sudo mkdir etc
cd etc
sudo mkdir dirvish

sudo vi dirvish/default.conf

client: my_server
# tree: use trailing "/" !
tree: /etc
exclude:                  
       # Exclude a specific directory                          
       /cache/  #excludes /etc/cache                   
       # Exclude any directory in all subdirectories
       cache/     #excludes e.g. /etc/cache, /etc/php5/cache, ...
#       /**/.*/**/*cache/
#       /**/.*/*cache/
#       /**/.*/**/*Cache/
#       .kde/share/thumbnails/

Example network config : /media/disk/my_remote_backup/home_my_user/dirvish/default.conf

client: my_user@remote_server.example.com
# tree: use trailing "/" !
tree: /home/my_user/
#exclude:
#       /**/.*/**/*cache/
pre-client: /home/my_user/bin/dirvish_mysql_dump.sh
    
zxfer: 1
speed-limit: 0.5

Initialize a vault

dirvish --vault my_vault --init

Run cronjob

sudo vi /etc/cron.d/dirvish

# /etc/cron.d/dirvish: crontab fragment for dirvish

# run every night
4 1 * * *     root      /etc/dirvish/dirvish-cronjob

Check

du -sch /srv/dirvish/myvault/*

Example MySQL dump script

on the remote machine

Example: /home/my_user/bin/dirvish_mysql_dump.sh

# dumps mysql databases locally to be included in remote dirvish backup cd /home/my_user/mysql_dumps mysqldump -u user1 --password=password1 --add-drop-table db1 > db1.mysql.dump mysqldump -u user2 --password=password2 --add-drop-table db2 > db2.mysql.dump chmod 600 *.dump

Duplicate / Copy a dirvish backup disk

# -H stands for "preserve hardlinks"
sudo rsync -avH --delete --stats --progress /media/disk/ /media/disk-1/

Automated replication to a backup disk

/etc/dirvish/backup-clone

#! /bin/bash
#
# daily cron job for the dirvish package
# Klemens Ullmann-Marx 2010-01-31
#
## Example of how to mount and umount a backup partition...

MOUNTPOINT="/media/inseq_backup_1"

mountpoint $MOUNTPOINT

# $? is the exit status of the last bash command
# ("mountpoint" in that case")
# The exit status "0" in case of success, an integer between "1" and "255" in case of error
if [ $? -eq 0 ] ; then
  echo "Good..."
else
  echo "Trying to mount it now..."
  mount $MOUNTPOINT
  if [ $? -eq 0 ] ; then
    echo "$MOUNTPOINT successfully mounted!"
  else
    echo "Failed to mount $MOUNTPOINT!"
    exit 1
  fi
fi

#rsync -aHq --delete /srv/backup $MOUNTPOINT
umount $MOUNTPOINT

Symlink latest backups to a directory for remote backup

/etc/divish/master.conf
- post-server: /root/bin/latest_backup_symlink.sh

/root/bin/latest_backup_symlink.sh

#!/bin/bash
#
# 2009-10-12 by Klemens Ullmann-Marx

BACKUP_DEST=/srv/backup_latest$DIRVISH_SRC

rm -rf $BACKUP_DEST
ln -s $DIRVISH_DEST $BACKUP_DEST

Remote Backup

http://apt-get.dk/howto/backup/

Alter expire date

Modify dirvish/default.hist

Use "Never" for expires

2016-01-21_08-35        2016-01-21 08:49:12     2016-01-21_08-02 Never

Modify 201x-xx-xx/dirvish/summary
- Use "Never" for expires

Delete a snapshot

Delete the directory
Modify dirvish/default.hist
- Delete correct line
- Alter reference

Delete a file or directory from all snapshots (DANGEROUS!)

Example: delete a directory "public" from a vault /srv/backup/srv

find /srv/backup/srv -type d -name public -exec ls -lah '{}' \;
Replace "ls -lah" with "rm -vrf" to actually delete the directories

Example limit the find directory depth to find only top level directories from the /var backup

find /srv/backup/var -maxdepth 3 -type d -name cache -exec echo '{}' \;

No unexpired good images

Situation: Remote backup of windows box. Some files had access denied. Furthermore the box was'nt up regularily, so a lot of images where missing.

Solution: Change "Status:" to "sucess" in an early backup (Note, this can expire a lot of images, and nearly all if e.g. the 1st of each month was missing!)

Dirvish vault can be on the same volume as the data to be backed up

Tested with

Data in /srv/test
- test.txt, content "apple"
Vault in /srv/backup/test
dirvish --vault test
Edit /srv/test/test.txt to "pear"
-> /srv/backup/test/20xx-yy-zz/tree/test.txt is still "apple"

Conclusion: Hardlinks seem to be created only from one dirvish snapshot to the following. Hardlinks are not used in the initial initialisation.

Updated at 06/14/2021 08:59

https://www.ullright.org/ullWiki/show/dirvish-linux-rsync-hardlink-backup-to-disk

Create

Links