• Edit
  • Delete

dirvish - linux "rsync hardlink" backup to disk

Dirvish - Linux Backup to Disk

http://www.dirvish.org/

Nice tutorial and tipps: http://wiki.edseek.com/howto:dirvish
http://www.ullright.org/ullWiki/show/howto-dirvish-copy-of-http-wiki-edseek-com-howto-dirvish

Dirvish & btrfs: https://www.mendix.com/blog/btrfs-dirvish-perfect-match/

Config Options

  • xdev: 0/1
    • Whether to stay in the filesystem or cross over to others
    • 1 = stay, do not cross, 0 = allowed to cross
    • Corresponds to "rsync -x" -> Stay in the filesystem - do not cross over to another filesystem
    • Set to "1" in most cases
  • zxfer:
    • 1 = Compress transfer. Corresponds to "rsync -z"
  • exclude
    • Use leading slash to exclude an "anchored" top level directory e.g. for vault /home.
      The exclude pattern is relative to the vault directory.
      • exclude:
            /myusername/

 

Execution

  • dirvish-runall

Master configuration file

  • /etc/dirvish/master.conf
    • ## Example dirvish master configuration file:
      xdev: 1
      index: gzip
      image-default: %Y-%m-%d_%H-%M
      
      bank:
              /srv/backup
      exclude:
              lost+found/
              *~
              .nfs*
      Runall:
              home
              etc
              var
      expire-default: +5 days
      expire-rule:
      #       MIN HR    DOM MON       DOW  STRFTIME_FMT
              *   *     *   *         1    +1 months
              *   *     1   *         *    +1 year
      

Note: If you have multiple backups per day, set the "HR" part of the expire-rule to the same hour as the cronjob.

Vault configuration

  • cd /srv/backup
  • sudo mkdir etc
  • cd etc
  • sudo mkdir dirvish
  • sudo vi dirvish/default.conf
    • client: my_server
      # tree: use trailing "/" !
      tree: /etc
      exclude:                  
             # Exclude a specific directory                          
             /cache/  #excludes /etc/cache                   
             # Exclude any directory in all subdirectories
             cache/     #excludes e.g. /etc/cache, /etc/php5/cache, ...
      #       /**/.*/**/*cache/
      #       /**/.*/*cache/
      #       /**/.*/**/*Cache/
      #       .kde/share/thumbnails/
  • Example network config : /media/disk/my_remote_backup/home_my_user/dirvish/default.conf
    • client: my_user@remote_server.example.com
      # tree: use trailing "/" !
      tree: /home/my_user/
      #exclude:
      #       /**/.*/**/*cache/
      pre-client:
      /home/my_user/bin/dirvish_mysql_dump.sh
      zxfer: 1
      speed-limit: 0.5

Initialize a vault

  • dirvish --vault my_vault --init

Run cronjob

  • sudo vi /etc/cron.d/dirvish
    • # /etc/cron.d/dirvish: crontab fragment for dirvish
      
      # run every night
      4 1 * * *     root      /etc/dirvish/dirvish-cronjob

 

Check

du -sch /srv/dirvish/myvault/*

 

Example MySQL dump script

  • on the remote machine
  • Example: /home/my_user/bin/dirvish_mysql_dump.sh
    • # dumps mysql databases locally to be included in remote dirvish backup cd /home/my_user/mysql_dumps mysqldump -u user1 --password=password1 --add-drop-table db1 > db1.mysql.dump mysqldump -u user2 --password=password2 --add-drop-table db2 > db2.mysql.dump chmod 600 *.dump

 Duplicate / Copy a dirvish backup disk

  • # -H stands for "preserve hardlinks"
    sudo rsync -avH --delete --stats --progress /media/disk/ /media/disk-1/

Automated replication to a backup disk

  • /etc/dirvish/backup-clone
  • #! /bin/bash
    #
    # daily cron job for the dirvish package
    # Klemens Ullmann-Marx 2010-01-31
    #
    ## Example of how to mount and umount a backup partition...
    
    MOUNTPOINT="/media/inseq_backup_1"
    
    mountpoint $MOUNTPOINT
    
    # $? is the exit status of the last bash command
    # ("mountpoint" in that case")
    # The exit status "0" in case of success, an integer between "1" and "255" in case of error
    if [ $? -eq 0 ] ; then
      echo "Good..."
    else
      echo "Trying to mount it now..."
      mount $MOUNTPOINT
      if [ $? -eq 0 ] ; then
        echo "$MOUNTPOINT successfully mounted!"
      else
        echo "Failed to mount $MOUNTPOINT!"
        exit 1
      fi
    fi
    
    #rsync -aHq --delete /srv/backup $MOUNTPOINT
    umount $MOUNTPOINT

Symlink latest backups to a directory for remote backup

  • /etc/divish/master.conf
    • post-server: /root/bin/latest_backup_symlink.sh
  • /root/bin/latest_backup_symlink.sh
    • #!/bin/bash
      #
      # 2009-10-12 by Klemens Ullmann-Marx
      
      BACKUP_DEST=/srv/backup_latest$DIRVISH_SRC
      
      rm -rf $BACKUP_DEST
      ln -s $DIRVISH_DEST $BACKUP_DEST

Remote Backup

 http://apt-get.dk/howto/backup/

Alter expire date

  • Modify dirvish/default.hist
    • Use "Never" for expires
    • 2016-01-21_08-35        2016-01-21 08:49:12     2016-01-21_08-02 Never
  • Modify 201x-xx-xx/dirvish/summary
    • Use "Never" for expires

Delete a snapshot

  • Delete the directory
  • Modify dirvish/default.hist
    • Delete correct line
    • Alter reference

Delete a file or directory from all snapshots (DANGEROUS!)

Example: delete a directory "public" from a vault /srv/backup/srv

  • find /srv/backup/srv -type d -name public -exec echo '{}' \;
  • Replace "echo" with "rm -rf" to actually delete the directories

Example limit the find directory depth to find only top level directories from the /var backup

  • find /srv/backup/var -maxdepth 3 -type d -name cache -exec echo '{}' \;

No unexpired good images

Situation: Remote backup of windows box. Some files had access denied. Furthermore the box was'nt up regularily, so a lot of images where missing.

Solution: Change "Status:" to "sucess" in an early backup (Note, this can expire a lot of images, and nearly all if e.g. the 1st of each month was missing!)

 

Dirvish vault can be on the same volume as the data to be backed up

Tested with

  • Data in /srv/test
    • test.txt, content "apple"
  • Vault in /srv/backup/test
  • dirvish --vault test
  • Edit /srv/test/test.txt to "pear"
  • -> /srv/backup/test/20xx-yy-zz/tree/test.txt is still "apple"

Conclusion: Hardlinks seem to be created only from one dirvish snapshot to the following. Hardlinks are not used in the initial initialisation.