« Back to list view

Rsync Cheat Sheet

Backup

rsync -av --keep-dirlinks --hard-links --numeric-ids --no-whole-file --inplace --info=progress2 --stats --human-readable /source /target

To produce minimal system load prepend:

nice -n19 ionice -c3 rsync ...

To produce minimal system load on the remote server use

rsync --rsync-path="ionice -c3 nice -n19 rsync ...

Note: to copy large file with fast internet connection: the fastest way is often to copy the file fully and directly (--sparse --whole-file, no compression)

Options:

  • -K = --keep-dirlinks = preserve directory symlinks
  • -H = --hard-links = preserve hardlinks (e.g. for dirvish backups)
  • --numeric-ids = don't map uid/gid values by user/group name
  • --no-whole-file = Sync only modified parts of big files. Rsync disables delta algorithm when syncing locally.
  • --progress = show progress during transfer
    • On newer rsync versions you can use --info=progress2 instead which gives a total percentage
  • --stats = give some file-transfer statistics after transfer
  • -h = --human-readable = output numbers in a human-readable format
     
  • -z = compress
    • Other compression algorithms can be used with --compress-choice=zstd
  • -v = --verbose
  • -vv = more verbose
  • -a =
    • -r = --recursive
    • -l = --links = copy symlinks
    • -p = --perms = preserve permissions
    • -t = --times = preserve modification times
    • -g = --group = preserve group
    • -o = --owner = preserve owner
    • -D = --devices --specials
      • --devices = preserve device files
      • --specials = preserve special files

Further interesting options:

  • --delete = delete files in the target which where removed in the source
  • --delete-excluded = delete files in the target which are in the exclude list
  • --ignore-errors = delete even if there are I/O errors. E.g. vanished files (sessions, temp data)
  • --log-file=/tmp/rsync.log = logging
  • --bwlimit=5m = limit the used network bandwidth to 5MB/s
  • -e'ssh -v' = show remote rsync command
     
  • --inplace =
    • This option changes how rsync transfers a file when its data needs to be updated: instead of the default method  of  creating a new copy of the file and moving it into place when it is complete, rsync instead writes the updated data directly to the destination file. This option is useful for transferring large files with block-based changes or appended data, and also on systems  that  are disk bound, not network bound.  It can also help keep a copy-on-write filesystem snapshot from diverging the entire contents of a file that only has minor changes.
  • --sparse = preserve sparse files
  • -W = --whole-file = do not use delta-transfer, always send whole file. Default for local paths.
  • --partial =
    • By default, rsync will delete any partially transferred file if the transfer is interrupted. Using the --partial option tells rsync to keep the partial file which should make a subsequent transfer of the rest of the file much faster.
    • When rsyncing from a  ssh remote host to local, this is a local option (not in server command options)
  • --ignore-existing = Do not update files that already exist on the destination

  • --exclude="/dir1*"

  • --msgs2stderr --debug=all5 = Debugging information

  • --itemize-changes = show only changes and what exactly changed

Non standard SSH port

rsync -e "ssh -p 2222" ...

Give ssh key

rsync -e 'ssh -i /home/pjharvey/.ssh/id_rsa_for_rsync' ...

Inlude / Exclude

http://unix.stackexchange.com/questions/2161/rsync-filter-copying-one-pattern-only

Copy only specific subdirectories

Copy only top level dirs which match a pattern ('example.com')

Example source directory:

rsync -avn --stats --progress --keep-dirlinks --hard-links --numeric-ids --human-readable --delete --include='/*.example.com/***' --filter='- *' /srv/backup/ /mnt/

--inlude with *** (3 stars) means to traverse in all subdirectories

--filter='- *' means to exclude everything

Copy only specific parts of the filesystem

Copies including subdirectories:

  • /etc
  • /home
  • /var/log/
  • /var/www

rsync -avn --keep-dirlinks --hard-links --numeric-ids --progress --stats --human-readable --include='/etc/***' --include='/home/***' --include='/var' --include='/var/log/***' --include='/var/www/***' --filter='- *' root@example.com:/ /srv/example_mirror/

Explained:

  • --include='/etc/***'    selects /etc + all subdirectories
  • --include='/var/log/***'   to sync only a subdirectory you need also to select the parent dir before with --include='/var'
  • The final --filter='- *'    rejects everything not defined

 

Example: only maildirs from homedirs

rsync -avz --stats --progress --filter='+ /home/*/' --filter='+ /home/*/Maildir/' --filter='- /home/*/*' root@example.com:/home ./

Example: only one subdir from a dir

rsync -av --info=progress2 --stats --human-readable --filter='+ /srv/nextcloud' --filter='- /srv/*' --delete --delete-excluded /source /target

Complete Linux Backup

--exclude="/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found/*"

Note: doesn't seem to work. better use --exclude-from="/path/to/rsync_exclude.txt"

rsync_exclude.txt for Ubuntu

nobackup/
/bin
/boot
/cdrom
/dev
/lib
/lib32
/lib64
/lost+found
/media
/mnt
/mount
/opt
/proc
/root/.cache
/root/.keyfiles
/run
/sbin
/selinux
/snap
/srv/backup
/sys
/tmp
/usr/bin
/usr/games
/usr/include
/usr/lib
/usr/lib32
/usr/lib64
/usr/sbin
/usr/share
/usr/src
/var/cache
/var/crash
/var/lib/amavis/tmp
/var/lib/apt/lists
/var/lib/lxcfs
/var/lib/mlocate
/var/lib/mysql
/var/lib/php/sessions
/var/lib/tmp
/var/spool
/var/tmp/
/initrd.img
/initrd.img.old
/swapfile
/vmlinuz
/vmlinuz.old

Rsyncing big files / virtual machine disk images

Test:

  • time rsync -a --stats --progress source.vmdk  /destination
    • Total file size: 14,360,313,856 bytes
      Total transferred file size: 14,360,313,856 bytes
      Literal data: 14,360,313,856 bytes
      Matched data: 0 bytes
      Total bytes sent: 14,363,819,927
      Total bytes received: 35

      sent 14,363,819,927 bytes  received 35 bytes  102,966,451.34 bytes/sec
      total size is 14,360,313,856  speedup is 1.00

      real    2m18.987s
      user    0m57.504s
      sys    0m11.678s
  • Boot and shutdown virtual machine to provoke changes in image
  • time rsync -a --stats --progress source.vmdk  /destination
    • -> Same result as above, because it was a local transfer, so rsync uses the --whole-file option per default
  • Boot and shutdown virtual machine to provoke changes in image
  • time rsync -a --no-whole-file  --stats --progress source.vmdk  /destination
    • Total file size: 14,360,444,928 bytes
      Total transferred file size: 14,360,444,928 bytes
      Literal data: 16,896,312 bytes
      Matched data: 14,343,548,616 bytes
      Total bytes sent: 17,379,339
      Total bytes received: 958,803

      sent 17,379,339 bytes  received 958,803 bytes  36,277.23 bytes/sec
      total size is 14,360,444,928  speedup is 783.09

      real    8m24.980s
      user    1m55.559s
      sys    0m17.257s
    • -> Only 17MB are sent, but it takes 8:30 instead of 2:20 minutes
  • Boot and shutdown virtual machine to provoke changes in image
  • time rsync -a --no-whole-file  --stats --inplace --progress source.vmdk  /destination
    • Total file size: 14,360,444,928 bytes
      Total transferred file size: 14,360,444,928 bytes
      Literal data: 16,536,816 bytes
      Matched data: 14,343,908,112 bytes
      Total bytes sent: 17,019,762
      Total bytes received: 958,803

      sent 17,019,762 bytes  received 958,803 bytes  51,440.82 bytes/sec
      total size is 14,360,444,928  speedup is 798.75

      real    5m48.298s
      user    1m53.624s
      sys    0m12.398s
    • -> A bit faster


       

Rsyncing sparse virtualisation images

Unfortunately “--sparse” and “--inplace” cannot be used together.

  • first time:
    • rsync --ignore-existing --sparse why ignore-existing???
    • rsync --sparse
    • When copying the file the first time, which means it does not exist on the target server use “rsync --sparse“. This will create a sparse file on the target server and copies only the used data of the sparse file.
  • subsequent times:
    • rsync --inplace
    • Use for VM images. Changes are made in place - the file is not transferred to a copy and then moved.
  • rsync -avz --sparse --stats --progress -e "ssh -p 2222" www.example.com">root@www.example.com:/var/lib/libvirt/images/image.img

Repeat rsync until it finishes correctly

RC=1  while [[ $RC -ne 0 ]] do    run-one rsync -a --partial .....       RC=$? done

Possible rsync problem with compression

rsync: connection unexpectedly closed (40371999 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [generator=3.1.1]

rsync 3.1.1 (Ubuntu 16.04 armhf) retreiving data from 3.1.0 (Ubuntu 14.04) 

Solutions:

https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503

Hardlink Snapshots

https://linuxwiki.de/rsync/SnapshotBackups

  • rsync -a --delete --link-dest=../backup.1 source_directory/ backup.0/

 

Compare files with checksumming

  • rsync --archive --checksum --dry-run --verbose --itemize-changes --delete --keep-dirlinks --hard-links --numeric-ids --no-whole-file --inplace --progress --stats --human-readable /mnt/srv/ /srv/

Performance

Make sure source disk has mount option "noatime"!

  • mount -o remount,noatime /mnt