Rsync Cheat Sheet
Backup
rsync -av --keep-dirlinks --hard-links --numeric-ids --no-whole-file --inplace --info=progress2 --stats --human-readable /source /target
To produce minimal system load prepend:
nice -n19 ionice -c3 rsync ...
To produce minimal system load on the remote server use
rsync --rsync-path="ionice -c3 nice -n19 rsync ...
Note: to copy large file with fast internet connection: the fastest way is often to copy the file fully and directly (--sparse --whole-file, no compression)
Options:
- -K = --keep-dirlinks = preserve directory symlinks
- -H = --hard-links = preserve hardlinks (e.g. for dirvish backups)
- --numeric-ids = don't map uid/gid values by user/group name
- --no-whole-file = Sync only modified parts of big files. Rsync disables delta algorithm when syncing locally.
- --progress = show progress during transfer
- On newer rsync versions you can use --info=progress2 instead which gives a total percentage
- --stats = give some file-transfer statistics after transfer
- -h = --human-readable = output numbers in a human-readable format
- -z = compress
- Other compression algorithms can be used with --compress-choice=zstd
- -v = --verbose
- -vv = more verbose
- -a =
- -r = --recursive
- -l = --links = copy symlinks
- -p = --perms = preserve permissions
- -t = --times = preserve modification times
- -g = --group = preserve group
- -o = --owner = preserve owner
- -D = --devices --specials
- --devices = preserve device files
- --specials = preserve special files
Further interesting options:
- --delete = delete files in the target which where removed in the source
- --delete-excluded = delete files in the target which are in the exclude list
- --ignore-errors = delete even if there are I/O errors. E.g. vanished files (sessions, temp data)
- --log-file=/tmp/rsync.log = logging
- --bwlimit=5m = limit the used network bandwidth to 5MB/s
- -e'ssh -v' = show remote rsync command
- --inplace =
- This option changes how rsync transfers a file when its data needs to be updated: instead of the default method of creating a new copy of the file and moving it into place when it is complete, rsync instead writes the updated data directly to the destination file. This option is useful for transferring large files with block-based changes or appended data, and also on systems that are disk bound, not network bound. It can also help keep a copy-on-write filesystem snapshot from diverging the entire contents of a file that only has minor changes.
- --sparse = preserve sparse files
- -W = --whole-file = do not use delta-transfer, always send whole file. Default for local paths.
- --partial =
- By default, rsync will delete any partially transferred file if the transfer is interrupted. Using the --partial option tells rsync to keep the partial file which should make a subsequent transfer of the rest of the file much faster.
- When rsyncing from a ssh remote host to local, this is a local option (not in server command options)
- --ignore-existing = Do not update files that already exist on the destination
-
--exclude="/dir1*"
-
--msgs2stderr --debug=all5 = Debugging information
-
--itemize-changes = show only changes and what exactly changed
-
>fcst...... > = from source to target f = it is a file (d=directory) c = checksum does not match (with -c option) s = size is different t = timestamp is different p = permissions are different o = owner is different g = group is different
- Also see great explanation http://andreafrancia.blogspot.com/2010/03/as-you-may-know-rsyncs-delete-options.html
-
Non standard SSH port
rsync -e "ssh -p 2222" ...
Give ssh key
rsync -e 'ssh -i /home/pjharvey/.ssh/id_rsa_for_rsync' ...
Inlude / Exclude
http://unix.stackexchange.com/questions/2161/rsync-filter-copying-one-pattern-only
Copy only specific subdirectories
Copy only top level dirs which match a pattern ('example.com')
Example source directory:
- hello/
- www.example.com/
- xavier/
- zebra.example.com/
rsync -avn --stats --progress --keep-dirlinks --hard-links --numeric-ids --human-readable --delete --include='/*.example.com/***' --filter='- *' /srv/backup/ /mnt/
--inlude with *** (3 stars) means to traverse in all subdirectories
--filter='- *' means to exclude everything
Copy only specific parts of the filesystem
Copies including subdirectories:
- /etc
- /home
- /var/log/
- /var/www
rsync -avn --keep-dirlinks --hard-links --numeric-ids --progress --stats --human-readable --include='/etc/***' --include='/home/***' --include='/var' --include='/var/log/***' --include='/var/www/***' --filter='- *' root@example.com:/ /srv/example_mirror/
Explained:
- --include='/etc/***' selects /etc + all subdirectories
- --include='/var/log/***' to sync only a subdirectory you need also to select the parent dir before with --include='/var'
- The final --filter='- *' rejects everything not defined
Example: only maildirs from homedirs
rsync -avz --stats --progress --filter='+ /home/*/' --filter='+ /home/*/Maildir/' --filter='- /home/*/*' root@example.com:/home ./
Example: only one subdir from a dir
rsync -av --info=progress2 --stats --human-readable --filter='+ /srv/nextcloud' --filter='- /srv/*' --delete --delete-excluded /source /target
Complete Linux Backup
--exclude="/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found/*"
Note: doesn't seem to work. better use --exclude-from="/path/to/rsync_exclude.txt"
rsync_exclude.txt for Ubuntu
nobackup/ /bin /boot /cdrom /dev /lib /lib32 /lib64 /lost+found /media /mnt /mount /opt /proc /root/.cache /root/.keyfiles /run /sbin /selinux /snap /srv/backup /sys /tmp /usr/bin /usr/games /usr/include /usr/lib /usr/lib32 /usr/lib64 /usr/sbin /usr/share /usr/src /var/cache /var/crash /var/lib/amavis/tmp /var/lib/apt/lists /var/lib/lxcfs /var/lib/mlocate /var/lib/mysql /var/lib/php/sessions /var/lib/tmp /var/spool /var/tmp/ /initrd.img /initrd.img.old /swapfile /vmlinuz /vmlinuz.old
Rsyncing big files / virtual machine disk images
Test:
- time rsync -a --stats --progress source.vmdk /destination
- Total file size: 14,360,313,856 bytes
Total transferred file size: 14,360,313,856 bytes
Literal data: 14,360,313,856 bytes
Matched data: 0 bytes
Total bytes sent: 14,363,819,927
Total bytes received: 35
sent 14,363,819,927 bytes received 35 bytes 102,966,451.34 bytes/sec
total size is 14,360,313,856 speedup is 1.00
real 2m18.987s
user 0m57.504s
sys 0m11.678s
- Total file size: 14,360,313,856 bytes
- Boot and shutdown virtual machine to provoke changes in image
- time rsync -a --stats --progress source.vmdk /destination
- -> Same result as above, because it was a local transfer, so rsync uses the --whole-file option per default
- Boot and shutdown virtual machine to provoke changes in image
- time rsync -a --no-whole-file --stats --progress source.vmdk /destination
- Total file size: 14,360,444,928 bytes
Total transferred file size: 14,360,444,928 bytes
Literal data: 16,896,312 bytes
Matched data: 14,343,548,616 bytes
Total bytes sent: 17,379,339
Total bytes received: 958,803
sent 17,379,339 bytes received 958,803 bytes 36,277.23 bytes/sec
total size is 14,360,444,928 speedup is 783.09
real 8m24.980s
user 1m55.559s
sys 0m17.257s - -> Only 17MB are sent, but it takes 8:30 instead of 2:20 minutes
- Total file size: 14,360,444,928 bytes
- Boot and shutdown virtual machine to provoke changes in image
- time rsync -a --no-whole-file --stats --inplace --progress source.vmdk /destination
- Total file size: 14,360,444,928 bytes
Total transferred file size: 14,360,444,928 bytes
Literal data: 16,536,816 bytes
Matched data: 14,343,908,112 bytes
Total bytes sent: 17,019,762
Total bytes received: 958,803
sent 17,019,762 bytes received 958,803 bytes 51,440.82 bytes/sec
total size is 14,360,444,928 speedup is 798.75
real 5m48.298s
user 1m53.624s
sys 0m12.398s - -> A bit faster
- Total file size: 14,360,444,928 bytes
Rsyncing sparse virtualisation images
Unfortunately “--sparse” and “--inplace” cannot be used together.
- first time:
rsync --ignore-existing --sparsewhy ignore-existing???- rsync --sparse
- When copying the file the first time, which means it does not exist on the target server use “rsync --sparse“. This will create a sparse file on the target server and copies only the used data of the sparse file.
- subsequent times:
- rsync --inplace
- Use for VM images. Changes are made in place - the file is not transferred to a copy and then moved.
- rsync -avz --sparse --stats --progress -e "ssh -p 2222" www.example.com">root@www.example.com:/var/lib/libvirt/images/image.img
Repeat rsync until it finishes correctly
RC=1 while [[ $RC -ne 0 ]] do run-one rsync -a --partial ..... RC=$? done
Possible rsync problem with compression
rsync: connection unexpectedly closed (40371999 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [generator=3.1.1]
rsync 3.1.1 (Ubuntu 16.04 armhf) retreiving data from 3.1.0 (Ubuntu 14.04)
Solutions:
https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503
- Install a newer version
- wget http://mirrors.kernel.org/ubuntu/pool/main/r/rsync/rsync_3.1.1-3_amd64.deb
sudo dpkg -i rsync_3.1.1-3_amd64.deb
- wget http://mirrors.kernel.org/ubuntu/pool/main/r/rsync/rsync_3.1.1-3_amd64.deb
-
Do not use -z (compression
Hardlink Snapshots
https://linuxwiki.de/rsync/SnapshotBackups
- rsync -a --delete --link-dest=../backup.1 source_directory/ backup.0/
Compare files with checksumming
- rsync --archive --checksum --dry-run --verbose --itemize-changes --delete --keep-dirlinks --hard-links --numeric-ids --no-whole-file --inplace --progress --stats --human-readable /mnt/srv/ /srv/
Performance
Make sure source disk has mount option "noatime"!
- mount -o remount,noatime /mnt