« Back to list view

bigsync

https://github.com/egorFiNE/bigsync

Allows to backup large files (eg. virtual machine image) to a slow media in a fast way.
Supports to read from raw device (/dev/hda1) or from file. Destination is always a file.
On the target a *.bigsync files with checksums is created

Does not support direct network support (e.g. ssh)

It works by reading blocks (--blocksize) and comparing them with checksums on the destination.
The destination block checksums are saved on the destination host in a file "mybigfile.img.bigsync".
 

Installation on Ubuntu (14&16.04)

Usage

  • Does not support ssh, use sshfs
  • bigsync -v --sparse --blocksize=10 --source /local/path/bigfile.img --dest /mount/destination/path/bigfile.img

Performance test via sshfs over 1GB LAN

Source machine: vm image on lvm snapshot

Target machine: duplicate of vm image


Initial sync (source mounted via sshfs, bigsync executed on target)

Fixing sparse file
Truncating file /srv/libvirt_images/root.bigsync.img to 115964116992
Total read = 108.000 Gb
Total write = 108.000 Gb
Total blocks changed = 7373
Elapsed 57m25s


Subsequent sync (on target)

Fixing sparse file
Truncating file /srv/libvirt_images/root.bigsync.img to 115964116992
Total read = 108.000 Gb
Total write = 0 b
Total blocks changed = 0
Elapsed 29m43s


Subsequent sync (on source)

Fixing sparse file
Truncating file /mnt/root.bigsync.img to 115964116992
Total read = 108.000 Gb
Total write = 0 b
Total blocks changed = 0
Elapsed 16m45s


Subsequent sync (on source, with 2 hours of modifications in image file)

Fixing sparse file
Truncating file /failover/mnt/spinpond/goldfish.ull.at.root.bigsync.img to 115964116992
Total read = 108.000 Gb
Total write = 4.175 Gb
Total blocks changed = 285
Elapsed 17m51s

 

 

Sync with   --blocksize=100

Total blocks changed = 1106
Elapsed 40m36s
 

Resync with   --blocksize=100

Total write = 4.175 Gb
Total blocks changed = 285
Elapsed 17m51s