rsync backup strategy

We use an external hard drive to store daily backups. At first it just made a big tar file of the home directory, which was really inconvenient and slow to restore from. What I wanted was to have an exact copy of the home directory, so I could view files normally, and restore files with a regular file copy. rsync lets me do exactly that.

The basic goal is to copy the entire home directory to the external hard drive each day, keeping say 14 days’ copies. This would require 14x the disk space, which I don’t have. The trick is to store each version of a file only once on disk. Each day any new or changed files are copied, but files that are unchanged are just a hard link to the same file contents from the previous day. So essentially a diff backup is done each day, but each day’s backup looks like a full copy, making it easy to find any file.

The diagram below shows backups of 2 files. Foo is updated on Tuesday, so Tues and Weds point to the new file. Bar is not updated, so all 3 backups point to the same file.

rsync is a handy program that makes this process very easy. It efficiently synchronizes two directories, locally or over a network. I developed a Python backup script based on this strategy. It uses a simple config file. Some files (like caches) I exclude from the backup using an exclude file. This technique was originally based on Mike Rubel’s website.

What are hard links?

Let me explain hard links if you’re not familiar with them. A filename is really just a pointer, called a hard link, to the file’s contents on disk. On Unix (and Mac OS X) there can be multiple filenames pointing to the same contents. Each hard link can be renamed or deleted without affecting other hard links. The contents are not deleted from disk until the last hard link pointing to them is deleted. (See diagram below.)

This is different than a soft link, created by "ln -s origfile newfile", where newfile points to the orig filename, which in turn points to the contents on disk. If origfile is renamed or deleted, newfile is left dangling, pointing to nothing. Windows shortcuts and Mac aliases are similar to soft links, though Mac aliases can often find the original file if it was just renamed.