I love backup. I know that sounds weird, but I love backup. Before most information was digital, backing it up was incredibly hard. Consequently, the data of our lives was always in danger of being destroyed. Now that most data is digital, we have the ability to back it up so that it can survive extreme situations.

I grew up in Slidell, Louisiana, just outside of New Orleans. My parents still lived there when Hurricane Katrina hit. They got out and were safe, but their house flooded with about 3 feet of water. Like most families, they had books, papers, and pictures of my childhood throughout the house. Much of these were lost. I don't want to risk losing pictures and video of my kids or the code that I've spent years working on.

The good news is that a huge portion of my life, both personal and professional, is digital. I can backup this digital data and be comforted that it could survive flood, fire, theft, or hack. So, I love backup. And because I love backup, I want to do it well.

My system

My system of backing up data includes two main backup locations. The first is local, complete, and high integrity. The second is remote (in the cloud). I'm going to talk about what I do first, and then talk about the principles behind this setup afterward.

Local backup

I have a backup system running CentOS Linux. In addition to the little system drive, it has 5 1.5TB drives in a software RAID-5 array, giving 5.4TB of usable space. Each night, I have a cron job/launchd job on my systems run rsync to create a new snapshot on the backup system. Each snapshot is a complete backup of the system, but I use rsync with the --link-dest option, which means unchanged files will just be hardlinks to the already existing file rather than a new file. This saves disk space, allowing me to keep lots of snapshots around.

Each day, after the backups are complete, a python script run from cron on the backup system processes the snapshots. I keep the last 7 days, one snapshot from each of the last 12 months, and one snapshot from each year since I started this. The script will remove old snapshots that aren't in this list.

To protect one system's snapshots from another, and prevent snapshots from being modified after the fact, I have a custom SELinux policy on the backup system. When each client system logs in, it gets a unique SELinux type. SELinux policy only permits it to backup to its backup directory, meaning it cannot mess with another system's backups. Also, each day when the server processes the backups, it relabels the new snapshots to a type that can only be read and linked to. So, subsequent backups cannot modify the contents of a previous snapshot.

Remote backup

I use CrashPlan Family for remote backup. I run the CrashPlan agent on all my machines (even headless ones) and backup everything to the cloud. It can certainly take a while to get 4TB up there initially, but the service runs on Mac and Linux and does not have restrictions on silly things like file size like many of its competitors.

Core principles

Backup data needs to be live

Many people use backup strategies that involve making a backup and storing it away. I used to do this, but stopped for two reasons. First, media can degrade, and if it's not live there's no way to detect this. I've burned CDs with backup data, only to find out years later that they had degraded and could no longer be fully read. The second reason is that standards change. I used to back things up to a Zip drive, only later to discover that I couldn't use these backups any more because no one (myself included) used Zip drives any more.

Backups should be automated

I don't clone to an external hard drive as part of my backup system (though I will occasionally use SuperDuper to clone a drive before I make a drastic change). I've tried to do this, but it's way too hard to remember to plug the drive in. So, I do only network backups. They just happen, and I don't have to take any action (though I do get emails saying it's been done).

Keep lots of versions

My backup is a bit of an archive as well. This is because I've found that the largest threat to my data is me. I delete things from my systems, thinking I don't need them any more. I'm often wrong, only to discover years later that I really did need that thing. So, I think it's important to keep a lot of old versions, which is one reason I don't ever delete my yearly snapshots. This is one reason I have local backups in addition to the cloud. No cloud backup service I've found will keep this many versions for this long.

Think about attacks

When planning backup strategies, people usually think about hard drive failures or natural disasters, but they often forget about the possibility of an attack. If someone successfully attacks your computer and corrupts your data, you don't want them to be able to alter your previous backups. This is a problem for simple solutions such as an external hard drive or a network share. If an attacker can corrupt your system, they can also corrupt your backups.

Backup everything

Most people backup things selectively. They backup only their documents, or specific directories, or perhaps their home directory. My advice - backup everything. That means everything on your drive(s). Why? Do you need everything? No. However, I've found that I always miss something in trying to list what needs to be backed up. If I select certain directories, inevitably I will add another directory later that needs to be backed up and forget to do so. Backing up everything solves this.

You may say that I'm wasting lots of space backing up things that can easily be reinstalled. System software and applications can be grabbed from install DVDs or the Internet, so I shouldn't waste the space. Well, if you're like me, your user data that needs backup is far, far larger than this. I have 4TB worth of used storage across my systems right now, Of that, less than 100GB is system software and applications. I could avoid backing this up, but I'd rather be sure I've got everything and use an extra 2.5% of storage.

Think about restoring

Backup to the cloud is outstanding, but it is slow. I average one dead hard drive per year between my systems. A system with a lot of data would take weeks to restore. CrashPlan (like other similar services) does option to overnight me a hard drive for restoring, but it's quite spendy. Local backups mean I can get back up and running much quicker.

Think about who controls your data

While it's important to trust any backup provider you use to treat your data appropriately, I'm more worried about them going out of business. If CrashPlan goes out of business, then those backups are gone. This is yet another reason I have local backups as well. If CrashPlan goes away, I still have some sort of backups while I get a new cloud backup solution going.

Yes, I'm crazy

I realize that this may all seem crazy, and it certainly is somewhat extreme. Hopefully this has inspired you or caused you to think a bit more about backups. If you're not backing up today, you need to start. The easiest thing to do is just sign up for CrashPlan or a similar cloud backup service. It's easy, relatively cheap, and can survive fire, flood, and most stupid mistakes. And remember, the ability to backup is a great attribute of the digital age, so be thankful for backups.