My Backup System
Last Updated: 8/12/2005

I've put together a fairly simple but effective backup solution, and I thought I would share the design so that others who are interested can do something similar. The system is completely automated, with the exception of manually swapping disks with an off-site copy about once a week, and occasional (annual) cleanup of the backup drives.

This solution does require a little bit of scripting, but don't be afraid of that. I’ll walk you through each step of creating the script and let you know where you can make changes specific to your environment.

First, a little information about the way my computers are set up. I have several computers in the house for my wife and I, and to make things easier to manage, we store all our data on a single server, and that is the focus of my backup system. However, the steps outlined here apply just as easily to a single desktop computer as long as it is left running overnight (when the backup occurs).

This setup requires the following:

  • A computer running Windows 95, Windows 98, Windows ME, Windows 2000, Windows XP, or Windows 2003
  • 1 available USB (preferably USB 2) or Firewire port
  • 2 hard drives (each equal or greater than the size of the data you want to back up).
  • 2 external USB or Firewire hard drive enclosures*

*You can purchase ready-made external hard drives already in enclosures, but I have found that it is less expensive to buy the enclosures separately and put the hard drives in them. You may also find it easier to upgrade to larger drives later if you purchase the drives and enclosures separately instead of buying a pre-built solution that may not be easily upgradeable.

Note that some pre-built external drives include software to perform scheduled backups. While you can use that software if you have it, I have seen compatibility problems with some versions of that type of software. I will be describing the use of a few built-in Windows features to accomplish the same thing.

See Also: Backing Up Your Computer

Step 1: Set up the hardware

Install the hard drives in the external enclosures per the instructions that should have come with the enclosures. Connect each one to your computer (one at a time) via USB or Firewire (whichever applies in your case), and make sure your computer can see and recognize the drives. Format the drives if necessary, creating one full-size partition on each one.

Note the drive letter that your computer assigns these drives. It should be the same letter for both drives (they should not be connected at the same time). You will need to remember this drive letter when you create the script.  In my examples below, I will be using drive F.

Step 2: Identify the data you want to back up

Make note of the locations of the files and folders you want to back up on your computer. An important key here is that you don't need to back up your entire system, as you might think. In fact, backing up your entire drive does not guarantee that you can simply restore it from that backup and it will work just fine.

Your operating system and programs can all be re-installed from the original disks. The really important files are your documents, photos, and other files that you couldn't get from anywhere else if they were to be lost.

It's a good idea to consolidate this information as much as possible, and that will make it easier to back up. As you will see when we begin creating the script, the fewer locations you have your data stored in, the simpler the backup script will be. If you can keep everything in one folder, such as "My Documents" or some other location you specify, that would be ideal.

Note that some programs, especially e-mail programs like Outlook and Outlook Express, store their data in special data files in some (usually obscure) location on your computer. You should be able to change the location of these data files to fit under your specific directory, or at least note the location so that you can include it in the backup script. Check the help files or documentation for the individual programs to find the location of these files.

If you have multiple folders that you will be backing up, choose one to use for the next step, and we’ll add the rest later.

Step 3: Copy files with xcopy

We're going to be using a Windows utility called xcopy. Xcopy is a command-line program, meaning it has no user interface to look at and click on – you just type commands (either manually or via script) and it performs a task and (optionally) returns some feedback about what it is doing or has done.

To start with, let's run xcopy and see exactly what it does. Click your Start menu and choose "Run…" In the box that opens up, type "cmd" (without the quotes), and press enter or click "OK". You should be presented with a black window, called the command window. There will be a command prompt, which looks something like this:

C:\>

If the letter at the beginning is different, or if there is extra path information (such as C:\My Documents\>), that's ok.

At the command prompt, type the following line, replacing “C:\Documents” with the location of the folder you want to back up, and “F:\Documents” with the location on your external drive that you want to copy your files to. At the end of the line, press enter.

xcopy /C /D /E /G /H /I /K /O /R /V /X /Y "C:\Documents" "F:\Documents"

What should have happened is xcopy read all the files in the first folder you specified, and made a copy of those files in the second folder you specified – and now you have a backup of those files.  The output should tell you what files it copied.

But what are all those slashes and letters? Those are called “flags”, and they turn on or off certain options in xcopy. For instance, the /D flag tells xcopy to only copy files where the date of the source file is newer than the date of the destination file. Since this was the first time you ran this command, there were no destination files, so xcopy copied everything. But, if you run that same command again, you should see the following:

0 file(s) copied

What happened here was xcopy looked at all the files you wanted to copy, and realized that the copies on your computer are not any more recent than the files on your backup drive, so it didn’t bother to copy them again. If you run the same command again, but remove the /D flag, then xcopy will re-copy all the files, no matter what the dates are.

Each of those flags represents another option. To see a list of all the options and what they do, type the following line at the command prompt and press enter:

help xcopy

Now you can see what each option does, and adjust the flags accordingly. I’ve listed the flags that I use for my backup, but you may want to use some different options. For instance, if you don’t want to see the names of every file it copied – just the total number, you can use the /Q flag to suppress the filenames. Try running xcopy with different flag combinations and see how it works. Just be careful what you copy and where you copy it – you don’t want to overwrite new files with older ones!

Step 4: Create the script

So, now you know what xcopy can do, but let's look at a way to automate this a little bit. Instead of typing all that into a command window each time you want xcopy to do something, you can script it using what is called a "batch file". A batch file is nothing more than a text file that has commands in it just like what you typed at the command line. When you execute (double-click) a batch file, it runs those commands, one after the other. Anything you can do at a command line, you can script in a batch file and combine it with other commands – you can even have batch files that execute other batch files. Once you get familiar with batch files, they can be very powerful tools.

Open up your favorite text editor (Notepad will do just fine), and type out the same command we used above. When you save the file, though, make sure you save it with a .bat extension instead of the default of .txt. Save it as something like "backupscript.bat".

Now find that file and double-click it. You will briefly see that same black command window pop up, and it will execute your xcopy command and disappear. Congratulations – you just created and ran your first batch file.

At this point, you can add additional xcopy commands (one on each line) for any other folders that you need to back up. Remember that using the /E flag copies all files and subfolders within a folder, so you don’t need to create a command for each subfolder if you are using this flag. You can also have xcopy copy a single file by specifying a filename instead of a folder name, like this (note that you can't use the /E flag when copying a single file):

xcopy /C /D /G /H /I /K /O /R /V /X /Y "C:\Data Files\Document.doc" "F:\Data Files\"

Once you've got everything listed in your script, you are welcome to skip to the next step if you are satisfied. However, I'm a little bit picky about the way my script looks and runs, so I'm going to enhance this thing just a bit.

First, I'm going to add that /Q flag so that all the filenames are not listed as they are copied, since I can't read them that fast anyway.  However, I do want to see some information about what is going on when my script is running. Yes, I know this will ultimately be a scheduled job that will run in the middle of the night, but I still might want the output to be a little more descriptive, in case I want to run it manually some time.

I'm going to add some echo commands to my script. Echo is another Windows utility, but all it really does is repeat what you tell it. You can use Echo to write things out to the screen while your batch file is running, to give you some visual clues about what it's doing.

Add a couple of echo statements to your file, like this:

@ECHO OFF

echo Running Backup Script...
echo ---------------------------------------------------------------

echo Backing up "Data Files"...
xcopy /C /D /E /G /H /I /K /O /Q /R /V /X /Y "C:\Documents" "F:\Documents"
echo ---------------------------------------------------------------

echo Backing up "Photos"...
xcopy /C /D /E /G /H /I /K /O /Q /R /V /X /Y "C:\Photos" "F:\Photos"
echo ---------------------------------------------------------------

echo Backing up "Music"
xcopy /C /D /E /G /H /I /K /O /R /V /X /Y "C:\Music" "F:\Music"
echo ---------------------------------------------------------------

echo Backup Script Complete

Adding "@ECHO OFF" to the beginning of the script tells the system not to echo the actual commands in your script (xcopy with all the flags), but to only display the output of the commands, along with your defined echo statements.

Now run your batch file again, and watch the output on screen. You will see your echo statements in between the results of your xcopy statements so that you have a better idea of what is going on.

It is not necessary to name your destination folders the same as your source folders.  You can call them anything you like, and even organized the files and folders differently on your backup drive.  However, naming them the same makes it easy to know what you have backed up and to find files when you need them later.

Step 5: Schedule the script

Now we're going to use another Windows utility, called Task Scheduler. This utility, as you might imagine based on its name, allows you to schedule a task (usually some executable) to run at a particular time. We're going to use Task Scheduler to schedule your batch file to run in the middle of the night and make a backup of your files, so that you don’t have to remember to do it each day.

Go to Start > Programs > Accessories > System Tools > Task Scheduler

Click on Create New Task. A wizard will guide you through the setup of your task. Select the batch file you created and choose a schedule for it to run. I recommend you have it run every night around 2:00 AM, or some other time when there will be no one using the system.

You may be prompted to enter a username and password for the script to use when it runs. If you have multiple user accounts on your computer, make sure the account you use has permission to access all the files you want backed up. If necessary, create a separate script and a separate task to back up each user's files.

You're almost done! You've got a scheduled, automatic backup of all your critical data happening on a nightly basis. Just don't forget the next step

Step 6: Swap your drives often

Your backup solution, as it stands right now, is pretty good. If you accidentally deleted some files, or made changes to some files you didn't mean to, you could get a copy from the night before. If your hard drive were to completely fail, you could buy a new drive, re-install your software, and then copy your data from your backup disk that is only one day old.

But what if something more drastic were to happen? What if a virus overwrote a bunch of your files, and you did not realize it until the next day, after your backup script made a new backup of those bad files? What if (God forbid) your house were to catch fire? What good would your backup – sitting right there next to your computer – be to you then? Not much.

That's why you purchased two hard drives and two enclosures. You should make an initial backup on both drives, and then take one away from your house. Store it in a safe deposit box, with a trusted friend or family member, or in a locked drawer at your office. Just remember to keep it safe – anyone who gets that disk potentially has access to all your documents and files.

On a regular basis, probably once a week or so, take the backup drive from your house, and go swap it with the one you have stored off-site. This way, you always have 3 copies of your data (the computer itself and 2 backups), and those 3 copies are NEVER all in the same place, and no copy is more than 1 week old. This will significantly reduce the risk of a disaster that wipes out all your data.

Once you get into a routine like this, the backup process becomes effortless.

Step 7 (optional): Logging

For a few months, I used the exact setup that you should have now, and was happy with it. At one point, though, I decided I'd like to be able to easily tell when a backup occurred, what was done, and how long it took. I'd also liked to be able to tell how old the backup on my off-site drive is (in case I forget to swap them regularly).

If this doesn't interest you, you are welcome to skip down to the "Things to remember" section.

First, I figured out how to write the output of my batch file to a log file instead of displaying it on screen. To do that, you call the batch file like this:

"C:\backupscript.bat" > "F:\Backup_Log.txt"

The filename on the end is where the output will be written. Using ">>" instead of ">" will cause the command to append the output to the end of the file (preserving what is there) instead of overwriting it.

In order to schedule this, what I did was create a second batch file that contained the above command, along with an echo statement to describe what is happening (in case anyone happens to see the job running).  I changed the Task Scheduler job to execute the second batch file, which will in turn execute the first batch file, and write the results to a log file.

Now I've got the output of my backup job, including the echo statements, being logged to a file. By checking the date on that file (when it was last modified), I can quickly see when the last backup was performed, and by checking the contents of that file, I can see what was done.

To make this log even more descriptive, I made a few changes to the first batch file:

  • First, I added some extra echo statements with dashed lines to help delimit the individual actions.
  • Second, I removed the /Q flag so that the names of the files that I'm backing up would be included in the output, and therefore written to the file.
  • Third, I added some timestamps to some of the echo statements, so that the date and time would be written to the log file as each command was executed.

The final results of my files are printed below.

File #1, named incremental.bat (which will be executed by file #2):

@ECHO OFF

echo ---------------------------------------------------------------
echo Incremental Backup Script Started %DATE% %TIME%
echo ---------------------------------------------------------------

echo Backing up "Documents"
xcopy /C /D /E /G /H /I /K /O /R /V /X /Y "C:\Documents" "F:\Documents"
echo "Documents" backup completed at %TIME%
echo ---------------------------------------------------------------

echo Backing up "Photos"
xcopy /C /D /E /G /H /I /K /O /R /V /X /Y "C:\Photos" "F:\Photos"
echo "Photos" backup completed at %TIME%
echo ---------------------------------------------------------------

echo Backing up "Music"
xcopy /C /D /E /G /H /I /K /O /R /V /X /Y "C:\Music" "F:\Music"
echo "Music" backup completed at %TIME%
echo ---------------------------------------------------------------

echo ---------------------------------------------------------------
echo Incremental Backup Script Completed %DATE% %TIME%
echo ---------------------------------------------------------------

File #2, named backuplog.bat (which will be executed by the Task Scheduler):

@ECHO OFF

echo Running Backup Process.
echo Logging output to F:\Backup_Log.txt

"C:\Backup Scripts\incremental.bat" >> "F:\Backup_Log.txt"

My log file looks like this:

---------------------------------------------------------------
Incremental Backup Script Started Fri 06/17/2005 2:10:00.49
---------------------------------------------------------------
Backing up "Documents"
D:\Documents\Document1.doc
D:\Documents\Document2.doc
D:\Documents\Document3.doc
3 File(s) copied
"Documents" backup completed at 2:10:12.17
---------------------------------------------------------------
Backing up "Photos"
D:\Photos\2005\June 2005\IMG_5686.JPG
D:\Photos\2005\June 2005\IMG_5687.JPG
D:\Photos\2005\June 2005\IMG_5688.JPG
D:\Photos\2005\June 2005\IMG_5689.JPG
D:\Photos\2005\June 2005\IMG_5691.JPG
D:\Photos\2005\June 2005\IMG_5692.JPG
D:\Photos\2005\June 2005\IMG_5693.JPG
D:\Photos\2005\June 2005\IMG_5694.JPG
D:\Photos\2005\June 2005\IMG_5695.JPG
10 File(s) copied
"Photos" backup completed at 2:10:35.47
---------------------------------------------------------------
Backing up "Music"
0 File(s) copied
"Music" backup completed at 2:10:38.46
---------------------------------------------------------------
Incremental Backup Script Completed Fri 06/17/2005 2:10:39.08
---------------------------------------------------------------

One more optional tweak is to make the scheduled batch file write the output from the main batch file to a new log file each day (with the date set as part of the name). This will make it a little easier to delete older logs as they build up on your drives, or to find the logs from a particular day if necessary. Instead of one large log file, you will end up with many smaller files.

I won't explain the reason for the syntax to do this – just trust me. Modify your second batch file (the one that is scheduled) to look like this:

echo Running Backup Process.
echo Logging output to F:\Backup_Log_%date:~10,4%_%date:~4,2%_%date:~7,2%.txt

"D:\Backup Scripts\incremental.bat" >> "F:\Backup_Log_%date:~10,4%_%date:~4,2%_%date:~7,2%.txt"

Now your log will be written to a date-specific file such as Backup_Log_01_01_2005.txt

Things to remember:

Never bring home your off-site copy to swap it with your local copy. Always take the local copy to swap it with the off-site copy, so that you never have all three copies in one place.

Some files cannot be copied while they are in use, which is why it is not usually possible to completely back up your entire computer. Make sure to shut down your e-mail and other unnecessary applications at night so that the backup script has full access to the files it needs to copy.

Test your backups often.

Hard drives will fail eventually. Every 3-5 years, you should replace the drives in your computer as well as the drives you are using for backup.

If you move your batch files, you will need to modify the scheduled job to point to the new location, and you may need to modify the files themselves (if you added the logging option in step 7)

If you create new directories that you need backed up, be sure to add them to your scripts as well.

If you delete a file from your computer, it does not automatically get deleted from your backup drives. If it is something very personal or private that you are worried about someone else finding, you will need to manually delete it from both backup drives.

Over time, if you delete a lot of files from your computer, you will have a lot of unnecessary files on your backup drives. The easiest way to correct this is to periodically (once a year is probably enough) delete all the files from your backup drive and let the backup script make a fresh copy of them all. Just make sure you only do one drive at a time this way, and double check that the backup is successful on the first drive before swapping your drives and doing it again.