Your backup is only as good as your last restore

Author: Kasper B. Graversen, 08/07/08
Keywords: Backup
Abstract: You should back up regularly. But you must also remember to do restores nearly as frequently. Your backup is only as good as your last restore. A restore attempt is comparable to running your test suites, and you don't run those only when bugs are discovered?
subscribe to my RSS feed


Bookmark and Share


Your backup is only as good as your last restore

I just read http://www.version2.dk/artikel/7843 (in Danish only, unfortunately). It's a story about pain. About how no attention to infrastructure can mean the complete ruin of a company. The story is about how Denmarks biggest search engine lost all its data, one morning at 4 am. No valid backups... only tears. It was a battle against all odds trying to restore the site before Denmark woke up to a blank search engine site! Fortunately for the implicated people, they managed to fudge the whole deal and partly recover data without anyone noticing it. The story inspired me to reflect upon the lessons I've learned in industry. I've basically been in the same situation. Fortunately on a much smaller scale, but never the less, with a similar potential heavy impact on the company.

More than anything else, the lessons learned are that your backup is only as good as your last restore! If you only backup and never try restoring it on a fresh system, you are in reality often no better of than not doing any backing up at all! You dead're when it comes to fulfilling any reasonable up time contracts. With no regular restore:
  • You never realize all the files you are forgetting to back up. Did we really store our source code in two different folders??
  • You have no procedure to follow. Recovery steps will be ad hoc, even flawed. Steps may easily be forgotten, requiring yet more re-recovery and you are feeling insecure and unsafe. Recovery progress will be slow.
  • You never realize all the infrastructure you thought you did not need to back up, but is needed in case of a crisis. Maybe you backup your source code, but what about all the external libraries, configuration files, 3rd party applications or shell scripts that keeps everything hanging together.
  • You never realize that your incremental restore is really expensive and takes 12 hours to resolve.
Well... You never realize any of this before it's too late!

I have worked in a very professional company. It was heavily data driven and so proper priority was given to backups plans and procedures. Frequently "emergency restores" were executed (launched as random days at random times) to ensure the support team were able to do the restore, and were comfortable. What I think the real lesson from that experience was, that despite frequent restores drills, the restores did not always go 100% smoothly, or did not complete 100%!. A drill was not only "just a drill", effectively it was a test of the back up, and lessons were learned, procedures and settings adjusted as a result. Actually, I think a back up drill is quite comparable to running you test suites! They must be run often in order to be worth anything.

Your backup is only as good as your last restore. Ensure your company prioritize backing up and restoring...



Comments

If you have any comments to this article, please drop me a mail at firstclassthoughts at gmail dot com please indicate if I can't publish whole or parts of your comment on the site.


If you like this site consider subscribing to my RSS feed or how about subscribing by Email.


Help spread the word

Share this post on your favorite social bookmarking sites:
If you enjoyed this article, found it thought provoking, educative or otherwise good, please link to this page from your page or social bookmarking page. If you have any texts you think are worth publishing on First Class Thoughts, don't hesitate to send me a mail! Quality always welcome.


Bookmark and Share


The most recent contributions
28/07/09 Magic in mathematics II Fun with the number cyclic numbers, and specifically with 142857 as it is the smallest of such numbers.
13/07/09 My top 8 time-saving Firefox shortcuts This article presents my favorite top 8 time-saving shortcuts in Firefox 3.0 and Firefox 3.5. Get to know these and you'll be saving a lot of time. They have been ordered by "the element of most surprise"
20/05/09 Board Game Jungle speed / Arriba Review of the cool game "Jungle Speed" aka. "Arriba".
16/05/09 Danish Twin words "Twin words" are words that not only have multiple meanings, they must be composed next to each other in meaningful sentences. This article explores the concept of twin words.
Nothing of interest? Try browsing the entire article archive...