Recovering From Disaster

Filed as Features on June 16, 2008 9:21 am

Repost This

Thursday June 12, 2008 is a night that will live in infamy around my house. It was when I discovered that my main site, Plagiarism Today, was down due to a storage failure at my host.

As dusk turned into dawn, the situation grew more and more bleak. With the storage failure becoming increasingly serious and backups failing to restore the site, it could have easily been a near-total loss for the site.

Yet despite almost no help from my then-host, I was able to get the site back up and running, almost completely up to date, within a few hours or realizing the magnitude of the situation.

However, the ordeal taught me a great deal about recovering from disasters and how to improve next time something bad happens. Lessons I hope others will not have to learn the hard way.

Before the Disaster

To successfully rebound from a hosting (or other) disaster, you need to prepare in advance by taking the following steps.

  1. Backups: Backup religiously. Databases should be backed up daily at least, WordPress users can use the WP-DB-Backup plugin to automate the process. Backups should never be stored on the same server as the site and one should also be sure to back up themes, plugins, images and anything else unique to the site. Other backups should be done at least weekly.
  2. Know Your Software: Do you know how to install WordPress, MovableType, Drupal or whatever software you are using? One-click installs are great, but in a disaster you may be asked to do it by hand. Be prepared if needed.
  3. Alternate Hosting: With the disaster at Plagiarism Today, I simply moved the site back to its previous host. Having an alternate hosting account, perhaps an inexpensive account on GoDaddy can save a lot of trouble at little cost.
  4. Offloading Media: By offloading images and other media, you can reduce the amount of material you have to move. however, one still has to maintain backups of the material stored there are no service is perfect.
  5. Email Preparation: If your site goes down, how will you get email? I moved my email to Google Apps to avoid that issue. Either set up an alternate email account or offload your mail services to a trusted company.

Though no amount of preparation can make a major outage an easy process, it can certainly mitigate the disaster and get you up on your feet quickly.

During the Outage

Last week, just before Plagiarism Today went down, Chris Garrett wrote an excellent piece about what to do during an outage. Specifically, he explains what to do in order to determine the cause of the outage and productive ways to spend your time.

When you discover the cause of the outage, however, you have to make a decision about what action you’re going to take. Usually, you have just three options.

  1. Wait it Out: The quickest way to resolve most outages is to simply wait. Most downtime is not catastrophic and only lasts a short period of time. If that is the case, the best thing you can do is nothing at all. Enjoy some free time.
  2. Fix the Problem: If the outage is caused by your own actions, you need to fix the issue either by removing the plugin, restoring from a backup or otherwise fixing the issue. Though many hosts have great support teams that will be happy to help, they are often overworked and will not be able to help for many hours. Thus, it is usually quicker to fix the issue yourself if you have an idea of what caused it.
  3. Move: Catastrophic server failure, hacking or other major disasters may create a situation, was with Plagiarism Today, where it is faster to move the site to a new host than it is to try and repair things at the old. Since DNS records can take many hours to propagate out, it is important to use this as a last resort, but it can be a lifesaver, especially with good backups.

The good news is that most outages are not caused by anything that should worry bloggers. The Internet tends to be fragile at times and, for the most part, an outage does not equal a major problem.

As long as you keep in touch with your host, stay on top of planned outages, subscribe to their alerts and generally watch your site, the odds of experiencing the kind of panic I experienced last week is pretty low.

After the Outage

Once your site is back up, your job is not necessarily over. There is still several things to do to ensure that things are running smoothly.

  1. Check Everything: It is not enough to see your home page and walk away. Check and make sure your images are up, ensure that your RSS feed is working, test any plugins or services that you use, send yourself a test message through your contact form, etc. Check every critical site function before moving on.
  2. Enlist Friends: Don’t just do it yourself, get friends, especially those in other geographic locations, to test it with you. This ensures that all of the connections are working and helps make sure that all features are tested completely.
  3. Relax: If everything is working, take a few moments and relax. Grab a nap, get something to eat, plays some video games, whatever. A downtime is a very tense period for most bloggers so it is important to give yourself some mental distance from it before moving on to other work.

Generally speaking, the “after the outage” portion is the easiest and most pleasant, but it is still very important. After all, one never knows what accidental damage might have been done getting the site back online. While having some functionality is better than none, there is little point in settling for a partially-working site.

Conclusions

I am no going to lie and say that I enjoyed the extended outage or the panic it caused, but I will say that it could have been much, much worse. Given what happened, I could have lost almost everything and I feel lucky to be back in business as quickly as I was.

The silver lining in the cloud was the outreach by my readers and friends who offered help and support during the storm. It not only touched me deeply but motivated me eve more to get the site back up after seeing how much it meant to my readers.

My thanks goes out to everyone who offered their help during that time. I’m hoping that, by sharing my experiences here, that can help others who find themselves in a similar position later.

Tags:

This post was written by

You can visit the for a short bio, more posts, and other information about the author.

Submissions & Subscriptions

Submit the post to Reddit, StumbleUpon, Digg or Del.icio.us.

Did you like it? Then subscribe to our RSS feed!



  1. By Michael Clark posted on June 16, 2008 at 1:22 pm
    Want an avatar? Get a gravatar! • You can link to this comment

    One thing you should do after moving your site to a new host/server is put up a brief post saying something like “This is the new server.” Then when your friends from around the world visit the site, they’ll know they are at the new site, not a cached version of the old site.

    Reply

  2. By Jonathan Bailey posted on June 16, 2008 at 3:57 pm
    Want an avatar? Get a gravatar! • You can link to this comment

    Good point. I usually put a small note in the footer saying “This is .” So that visitors know where I am. Right now it is a bit messed up due to the craziness but that is a very good point.

    Excellent suggestion!

    Reply

    Your words are your own, so be nice and helpful if you can. If this is the first time you're posting a comment, it might go into moderation. Don't worry, it's not lost, so there's no need to repost it! We accept clean XHTML in comments, but don't overdo it please.

    Current day month ye@r *