Today is not your lucky day. The production server went down due to a hard disk error on the VM's host machine. You don't have a high availability setup because running additional servers 24x7 is quite costly. However, you need a new machine quickly.

In this blog post, I will tell you how we deploy brand new production machines in under 20 minutes. First, I'll talk about why it is so important to be able to deploy servers quickly.

Benefits of rapid deployment

  • Better testing environments. If you want to try Memcached out on the site, you want to reproduce the current production environment quickly. You are, after all, just trying it out, and you don't want to lose a day on it. If you can run one command and have the server running, the experiment is much more feasible during your busy work week.
  • Better development environments. If you just hired a new developer, and you want them to fix a minor bug on a site you're maintaining, they shouldn't have to waste time setting up the environment before fixing the bug. They shouldn't need to know about all the legacy baggage the site has, and they shouldn't have to pester you about how to set up every component of the site.
  • Better documentation. Instead of writing extensive documentation on each step of the deployment process, you can write a deployment script. Well-written code can rarely replace good documentation, but this is one case where relying on the code is suitable. With minimal documentation effort, you should be able to roll out of bed and deploy a new copy of your most mission-critical production sites in minutes.

Steps to Rapid deployment

Set up your Chef infrastructure

At Evolving Web, we use Chef, a configuration management tool. With Chef, you manage all your configurations in one git repository, and you write them in Ruby. This lets the Chef server build a new server or update stale configurations automatically. For example, Chef will occasionally add new public keys to our authorized_keys files as the team grows.

With Chef, your site's environment is reproducible and standardized. When you're attempting to reproduce bugs, it's good to know that your development VM is the same as the one running the production server.

Make a site-specific cookbook for launching the site

We use site-specific cookbooks to launch a specific site. This will use a rotating read-only SSH key to pull code and configuration from a git repository. Once Chef builds the site, you can sync a database from backups or the production server.

Document

Write a step by step guide that anyone can read without messing up. This takes clarity and detail. Having a reproducible process for creating the VM makes this much easier; you can document as you build a test VM with VirtualBox on your own desktop. If you find that a certain part of the process doesn't work properly, you'll notice while documenting. You can fix the Chef script and go through the process again. Since it's a short process (especially with VirtualBox templates), it's not a problem to start over.

Test the process with a beginner

Find the newest recruit on the team and send them a link to your documentation page. Tell them to set up a server in 30 minutes. Can they do it without your help? If so, you've succeeded. If not, it's back to the drawing board. Make those docs shine.

Having good documentation for this makes getting new team members up to speed much faster. Passing your deployment process down generation by generation through oral tradition is not how the professionals do it.

You're done!

Now you can rest assured that you can get that site running again with no hassle. The entire development process should be more pleasant too; you can boot up a development VM in half an hour, and make it a clean environment separate from all the experimental hacks you do on your desktop.

Resources

Photo credit: "sheep" by Joost J. Bakker IJmuiden, CC-BY-2.0

Tags
Drupal Planet