There is a configuration management system out there called "Puppet."  It's a very cool system that basically allows configuration of servers on a massive scale.

In comes Vagrant.  It basically creates pre-configured virtual machines, or virtualized servers (basically the same thing).  It's setup so that anybody can build a "base" virtual machine that future machines can be built on.  It has scripting abilities so that the machine can be customized after it is initially created.  In other words, there is a way to fairly quickly build a massive number of virtual machines, with each one having their own set of requirements.

So we've got Puppet, which can configure systems enmasse.  We've got Vagrant, that can build virtual machines enmasse.  With these tools, I can create a ton of virtual servers, and ensure that all of them are configured a specific way (and continue to be configured that way).

The servers eventually (if not from the word "go") get their own special configuration file.  This file ensures that the servers have certain programs installed, users present (or purposely missing, in the event that a sysadmin leaves).

So... let's imagine an environment with a thousand servers, all with their own Puppet configuration.  If one server goes down, a replacement can be built in just a few minutes, and fully configured just a few minutes later, fully prepared to take over the server that went down.

In fact, a replica of any of the existing systems can be built on demand in just a few minutes.  A virtual replica of a production server can be built for a developer to run on their machine for testing.

It's freaking cool.  Words cannot express my excitement.