Article on how Google automatically routes around outages

[danielberm]

Here's a link to an interesting article that talks about how Google coordinates their infrastructure to route around errors:
http://www.datacenterknowledge.com/arch ... d-outages/

"There are several different layers of “routing around” problems - a failing Google File System (GFS) chunkserver can be routed around by the GFS client automatically, whereas a datacenter power loss may require some manual intervention. In general, we try to develop scalable solutions and build in the “route around” behavior into our software for problems with a clear solution. When the interactions are more complex and require sequenced steps or repeated feedback loops, we often prefer to put a human hand on the wheel."

The article goes on to state that automated testing software runs constantly to test for error situations and handle appropriately.

It sounds like they're using several layers of communication to check their networks for consistency. Then, if an error is discovered, they (in many cases) automatically reconfigure the network around the problem area.

A real-world situation were advance routing protocols (possibly open or proprietary) play a key role in the operations of a huge corporation.