Even as 365 Main takes the unprecedented step of publicly updating everyone on their root cause investigation of the generator failures that darkened many well-known web sites, will the lessons learned become lessons remembered? Take this short quiz for your own organization:
1. Are your DNS servers geographically dispersed? If all of your DNS servers go down with your other servers, then you will not be able to react and re-point critical functions to operational servers. Dispersing your DNS is simple to do and yet many organizations have not done even that.
2. Do you put too much faith in your utility’s past history? Like the stock market, past performance is no guarantee of future results. Many things outside their control can and do interrupt power.
3. Do you believe that generator and UPS testing are actually done under real-world conditions? The real story is that generators are manually started and tested and UPS systems are put into bypass mode. When actual events cause these systems to engage, unexpected things do happen.
When we move data centers, we often encounter single points of service delivery that must be corrected to move the data center while not affecting the customer’s service. You don’t need to move a data center to understand the value of geographic diversity. You also don’t need to brute force the problem by duplicating everything you have somewhere else.
The real question is will you act before an event costs you money?