A Cautionary Look at Datacenter Consolidation
We've regularly looked at datacenter and equipment consolidation. Consolidation can reduce resource consumption and environmental toxins by reducing energy use and equipment purchasing, yet consolidation efforts without careful implementation and appropriate safeguards can lead to catastrophic failure points. Two articles in the same week about a major American university illustrate the point.
A June 2014 article in Campus Technology reports, "The University of Wisconsin-Madison is expecting major savings and improved performance from a massive effort to aggregate its 97 data center locations into as few as eight." The article quotes the University's consultants as finding the current system to be "inefficient, resulting in duplication and overspending in areas including: hardware purchases, utility costs for power and cooling, labor and facilities."
Six days later, the Wisconsin State Journal reports that problems with the largest data center's universal power supply caused a major service failures. "The library couldn't loan out books...Websites hosted on wisc.edu, including the university's homepage, went dark. Email addresses that end in wisc.edu -- about 80,000 students and employees have one -- stopped working. Class websites hosted on the Desire2Learn platform, used by campuses throughout the University of Wisconsin system, also took the day off due to the outage...The outage disabled all wireless Internet connections on campus."
The newspaper article goes on to note that while the battery back-up system failed, the datacenter has "...a backup, diesel generator that can supply power for up to 72 hours. However, it can't be activated in this case because it, too, is run through the Universal Power Supply system."
The Campus Technology article is all about building user confidence in datacenter consolidation efforts and offers five points to consider for a successful initiative. Maybe it ought to add a sixth about addressing concerns over creating major failure points.
And the university might want to evaluate manual backup switching. Smart power management can yield significant energy savings, but a human has to be able to intervene in a crisis.
(We've written this post the day of the failure and will update it if significant new information becomes available.)