Musings on IT, data management, whitewater rafting, and backpacking

Friday, August 20, 2010

Data Center Consolidation?

The Powers That Be have decided we have too many data centers, and we must consolidate. No doubt they will set a goal like 50% reduction, pulled out of thin air or from an airline's in-flight magazine.

Let's consider some of the hard problems with data center consolidation:

  • Nobody's repealed the speed of light. If we're forced to use a data center 1000 miles away (very likely), lots of applications will break due to latency. Add more latency from more hops through routers, firewalls, IDPs, and other security-boxes-of-the-week. I don't see any solution to this problem for certain mission-critical applications, even with magic WAN acceleration boxes.
  • Nobody's considered WAN upgrades and costs. If most of my servers move 1000 miles away, I'll need a massive WAN link upgrade to my local corporate node, and we'll need a massive corporate WAN backbone upgrade. We have constant battles over sharing the costs of our currently overloaded corporate WAN, I can just imagine the battles over the costs of a massive upgrade. And some locations can't get massive link upgrades. Most of our locations have one, maybe two WAN providers, usually just the local telco. Maybe they can't install a 1 Gbps WAN link to Timbuktu.
  • Nobody's considered the security implications. If you put all your eggs in a few baskets, you better guard those baskets really well. We usually don't, for many reasons. Yes, most of our small data centers have poor physical security. But most of them don't hold sensitive information, and we haven't had many problems with physical security.
  • We don't know how to run consolidated data centers. We have extremely distributed management, extremely distributed IT support, and extremely diverse IT requirements ("one of everything, please"). How will we manage and support all of that in a few locations, thousands of miles from the users and application experts? Cloud computing models are great – if you have relatively uniform IT requirements.
  • We don't know how to fund data center consolidation. We have extremely distributed IT funding. Consolidated data center funding will be, well, consolidated. Historically, unless we got new funding for a national IT project, we could not find a way to "pass the hat", or tax, or otherwise extract funds from possibly thousands of internal accounts.
  • We don't have funds for consolidation. We might save money in the long run, but nobody will front the transition costs. We expect 10% budget cuts for IT next year. 
  • Nobody will do an apples-to-apples cost comparison. We never have, and I see no reason this time will be different. So we'll never know if consolidation saved money or not.
  • We have a bad definition of a "data center" - "three or more servers."  I've had three physical servers running in my office sometimes!  Do virtual servers count?  Do desktops in a lab running automatically-enabled remote control, backup, file, print, and web servers count?  How about telecom closets where each switch is an SNMP, SSH, and HTTPS server?
Many of these problems can be solved in many locations. Some of these problems are layers 1-7 (technical), others are layer 8 (financial), or layer 9 (political) (get the t-shirt!), and those are usually the hardest to solve.

I'm not opposed to data center consolidation, just the dumb way we're approaching it. Setting arbitrary goals to reduce the number of data centers without knowing how we'll solve all of these problems is stupid.

My suggestions:
  • We should go after the low hanging fruit. Our nearest regional headquarters has nine "data centers" (with three or more real servers) that I know about, and that regional hq is the smallest of several.
  • We should make consolidated data centers attractive.  No matter how hard you try, a distributed organization will find a way around consolidation orders.
My ideal setup:
  • A fast WAN link to easy-to-use, secure, reliable, inexpensive, internal or outsourced cloud computing and storage. I'll gladly move as many applications as I can to that cloud.
The first data calls have gone out. Let the games begin!


No comments:

Post a Comment