Musings on IT, data management, whitewater rafting, and backpacking

Friday, August 12, 2011

New server room first impressions

We're moving into our new server room (data center is a dirty word now). I've mentioned many of the design elements in previous posts.

Here are my first impressions and a few lessons learned.

Major design elements:
  • 480 square feet
  • 60 KW IT power feed
  • Concrete slab floor with 6 inch high concrete pedestals under the server cabinets
  • Six sealed server cabinets with blanking panels and chimneys for heat exhaust
  • Room for six more server cabinets
  • White cabinets, white walls, and soon, white flooring
  • Sound absorbing foam on two walls
  • Two HVAC units on the roof; each can handle 80% of maximum load
  • Aggressive air-side economization
  • Supply temperature floats between 70 F and 86 F
  • No humidity controls
  • Overhead telecom ladder racks
  • Overhead power busses
  • 208 VAC single phase power for most IT loads
  • Rack-mount UPSes with generator backup for IT loads
  • Generator backup for HVAC
We have about 5 KW of active IT load now (as measured on the buss), with more servers moving in over the next few weeks.

So far, I'm happy with this design.

Some compromises are causing minor issues:

– Our off-the-shelf HVAC units do not have variable speed blowers (VFDs). Each unit blows 5,000 CFM, and is on or off. 5 KW IT load needs about 500-600 CFM, so we are overpressurized. The room doors tend to blow outward, but not dangerously so. We can reduce the overpressurization simply by removing some blanking panels, but we need to be careful not to mix cool and warm air. Custom HVAC units with VFDs were several times more expensive.

– Our contractor surprised us by painting the existing stained and scarred concrete floor (the old floor was fine with me). Now, we're vacuuming up small snowdrifts of paint chips daily, to keep them out of the servers. I'm getting an estimate for floating vinyl flooring next week. Next time I would spec vinyl flooring.

– We have a 75 KW transformer in the server room, in spite of my spec for "no high EMF sources in the server room". We moved the transformer to a corner just inside the door to minimize the EMF impact on the rest of the room. Transformer magnetic fields fall off as the cube of the distance, so this shouldn't be a big problem. The transformer generates some heat, too. Our main problem is keeping people from parking sensitive equipment or media on the handy flat topped surface.

– The cabinet chimneys are fitted into an overhead exhaust duct, instead of a dropped-ceiling plenum. This design, chosen by the HVAC subcontractor, took a lot of custom sheet metal work and minor modifications to the chimney mounts. Adding more cabinets in the future will require more custom work.

Other issues that came up:

Clear communication with the HVAC subcontractor was challenging. My design goals were so different from what they had installed elsewhere, that they kept making incorrect assumptions. However, the back-and-forth was useful. The lower temperature bound of 70 F is much higher than my original spec of 60 F, but recirculating IT-generated heat is free. 70 F gives us plenty of margin above the highest dew point we've recorded in the past year of 62 F. We also lowered my upper bound spec from 90 F to 86 F. The cooling lag when the compressors kick in mean we could peak at about 90 F.

– The HVAC subcontractor ran supply ductwork into the "cold" aisle, even though my design doesn't require that. Could have just dumped the cool air through a large screen in the wall. We don't need precise distribution of cool air since we have almost no warm and cool air mixing.

Never again will I say something like "how hard can it be to install cabinets?"  Installing 6 cabinets took 2-3 of us 3 full days, not including connecting chimneys to the overhead exhaust duct. And then we spent several days off-and-on fixing problems with the floor bolts and brackets. Next time, I'll pay for professional installation.


  1. Updates:

    – The HVAC subcontractor installed a manually-adjustable air bypass, which almost eliminates the pressure on the doors. We need to adjust only when we add or remove significant IT equipment.

    – We had inexpensive, light-colored vinyl flooring installed over the peeling painted floor. Much better looking, improves light levels in the room, and no more paint chip snowdrifts.

    - Bought a Gauss meter from a reputable source, and measured the levels around the transformer. Except for one spot right where power enters and exits, the magnetic fields were well below anything that could affect magnetic media. Our electrical subcontractor says modern transformers are much better about this.

    - No problems so far with room temperatures, and we've gone through the hottest and possibly coldest parts of a typical year since move-in.

  2. Nice blog.....The structural support offered by server cabinets is crucial to keeping expensive hardware in place. And also to making that hardware easy to fix when problems arise, or to swap out when it becomes superannuated.
    server rack and cabinets