It Looked Good On Paper

collocation-cage-space-300x227.jpgWhen it comes to technology and automation for data centers IT is not a hard sell.  But sometimes it pays to step back and take a second look.  For example:  We worked with an enterprise who was well underway in their build out of a new remote data center.  This was their first remote "lights out" data center and like many they worried about not being on-site to fix issues or know what was going on in their space.  Engineers were used to walking into the data center and performing their own cabling work.  As a result they purchased every monitoring sensor known to man and ordered cameras to ensure no one could hide in a dark corner.

On paper they would monitor the top and bottoms of every rack for temp, humidity and just about anything in the cabinet you could imagine.  The NOC would be altered when a cabinet door was opened, this is after a card key is swiped to allow (or not) a person to open a cabinet door as well as to get into the private data center and the cage area.  Lots of USB cameras point up and down every row so the NOC could monitor anyone moving in their data center and ensure no funny business is going on.

In practice... only a select few operational staff members and trusted vendors are allowed into the data center.  Customer engineers are always accompanied by operational staff.  The temperature does not very much from top to bottom of the cabinet (they have good air flow) and the temperature across all cabinet is consistent.  The cameras work great for seeing general shapes of people but you would be hard pressed to identify anyone and the cameras can't see into the specific cabinet an individual is working on or when the door is open and in the way of it's view.  The software required to configure all the monitors is cumbersome and it's monitoring components are ignored since all the SMTP traps are captured in their NOC's monitoring tool, SolarWinds.  Cabinet key cards require a separate system for managing card access. The additional $5,000 in monitoring equipment and installation fees for each new cabinet is becoming a difficult sell to management who now question its value.

A step back.  Had they looked at their old data center's work flow, considered the use of "Remote Hands", examined their monitoring practices and evaluated real risks verses irrational fears they may have come up with something more useful and maintainable.  Some of the key suggested changes we made included:

  • Remote Hands assigned a primary individual who is responsible for the customers space and attending a weekly customer operations meeting
  • Operations created a check list of what to look for (indicator lights, PDU power readings, open doors, tools left on cabinets, unplugged (and untagged) cables...).  This check list is used by Remote Hands who perform a weekly walk around and report.
  • Dozens of low-res cameras were swapped out for a couple high-res cameras aimed at the private ingress and egress points.
  • Temp and humidity sensors were shifted from every cabinet to every third cabinet.  Unused sensors and collectors are now stock for future cabinets. Electronic card cabinet locks were replaces with the original key locks.  The keys are maintained by data center security office and handed out at sign-in.  Master keys are available only to specific operational staff.
  • Monthly audit reports are produced by security for pre-authorized entry and emergency authorization showing who, time in, time out and other security items.
  • Quarterly audit visits are made by operations for improvements, capacity planning, data center relations.
  • Templates were created for engineers to complete on any new equipment requiring rack, stack, cabling services.  "As Built" is documented on every major device and rack with quarterly audits to ensure accuracy is maintained.

We love technology and automation when it comes to monitoring items in the data center but it is extremely difficult for an organization to effectively make leaps up the maturity model solely based on automation.  Sometimes the most cost affective and efficient approach is to step back and make sure your workflow and real needs match up with what you are automating in order to receive the ROI you are hoping to see from your data center.