Information Security Metrics Can Be Dangerous

Unknown Metrics.jpgInformation security metrics can be very misleading when taken out of context.   Metrics are the distilled insights of data measurement and can provide valuable information to support good decisions.  Unfortunately this is not always the case, as metrics also have a dark past of being used to mislead, misdirect, distort facts, and distract audiences.  Mark Twain popularized the phrase: "There are three kinds of lies: lies, damned lies, and statistics." 

The information security industry is in dire need of good metrics to support decision making, but also represents a very easy domain to argue almost any position with poor, questionable, or unsuitable metrics.   Given the current lack of standards, oversight, sources, and varying assumptions, security data is easily misrepresented or misunderstood.  This can cause problems both intentional as well as unintentionally.   Charlatans can wield numbers to sway opinion to their benefit, while good natured advocates may use available data in ways not applicable or relevant to the situation.  The result is the same.  Bad information results in poor decisions.

The underlying details are important when looking at and incorporating metrics. In many cases, failure occurs when metrics are de-coupled from the original decision they were generated for and used for other purposes without clear consideration of the principal assumptions.  To keep reigns on the use of security metrics, it is important to understand how they are derived and what circumstances they were crafted to measure.  With the purpose of metrics being to support decisions, this information is imperative.  Some would argue that metrics without purpose is simply measurement data.

My recommendation is to question everything!  Challenge the origins of security metrics.  Know the source and what they were intended to show.  Understand the applicability and limitations.  Only then can you truly understand if what is being presented is relevant, accurate, and supportive in making intelligent decisions.

Published on Categories Archive
Matthew Rosenquist

About Matthew Rosenquist

Matthew Rosenquist is a Cybersecurity Strategist for Intel Corp and benefits from 20+ years in the field of security. He specializes in strategy, measuring value, and developing cost effective capabilities and organizations which deliver optimal levels of security. Matthew helped with the formation of the Intel Security Group, an industry leading organization bringing together security across hardware, firmware, software and services. An outspoken advocate of cybersecurity, he strives to advance the industry and his guidance can be heard at conferences, and found in whitepapers, articles, and blogs.