I told Pete Lindstrom last time I talked to him that I was going to be doing metrics in 2008. That was back before Christmas and since then I’ve been reading and thinking a lot about the topic of metrics in general and security metrics in particular. Metrics has been a side interest of mine for a few years, but up until recently, I was not in a position to build a metrics program.

As I develop and deploy my Security Metrics program, I now find myself much more attuned to the metrics in the world around me, especially how metrics are composed and then presented. The polite word for it is “spin.” Or, as my long-ago post-college room mate, a professional statistician, used to tell people when asked what he did for a living, “I lie with numbers.”

For example, consider this example Fuel Economy Statistics from Mark Kleiman:

Two facts I learned at the annual meeting of the Canadian Centre on Substance Abuse:

* The average Canadian walks 900 miiles per year.
* The average Canadian drinks 22 gallons of beer per year.

Canadians have a right to be proud: they’re getting 41 miles to the gallon.

I realize that it is entirely possible to be both absolutely accurate and utterly wrong. That is what I hope to avoid as I develop my program. As such, I’m being very careful to ensure that I both provide a baseline for the overall level of IT Security noise as well as to determine the effectiveness of our risk reduction efforts.

Some of what I’m looking at is the easy, obvious stuff like spam and virus detection & correction rates, volume of trouble tickets related to malware, patch latency and effectiveness, etc. They’re not tremendously valuable, other than as a barometer against which other trends can be compared (e.g. are our malware ticket rates correlated to patching? Are outbound email virus rates correlated to any activity? What’s the potential impact of a theoretical worm targeting an exploit?)

Others are more derived, such as the effectiveness of our network perimeter, which, if done correctly, should allow me to answer questions like How much and why type of effort should we put into perimeter defenses? As a formerly-active participant in The Jericho Forum, I have my hypotheses and biases, but will rely on transparency to both keep me honest and support my case in that regard.

For the time being, I’m developing a table which identifies potential metrics I might look at. Some of them are taken from :

• Category: The high-level type of metric, e.g. Coverage, Control Effectiveness, or Risk Management
• Sub-Category: The type of item being measured, e.g. Perimeter Effectiveness, Desktop or Network Hygeine
• Name: The metric itself, e.g. ACL’s on the firewall, Number of external business partners, Number of reported security incidents
• Unit of Measure: The type and scale of the metric. Typically, these tend to be a number (e.g. # of hosts, #of ACL’s, # of tickets), a percentage (e.g. % of windows hosts without anti-virus by region), or cost (cost in hours or dollars to handle a password reset or manually clean an infected system)
• Accuracy: How good is this data, either as a confidence interval (e.g. +-5% or 50%?) or just qualititative (H/M/L). This is used both the prioritize efforts (no point in killing my self or burning a bunch of political capital to get crummy data) and to ensure that we don’t place too much reliance on sketchy information.
• Scorecard?: Whether I would expect to use this metric on a management scorecard. This is, to a certain extent, my field for what Andrew Jacquinth calls the “So what?” test.
• Raw, Calculated or Derived: How much analysis is done to gather this data. This is one of:
  • Raw:Whether this is simply counts of things that do or don’t match a criteria (e.g. are Internet-facing, are fully patched, etc.) at the point in time the metric is collected
  • Calculated:A simple transformation of raw data into a more comparative format. For example, calculating percentage of hosts missing patches based on the total number of hosts or in-scope hosts compared to the number reported as patched by the patch manager or security scanner.
  • Derived: A metric which requires extrapolation or assumption. For example, if I were to attempt to estimate the number of unpatched hosts across the Enterprise based on the data from my patch management system and some random spot checking of out-of-scope systems, I might come up with a ceiling and estimate (combined with appropriate adjustment of my accuracy field) of the patch levels across the enterprise.
• Value: How valuable is this metric to me in determining our risk posture as a company? Ironically, there seems to be an inverse correlation between ease of gathering and actual value. This is because the metrics which are easily automated are those that are provided by the vendors and thus are most likely to be 1) generic; and 2) useful for demonstrating how well their tool works. Also, since security tools are a means to an end, they can , at best, supply inputs to assess progress toward the end.
• Implementation Difficulty: How hard will it be to gather this metric, both initially and as a reapeatable (ideally, automated) process?
• Raw Data Source: What application, tool, or system can produce this metric?
• Raw Data Supplier Organization: What group in the organization manages the data source and can provide the information?
• Metric Format: How will the metric be delivered? In what format? Will it be emailed, pulled from a Web server, generated as a report from a management console? Pulled from a SQL database? etc, etc, etc.
• Aggregation Point: Where will this information be aggregated and reported until I get a system in place to centrally manage and report? (That’s planned to kick off in the second half of the year)

I’ve got well over a hundred potential items I’d like to measure and am still addiing new items with some frequency. One thing that has astounded me here is how little measurement goes on, period. Everyone I’ve talked to has agreed that metrics would be great, but no one is doing anything about it. The real fun comes when people realize I’m serious about this.

Once the program is fully-formed, I’ll post a proper paper on the topic. For now, though, I expect this to be a running series over the next year or so chronicling my efforts, so add me back to the ol’ feed reader.

Finally, part of my reasoning in writing publicly about this while it’s in-progress is to solicit feedback and comments from the excellent security and risk management community members out there, so don’t hesitate to comment, both good or bad.

Chandler,

it will be interesting to see what kind of metrics you will define. Perhaps http://www.kpilibrary.com can give some inspiration and input, although i think the IT security categories could benefit from your experience and participation:

http://kpilibrary.com/?cat=145
http://kpilibrary.com/?cat=95

best regards.

Erik,

Thanks for that pointer. The library seems to be fairly preliminary for now (not a criticism–everything has to start somewhere), but it’s certainly an interesting starting point and I’ll be sure to add my own KPI’s as we select them.

Chandler,

You’ve certainly chosen a challenging area. Yes - gathering measurements is not really the problem. The hard part is using meaningful metrics meaningfully, so they build a credible picture on the performance and effectiveness of the thing you’re measuring. This in turn requires a measurement model which delivers sound, objective, consistent results. I’ll look forward following your reports on progress, not least as valuable contributions to our Risk Management project (FAIR - http://www.opengroup.org/projects/security/doc.tpl?CALLER=index.tpl&gdid=13231).

Regards,
Ian Dobson.

Ian,

Don’t worry…I’m definitely thinking hard about how best to measure both traditional perimeter effectiveness as well as thinking about how to evaluate the trade-off’s that deperimeterization brings. The challenge is getting over the “so what?” hump from operational to true KPI’s.

Also, I knew from Alex Hutton and the rest of the team at Risk Management Insight that FAIR was under the Open Group’s aegis.