As I continue to ponder metrics, I have also been looking beyond the realm of Information Security & Risk Management for inspiration. That led me to a great example of effective metrics for measuring the success of foreign aid programs, a notoriously-difficult problem given the high rates of corruption and fraud in parts of the world most in need of development aid:

The aid industry faces a dilemma. On the one hand, countries are more likely to grow rich if their citizens are provided with some important basics, such as a legal system that works, or protection from corrupt officials. Such basics might seem the priority for aid money. On the other hand, it is much easier to measure success in simpler projects, such as building roads and laying pipes.

If development agencies focus on pouring concrete, they may be spending money on infrastructure that will never be used – and perhaps never even be built – because of corruption in the background. But if they focus on the broader stuff – democracy, corruption, human rights – they risk trying to do everything and achieving nothing. William Easterly, the World Bank’s most prominent apostate, argues that development agencies love big agendas because their contribution cannot be measured and found wanting. It is a bitterly cynical view, but that doesn’t mean he’s wrong.

This is an interesting problem statement, and one that resonated with me as I continue to struggle with the question of what my KPI’s should be (more on that tomorrow, maybe). I find that, all-too-often, my world of metrics breaks into one of two categories: “Things I can measure” and “Things I Need to Measure.”

So maybe it’s time for a little lateral thinking. Rather than focusing on measuring the end result, measure the effectiveness of the process. Be sure to consider, however, the motivations and reactions to the measurement. For example, how a project is measured can shift the fraud:

Another young economist, Ben Olken of Harvard, used a similar randomisation technique to work out whether corruption in Indonesian road-building projects was best fought top-down, using audits, or bottom-up, soliciting comments from local villagers about whether money was being embezzled. One challenge was to work out how much embezzlement was taking place. Olken enlisted engineers to take samples of the road’s structure and to estimate how much it should have cost to build; he compared that estimate with how much spending was claimed in the project’s accounts. The missing funds were a rough guide to the amount embezzled.

In contrast to Duflo’s results, Olken found that the bottom-up monitoring was not effective – it shifted the embezzlement from something the villagers cared about (wages) to something they did not (building materials). The threat of a guaranteed audit – a threat that was later carried out – was much more effective, reducing the estimates of missing funds by a third.

So what does this mean for me as the developer of Security Metrics? On one level, nothing. But in reality, it’s a reminder that metrics are instrumentation for the procesess and systems that comprise (in my case) an Enterprise. That information is provided in an environment where people will use it both for good and less-than-good purposes. How people interact with and utilize metrics will have significant impact on how they use or produce them.

First, it’s a reminder that people will, at a minimum, try to leverage metrics to their benefit. This is where transparency and objectivity of the data comes in. A well-designed metric will be empirical enough that the results are, if not indisputable, at least relatively spin-resistant.

Second, if results are being measured as progress toward a goal, ensure that principles of designing a secure system are properly incorporated into the process. Especially if money is involved, expect people to try to game the system.

For example, I remember from Freakonomics the case of what happened when the Chicago Public School system tied student performance on standardized tests to teacher raises and bonuses. The result was tremendous improvement, but only because of widespread, independent instances of fraud on the part of the teachers. Sure, pay-for-performance increased test scores, but not because the students were actually learning more.

Core security principles such as Segregation of Duties (teachers should not have had access to their own students’ test forms), the Two-Man Rule (minimize collusion risk), and some basic integrity checks or chain of custody (ensure the forms couldn’t be altered after the student completed the test in any case) would almost certainly have prevented both the embarrassment that CPS ultimately suffered as well as avoided rewarding the most-corrupt rather than most-skilled teachers on their payroll.

While I don’t anticipate having to deal with flat-out fraud as I design my metrics, I still want them to be resistant to manipulation or “moving the goal posts,” yet still provide the information that people want and need to effectively secure the business.

This is from last month, but I’m slow some days. Anton Chuvakin is shocked! Shocked, I say! that an as-yet unpublished survey says most people view IT Security as separate from IT Risk:

How can security be THAT disconnected from risk? Can somebody explain this to me? (Please don’t explain by stating “crappy survey methodology” - I can pull this one myself, thank you very much :-))

What I find most interesting about this is that he found it newsworthy, especially given his overall strong level of understanding of what people really do with regards to security (e.g. his accurate 2007 predictions, which I agree with myself). Rant-worthy? Maybe if I need to vent, but other than that, not so much.

Of course, this is a topic that I’ve written any number of posts about in the past, so I’ll let this be a bit of a retrospective, then I can get back to digging for the quote from Anton’s blog that I need for the presentation I’m putting together.

First, I’ll provide my definitions of IT Security versus Risk Management:

Information Security is the practice of designing and implementing countermeasures and other preventative (usually technical) controls on information. Security experts tend to understand the nuances of their tools, but all-too-often fall prey to the adage that, “When your only tool is a hammer, ever problem begins to look a lot like a nail.”

Information Risk Management (IRM) is the practice of determining which Information Assets need protection and what level of protection is required, then determining appropriate methods of achieving that level of protection by understanding the applicable vulnerabilities, threats and countermeasures.

Moving this dichotomy into the real world, let’s re-visit what happens when we try to explain a potential risk to people. First, I think of Niels Bohr, who famously pointed out (albeit in a different context), that “Prediction is hard, especially about the future.”

Picking freqency or likelihood of occurance doesn’t do us much good since, unfortunately, there are only two levels of likelihood in most people’s mind when dealing with risk:

Definitely will happen (probability 1) — i.e. worms or viruses. In the eyes of The Business, things only move into this category after technology solutions exist, which I should be busy providing instead of wasting their time with meetings and trying to turn their project’s status to Yellow.

Never will happen (probability 0) — i.e. Systems intrusions by skilled attackers resulting a massive loss-of-data or a hurricane destroying New Orleans. (Note: this eventually gets turned into, “Definitely will happen,” but only after it’s too late to do anything about it. For further reading, see Code Red, SQL Slammer, or anything in Adam Shostack’s privacy breach category.)

…

As a result, I spend a lot more time trying to explain to people why we’re all sitting on the phone or in a room together, especially given that their default viewpoints are
1) The incident in question Never Will Happen
2) I’m probably about to turn their project status Yellow and thus must be opposed with all their mortal vigor

This is a problem I’ve been fighting this fight here in my new role as I try to help people understand risk and the idea of managing to a desired level of risk. Risk and Loss Tolerance are, I’m realizing, much more difficult concepts that I’d apparently given them credit. The idea that “more secure” is not inherently better is apparently too counter-intuitive for some people to understand, and the idea that I would even imply such a thing means that I’m not a Real Security Guy. I’ve pretty much worn out Schneier’s bullet-proof vest example from Beyond Fear, to questionable-at-best effect.

So unfortunately, Anton, I’d argue that, if anything, the survey was probably spot-on, other than that the first response (ISM is part of a risk management strategy) is probably inflated.

I told Pete Lindstrom last time I talked to him that I was going to be doing metrics in 2008. That was back before Christmas and since then I’ve been reading and thinking a lot about the topic of metrics in general and security metrics in particular. Metrics has been a side interest of mine for a few years, but up until recently, I was not in a position to build a metrics program.

As I develop and deploy my Security Metrics program, I now find myself much more attuned to the metrics in the world around me, especially how metrics are composed and then presented. The polite word for it is “spin.” Or, as my long-ago post-college room mate, a professional statistician, used to tell people when asked what he did for a living, “I lie with numbers.”

For example, consider this example Fuel Economy Statistics from Mark Kleiman:

Two facts I learned at the annual meeting of the Canadian Centre on Substance Abuse:

* The average Canadian walks 900 miiles per year.
* The average Canadian drinks 22 gallons of beer per year.

Canadians have a right to be proud: they’re getting 41 miles to the gallon.

I realize that it is entirely possible to be both absolutely accurate and utterly wrong. That is what I hope to avoid as I develop my program. As such, I’m being very careful to ensure that I both provide a baseline for the overall level of IT Security noise as well as to determine the effectiveness of our risk reduction efforts.

Some of what I’m looking at is the easy, obvious stuff like spam and virus detection & correction rates, volume of trouble tickets related to malware, patch latency and effectiveness, etc. They’re not tremendously valuable, other than as a barometer against which other trends can be compared (e.g. are our malware ticket rates correlated to patching? Are outbound email virus rates correlated to any activity? What’s the potential impact of a theoretical worm targeting an exploit?)

Others are more derived, such as the effectiveness of our network perimeter, which, if done correctly, should allow me to answer questions like How much and why type of effort should we put into perimeter defenses? As a formerly-active participant in The Jericho Forum, I have my hypotheses and biases, but will rely on transparency to both keep me honest and support my case in that regard.

For the time being, I’m developing a table which identifies potential metrics I might look at. Some of them are taken from :

• Category: The high-level type of metric, e.g. Coverage, Control Effectiveness, or Risk Management
• Sub-Category: The type of item being measured, e.g. Perimeter Effectiveness, Desktop or Network Hygeine
• Name: The metric itself, e.g. ACL’s on the firewall, Number of external business partners, Number of reported security incidents
• Unit of Measure: The type and scale of the metric. Typically, these tend to be a number (e.g. # of hosts, #of ACL’s, # of tickets), a percentage (e.g. % of windows hosts without anti-virus by region), or cost (cost in hours or dollars to handle a password reset or manually clean an infected system)
• Accuracy: How good is this data, either as a confidence interval (e.g. +-5% or 50%?) or just qualititative (H/M/L). This is used both the prioritize efforts (no point in killing my self or burning a bunch of political capital to get crummy data) and to ensure that we don’t place too much reliance on sketchy information.
• Scorecard?: Whether I would expect to use this metric on a management scorecard. This is, to a certain extent, my field for what Andrew Jacquinth calls the “So what?” test.
• Raw, Calculated or Derived: How much analysis is done to gather this data. This is one of:
  • Raw:Whether this is simply counts of things that do or don’t match a criteria (e.g. are Internet-facing, are fully patched, etc.) at the point in time the metric is collected
  • Calculated:A simple transformation of raw data into a more comparative format. For example, calculating percentage of hosts missing patches based on the total number of hosts or in-scope hosts compared to the number reported as patched by the patch manager or security scanner.
  • Derived: A metric which requires extrapolation or assumption. For example, if I were to attempt to estimate the number of unpatched hosts across the Enterprise based on the data from my patch management system and some random spot checking of out-of-scope systems, I might come up with a ceiling and estimate (combined with appropriate adjustment of my accuracy field) of the patch levels across the enterprise.
• Value: How valuable is this metric to me in determining our risk posture as a company? Ironically, there seems to be an inverse correlation between ease of gathering and actual value. This is because the metrics which are easily automated are those that are provided by the vendors and thus are most likely to be 1) generic; and 2) useful for demonstrating how well their tool works. Also, since security tools are a means to an end, they can , at best, supply inputs to assess progress toward the end.
• Implementation Difficulty: How hard will it be to gather this metric, both initially and as a reapeatable (ideally, automated) process?
• Raw Data Source: What application, tool, or system can produce this metric?
• Raw Data Supplier Organization: What group in the organization manages the data source and can provide the information?
• Metric Format: How will the metric be delivered? In what format? Will it be emailed, pulled from a Web server, generated as a report from a management console? Pulled from a SQL database? etc, etc, etc.
• Aggregation Point: Where will this information be aggregated and reported until I get a system in place to centrally manage and report? (That’s planned to kick off in the second half of the year)

I’ve got well over a hundred potential items I’d like to measure and am still addiing new items with some frequency. One thing that has astounded me here is how little measurement goes on, period. Everyone I’ve talked to has agreed that metrics would be great, but no one is doing anything about it. The real fun comes when people realize I’m serious about this.

Once the program is fully-formed, I’ll post a proper paper on the topic. For now, though, I expect this to be a running series over the next year or so chronicling my efforts, so add me back to the ol’ feed reader.

Finally, part of my reasoning in writing publicly about this while it’s in-progress is to solicit feedback and comments from the excellent security and risk management community members out there, so don’t hesitate to comment, both good or bad.

I liked this A comment on the Freakonomics Blog examining economics compared to visiting a physics demo:

There’s the old tried and true demo of taking kids and offering them their favorite candy and asking them how much they’d pay for a piece, and then after they have had it, make the offer again and again and again until they puke or realize the next piece has no value. You can enjoy that kind of demonstration only so much.

Economics has vacuums, chilling effects, over pressurizations, laws and theories just like Physics, but unfortunately it does not lend itself as readily to demonstrations of these principles. Seems people have to suffer and make hard decisions to truly see Economics in action, and who wants to demonstrate that?

Reading this comment set me to thinking about what would be an effective demonstration to help people understand why risk management matters. I know that most people, when you get right down to it, can’t seem to differentiate between different levels of probability, especially when looking at relatively abstract concepts or discussing different risk management strategies. How have you successfully (or unsuccessfully) demonstrated effective risk management? Did it involve teaching someone empirically by doing something to them that they idn’t want to happen?