I’ve been pretty busy lately, which has impacted, among other things, my time and energy for blogging. I don’t know when I expect this to materially improve, so I’m going to fire these thoughts out before I run off to my next meeting even though they may still be a bit partially-formed.

Lately, I’m feeling that there are two fundamental problems with risk and security metrics.

The first, which I’ve written about previously, is that they don’t scale the corporate ladder well. The second, and perhaps more serious from an industry perspective, is that the metrics which do scale the corporate ladder well don’t compare well across industries or even within industries. Thus, there seems to be a paradox here: the more business value a security metric represents, the less either generic or share-able it will be.

For example, take metrics related to policy compliance, one of my KPI’s. I assume that policy (or lack thereof) is an expression of or proxy for a company’s tolerated level of risk. Given that no two companies have the same policies (unless they both cribbed them whole-cloth from SANS), the risk measurement is going to be inherently different between companies. Throw in the fact that most companies won’t be willing to share this data in anything but a tightly-controlled forum, and you’ve got a real problem.

Nevertheless, I’d still be pretty happy if we could get general agreement (or even understanding) across so-called risk managers that, like it or not, policy effectively defines organizational risk acceptance. With that starting point, we might then actually be able to begin doing meaningful comparisons of different policy/control sets (e.g. does CObIT+SoGP produce better compliance (as measured through audit findings & exceptions) than CObIT+ISO-27001? *That* would be an interesting and worthy research project, IMHO), although the vested interest factor could definitely hurt the effort.

And one final piece of the puzzle (and this is probably too much to even dream of, but keystrokes are cheap) would be to then correlate these measures of relative compliance to operational metrics. While correlation is not causation, we still might then be able to begin using compliance as an attribute to describe our accepted level of risk rather than as an end unto itself.

“I assume that policy (or lack thereof) is an expression of or proxy for a company’s tolerated level of risk.”

I’m going to call bad assumption. Most of the issues I’ve seen with security policies and non-compliance (to policy) is that the policies are that they are written as a representation of a “best practice” by a CISSP-type regardless of the risk tolerance or appetite of management with little consideration for user experience and productivity impacts. So when policy is challenged, or KPIs are given based on assumed risk tolerance/appetite/interest - credibility and usefulness of information risk management suffers.

I advocate the following:

1.) Some sort of risk tolerance (& appetite?) is developed by Mgmt.

2.) That is expressed as a ‘pain scale’ (http://www.anes.ucla.edu/pain/FacesScale.jpg) target for KPIs to hit, preferably as a number that is in probable frequency/amount of loss and/or ability to manage risk.

3.) Policy conditions are established:

if current state of risk or risk management capability is

Alex,

I agree that in many places I can think of (now that you put it as you have), defining IT Risk tolerance is an ungoverned, bottoms-up activity.

I guess I have a certain blindness in my day-to-day, since part of my job is to prevent that model from taking shape. Thanks for pointing out that I don’t live in most people’s risk management reality ;-) .

Chandler,

Somehow the greater than / less than symbols cut off my comment. Whooops.. Or maybe you just thought I was being to wordy :)

So your job is to make sure that management knows risk or is oblivious towards it? Didn’t know which model you were preventing.

(the fact that I have to ask that question should cause us all to really get kind of depressed about our current state, leave the office early on this beautiful spring day and go do something we really like, preferably involving adult beverages)

Alex,

No editing on my part, so I’ll blame wordpress.

I often feel that my job is to worry about risk so the rest of management doesn’t have to, which I think would fall into the “enabling oblivousness” option.

That’s not necessarily a bad thing–more of a division of labor option. They worry about things like how to make better & sell more products, and I worry about keeping things safe.

More on this challenge in my next post…

Oh, and while it’s pretty crummy weather here in Chicago today, I completely agree with your sentiment in principle, even if today I can’t do it out on Lake Michigan or in a nice, sunny beer garden.