I’ve been thinking hard about what makes a good Key Performance Indicator. The wikipedia entry for KPI opens with:

Key Performance Indicators (KPI) are financial and non-financial metrics used to quantify objectives to reflect strategic performance of an organization.

That’s all well and good, but it creates a new problem: They don’t resonate with people working and thinking at an operational level. This means that they effectively fail the “So what?” test applied by in-the-trenches operations staff. And if they don’t believe the metric has value, then they won’t execute in such a manner as to improve it.

For a non-security example, consider Gross Domestic Product. This is total value of goods and services produced by an economy, typically a country. Economists along with government and financial leaders use this as The Number for measuring how well or poorly an economy is doing.

There’s only one problem with it: it’s largely useless to the Average Joe who’s getting up and going to work every day to actually produce the Product.

He’s more likely interested in a KPI like “dollars earned per hour/week/year” or paycheck size, or something otherwise downward-focused and inconsistent across the economy.

I know that no one told me this was going to be easy, but that’s cold comfort some days.

Hello, me again.

Have you looked at Andrew Jaquith’s book Security Metrics? I’m still reading my copy, about 3 months after buying it. Parts are useful and interesting, others (such as the laborious description of various types of graph) fail the ’so what’ test, but overall it’s well worthwhile for those of us looking for The Answer.

G.

Hi Gary,

I will freely confess that I picked up the term “So what? Test” from Jaquith’s book. I thought I mentioned in a previous metrics post that I had have been reading it. Unfortunately, I still haven’t finished it yet even though it has been riding around in my bag for a couple of months.

Also, Alex Hutton had some good comments on this post over at Risk Analys.is, though I didn’t get a trackback for whatever reason.

Chandler -

I went through, like, two iterations of trying to send a trackback. Have no clue as to how to get that to work better. I’m afraid that the technology behind it is something I’ve not (and given time constraints, most likely will never) researched, and as such unless there is shiny “trackback” button I’m lost.

RE: Jaquith’s Security Metrics

It’s a favorite of mine, and I tell everyone I know to buy the book. However, if there is one friendly criticism, it’s that it gives little guidance to the context of metrics, that is, what is the value of this information we’re measuring. Now Andrew’s a professed ‘non-modeler’, and that’s ok (I’m not really wishing to re-hash that discussion) but until we can understand what is “Key” to us, and to some extent, what is “Performance” to us, we’ll never have anything more than many discreet indicators from which we’re left to draw our own conclusions.

Alex,

No worries. I think that the trackback has died the death of a thousand spam links, which is why I added a link to your post in my comments.

As to the limitations of Jaquith’s book, I think that the “it’s just a toolkit” complaint that I’ve read on more than one occastion is indicative of a much more fundamental problem, namely the fact that too many people want their answers handed to them.

Just as entirely too many people seem to think that “security” either is or should be a box they can buy and plug in to their network, so they believe that “metrics” should be a list of things they can track and suddenly the clouds open and sunlight shines down.

Unfortunately, (or, if they thought about the implications for their job security, fortunately,) measuring risk and security aren’t that simple. Every business has a different definition of “enough” when it comes to defining adequate protection. Our value comes from our ability to determine how much is enough, then drive that understanding and the accompanying changes through our organizations.

The metrics are simply a numerical manifestation by which we track our value over time.

eople’s failure to understand what’s “core” and what’s “context,”it’s a f

Jacquith’s book is a good appetizer but for a main course I’d suggest Debra Herrmann’s “Complete Guide to Security and Privacy Metrics” though I tend to shy away from books with “Complete” in the title. :-)