Thinking about measuring the effectiveness of security training and awareness, I ran across a post by Tim Hartord, the Undercover Economist, provides an excellent reminder of why metrics can fail to properly assess the actual outcomes environmental or infrastructural improvement projects:

Social capital is important but elusive. It is an attempt by social scientists to expand the traditional list of economic assets beyond physical capital, such as computers or roads; and human capital, that is you and me and whatever tricks we’ve learned along the way. Economists have attempted to account for why, in the words of P.J. O’Rourke, “some parts of the Earth prosper and others suck”. But as long as those attempts have simply measured the roads and counted the number of IT graduates, they have failed. Social capital is supposed to explain why, and social scientists, including economists, embraced the concept with enthusiasm.

(emphasis mine)

This is the sort of thing that makes security awareness so hard. What we can measure is how many people attended the training and maybe even passed a test filled with example-sized problems. Then we ignore the fact that, pretty much regardless of subject area, they will forget 95% of what they learned in the next two weeks and revert to type.

Security awareness is a driver of getting buy-in for and adoption of security-related changes to processes and technology, but is not an end unto itself, regardless of what many seem to think (unfortunately, the whole “people, process, technology” mantra drives the wrong conclusions for some people).

While it’s important to do awareness training, the changes it’s driving should be understood and incorporated through changes to processes.

‘The number of people who attend the training’ is about the lamest of lame awareness metrics - but you’re right, organizations do measure it for some obscure reason, mostly extreme creativity bypass I guess. Tests provide metrics with a bit more meaning and purpose but you’ve hinted at a different kind of metric: there are lots of ways of measuring ‘buy-in and adoption of security-related changes’ if you’re prepared to embrace the social scientists, psychologists and others who specialize in measuring people and what they do. They use surveys and observational techniques. They describe scenarios and invent cunning experimental situations to home-in on individual facets of behavior. And they struggle to measure even the simplest of animal behaviors.

I agree wholeheartedly with you that security awareness is a means to an end not an end in itself. However, end-focused security metrics are unfortunately a poor guide to the effectiveness of an awareness program: there are far too many variables to assign a reduction in security incidents and costs directly to security awareness initiatives with any real certainty, in a real-world non-lab setting anyway.

Perhaps the answer is to turn the problem on its head. How much does it cost NOT to run an effective awareness program? One way of doing that is by benchmarking/comparisons of otherwise similar organizations that differ according to the amount of security awareness, training and education performed. Just one thing though: organizational benchmarking is itself costly and difficult and may not provide generally applicable lessons.

In short, there are no easy answers. But you’re not the only one looking.

G.

You could replace the expression “social capital” in the quoted material (indeed in the entire article from which it was drawn) with, say, “phlogiston”, and have a statement of equal explanatory and scientific value.

As to the merit of security awareness programs, and how to measure it, I’ve seen a focused awareness campaign — say , about the dangers of social engineering — followed up on with a pen test using the technique about which awareness training had been provided. Compare the number of people who “fell for it”, and how bad they fell, with results from a similar exercise on a group w/out training (or the same group prior to training) and you have a crude measure of the effect the training has.

(If you want a non-phlogiston based attempt to explain car-pooling and every other form of social action, read
this.