Comments on: Do awareness metrics fail the “So What?” test? http://thurston.halfcat.org/blog/2008/01/26/do-awareness-metrics-fail-the-so-what-test/ We are the people your IT department warned you about Mon, 15 Mar 2010 12:10:31 +0000 http://wordpress.org/?v=2.0.5 by: Chris http://thurston.halfcat.org/blog/2008/01/26/do-awareness-metrics-fail-the-so-what-test/#comment-190495 Wed, 30 Jan 2008 02:14:18 +0000 http://thurston.halfcat.org/blog/2008/01/26/do-awareness-metrics-fail-the-so-what-test/#comment-190495 You could replace the expression "social capital" in the quoted material (indeed in the entire article from which it was drawn) with, say, "phlogiston", and have a statement of equal explanatory and scientific value. As to the merit of security awareness programs, and how to measure it, I've seen a focused awareness campaign -- say , about the dangers of social engineering -- followed up on with a pen test using the technique about which awareness training had been provided. Compare the number of people who "fell for it", and how bad they fell, with results from a similar exercise on a group w/out training (or the same group prior to training) and you have a crude measure of the effect the training has. (If you want a non-phlogiston based attempt to explain car-pooling and every other form of social action, read <a HREF="http://www.amazon.com/Foundations-Social-Theory-James-Coleman/dp/0674312260/ref=pd_bbs_1?ie=UTF8&s=books&qid=1201659117&sr=8-1" rel="nofollow">this</a>. You could replace the expression “social capital” in the quoted material (indeed in the entire article from which it was drawn) with, say, “phlogiston”, and have a statement of equal explanatory and scientific value.

As to the merit of security awareness programs, and how to measure it, I’ve seen a focused awareness campaign — say , about the dangers of social engineering — followed up on with a pen test using the technique about which awareness training had been provided. Compare the number of people who “fell for it”, and how bad they fell, with results from a similar exercise on a group w/out training (or the same group prior to training) and you have a crude measure of the effect the training has.

(If you want a non-phlogiston based attempt to explain car-pooling and every other form of social action, read
this.

]]> by: Gary Hinson http://thurston.halfcat.org/blog/2008/01/26/do-awareness-metrics-fail-the-so-what-test/#comment-190120 Tue, 29 Jan 2008 08:04:06 +0000 http://thurston.halfcat.org/blog/2008/01/26/do-awareness-metrics-fail-the-so-what-test/#comment-190120 'The number of people who attend the training' is about the lamest of lame awareness metrics - but you're right, organizations do measure it for some obscure reason, mostly extreme creativity bypass I guess. Tests provide metrics with a bit more meaning and purpose but you've hinted at a different kind of metric: there are lots of ways of measuring 'buy-in and adoption of security-related changes' if you're prepared to embrace the social scientists, psychologists and others who specialize in measuring people and what they do. They use surveys and observational techniques. They describe scenarios and invent cunning experimental situations to home-in on individual facets of behavior. And they struggle to measure even the simplest of animal behaviors. I agree wholeheartedly with you that security awareness is a means to an end not an end in itself. However, end-focused security metrics are unfortunately a poor guide to the effectiveness of an awareness program: there are far too many variables to assign a reduction in security incidents and costs directly to security awareness initiatives with any real certainty, in a real-world non-lab setting anyway. Perhaps the answer is to turn the problem on its head. How much does it cost NOT to run an effective awareness program? One way of doing that is by benchmarking/comparisons of otherwise similar organizations that differ according to the amount of security awareness, training and education performed. Just one thing though: organizational benchmarking is itself costly and difficult and may not provide generally applicable lessons. In short, there are no easy answers. But you're not the only one looking. G. ‘The number of people who attend the training’ is about the lamest of lame awareness metrics - but you’re right, organizations do measure it for some obscure reason, mostly extreme creativity bypass I guess. Tests provide metrics with a bit more meaning and purpose but you’ve hinted at a different kind of metric: there are lots of ways of measuring ‘buy-in and adoption of security-related changes’ if you’re prepared to embrace the social scientists, psychologists and others who specialize in measuring people and what they do. They use surveys and observational techniques. They describe scenarios and invent cunning experimental situations to home-in on individual facets of behavior. And they struggle to measure even the simplest of animal behaviors.

I agree wholeheartedly with you that security awareness is a means to an end not an end in itself. However, end-focused security metrics are unfortunately a poor guide to the effectiveness of an awareness program: there are far too many variables to assign a reduction in security incidents and costs directly to security awareness initiatives with any real certainty, in a real-world non-lab setting anyway.

Perhaps the answer is to turn the problem on its head. How much does it cost NOT to run an effective awareness program? One way of doing that is by benchmarking/comparisons of otherwise similar organizations that differ according to the amount of security awareness, training and education performed. Just one thing though: organizational benchmarking is itself costly and difficult and may not provide generally applicable lessons.

In short, there are no easy answers. But you’re not the only one looking.

G.

]]>