As I continue to ponder metrics, I have also been looking beyond the realm of Information Security & Risk Management for inspiration. That led me to a great example of effective metrics for measuring the success of foreign aid programs, a notoriously-difficult problem given the high rates of corruption and fraud in parts of the world most in need of development aid:

The aid industry faces a dilemma. On the one hand, countries are more likely to grow rich if their citizens are provided with some important basics, such as a legal system that works, or protection from corrupt officials. Such basics might seem the priority for aid money. On the other hand, it is much easier to measure success in simpler projects, such as building roads and laying pipes.

If development agencies focus on pouring concrete, they may be spending money on infrastructure that will never be used – and perhaps never even be built – because of corruption in the background. But if they focus on the broader stuff – democracy, corruption, human rights – they risk trying to do everything and achieving nothing. William Easterly, the World Bank’s most prominent apostate, argues that development agencies love big agendas because their contribution cannot be measured and found wanting. It is a bitterly cynical view, but that doesn’t mean he’s wrong.

This is an interesting problem statement, and one that resonated with me as I continue to struggle with the question of what my KPI’s should be (more on that tomorrow, maybe). I find that, all-too-often, my world of metrics breaks into one of two categories: “Things I can measure” and “Things I Need to Measure.”

So maybe it’s time for a little lateral thinking. Rather than focusing on measuring the end result, measure the effectiveness of the process. Be sure to consider, however, the motivations and reactions to the measurement. For example, how a project is measured can shift the fraud:

Another young economist, Ben Olken of Harvard, used a similar randomisation technique to work out whether corruption in Indonesian road-building projects was best fought top-down, using audits, or bottom-up, soliciting comments from local villagers about whether money was being embezzled. One challenge was to work out how much embezzlement was taking place. Olken enlisted engineers to take samples of the road’s structure and to estimate how much it should have cost to build; he compared that estimate with how much spending was claimed in the project’s accounts. The missing funds were a rough guide to the amount embezzled.

In contrast to Duflo’s results, Olken found that the bottom-up monitoring was not effective – it shifted the embezzlement from something the villagers cared about (wages) to something they did not (building materials). The threat of a guaranteed audit – a threat that was later carried out – was much more effective, reducing the estimates of missing funds by a third.

So what does this mean for me as the developer of Security Metrics? On one level, nothing. But in reality, it’s a reminder that metrics are instrumentation for the procesess and systems that comprise (in my case) an Enterprise. That information is provided in an environment where people will use it both for good and less-than-good purposes. How people interact with and utilize metrics will have significant impact on how they use or produce them.

First, it’s a reminder that people will, at a minimum, try to leverage metrics to their benefit. This is where transparency and objectivity of the data comes in. A well-designed metric will be empirical enough that the results are, if not indisputable, at least relatively spin-resistant.

Second, if results are being measured as progress toward a goal, ensure that principles of designing a secure system are properly incorporated into the process. Especially if money is involved, expect people to try to game the system.

For example, I remember from Freakonomics the case of what happened when the Chicago Public School system tied student performance on standardized tests to teacher raises and bonuses. The result was tremendous improvement, but only because of widespread, independent instances of fraud on the part of the teachers. Sure, pay-for-performance increased test scores, but not because the students were actually learning more.

Core security principles such as Segregation of Duties (teachers should not have had access to their own students’ test forms), the Two-Man Rule (minimize collusion risk), and some basic integrity checks or chain of custody (ensure the forms couldn’t be altered after the student completed the test in any case) would almost certainly have prevented both the embarrassment that CPS ultimately suffered as well as avoided rewarding the most-corrupt rather than most-skilled teachers on their payroll.

While I don’t anticipate having to deal with flat-out fraud as I design my metrics, I still want them to be resistant to manipulation or “moving the goal posts,” yet still provide the information that people want and need to effectively secure the business.

Good post!

> While I don’t anticipate having to deal with flat-out fraud as I design my metrics…

Calling it fraud is a sort of emotional response to questioning what sort of badness are we dealing with here. In practice, all people work according to the metrics placed in front of them. If the metrics call for more students to pass with higher grades, that is what teachers will do, morals and principles be damned!

Or, perhaps a more emotionally satisfying explanation is that there will always be some percentage that operates blindly to the metrics, so you may as well consider that the metrics drive all people. For a personal example, I observed a voting contest amongst socio-democratic leftie artists who all demanded the right to vote for their funding; some 10% cheated at the system and pumped up their vote, even while demanding their right for a fair vote…

The theory of “public choice” was one of the big revolutions in economics of the last 50 years; it replaced the concept “government works for the people” with the theory that “government employees work for themselves.”