Maybe the problem with Quantitative Risk Management is the same problem which Mark Kleinman notes about Game Theory in the Real World

“Game theory” is a branch of mathematics usable by social scientists, not a social-scientific theory. It’s a deductive account of what will happen if actors act so as to maximize their own outcomes in situations in which the outcome for each depends on the behavior of others as well as his own. It doesn’t predict anything about the real world, any more than algebra predicts anything about the real world.

In order to generate predictions using game theory, you need to add some facts: about the outcomes of different combinations of actions, about what the actors want, and about their rationality (vel non). Given such assumptions, it is possible to compare the results of real-world events to game-theoretic conclusions.

When they match, then it’s reasonable to think that you have correctly identified the outcomes as the players evaluate them and that the players are acting as selfishly rational actors. When they don’t match, then either you’ve got the outcomes wrong, or the players aren’t trying to act selfishly, or they’re trying to act selfishly but making mistakes.

You could pretty much say the same thing about risk analysis. Try it out:

In order to generate predictions using risk analysis, you need to add some facts: about the outcomes of different combinations of actions, about what the actors want, and about their rationality (vel non). Given such assumptions, it is possible to compare the results of real-world events to risk analytic conclusions.

That’s not to say that there’s not still a lot that can be done using Risk = threat * vulnerability * impact, but that’s still a long way away from using tools like Monte Carlo Simulation or methodologies like Value At Risk to deal with the inherent uncertainty of Risk.

Maybe Dan Geer is right and the future of Information Security truly does belong to the Quants.

While procrastinating from mapping controls to risks, I just flipped over to eWeek to kill a little time and learned that Skype just released updates to cover multiple critical security vulnerabilities.

Multiple security flaws in the popular Skype voice chat application could put millions of users at risk of computer takeover attacks, the company acknowledged Tuesday.

Skype Technologies S.A., which is being acquired by eBay Inc., warned in two separate advisories that the vulnerabilities could lead of system access or denial-of-service attacks.

The Skype program, which uses peer-to-peer technology to route phone calls over the Internet, is one of the most popular desktop applications sitting behind firewalls, making the threat vector even more serious.

So I head over to the Skype’s Web site to see how they present it. And what I discovered is that they’re not presenting it. Not on the homepage, not on the download page, not on the developer page, not even on the main security page, which I found only by manually typing “http://skype.com/security” into my browswer. But was the vulnerability notice there? Nope. Their bought-and-paid-for cryptography review was there, though. Finally, I found it on their vulnerability bulletins page. Whew. After all that, I was almost too tired to read the notices.

Still, I plowed on and soon discovered that the first vulnerability is a nasty one (The second is a boring ol’ DOS). An exploitable buffer overflow in Skype’s URI parser for “callto://” and “skype://”. Not good. Host the malicious link on an SSL’ed Web server and you’ll blow right past any IDS or IPS countermeasures and quickly be 0wning machines inside firewalls in no time, then potentially using Skype’s own crypto and peer-to-peer architecture as the control channel for whatever botnet or other bit of nastiness the attacker wants to install.

The Blackhat in me salivates at the prospect. It’s beautiful security judo, leveraging tools designed to protect confidentiality (crypto) and Availability (peer-to-peer) to better hide my nefarious doings. Combine it with a skype API-based payload and you’ve got a Skype worm that can leverage the implicit trust relationship of contact lists to propagate further, all potentially wrapped inside Skype’s own crypto.

Too bad the first that most of Skype’s 60 million-and-growing users will ever hear of it will be after someone who does pay attention to these sorts of things decides they want to see if it’s possible to create a 60-million node botnet or retire after making The One Big Score with SkypeOut and toll fraud.

Hey Skype, Ignoring Risk is Accepting Risk–NOT Avoiding it. Put this on your main page while upgrading is still prevention rather than incident response.

So according to SecurityFocus, there’s now a trojan which claims to be a new version of Skype

The malware arrives in an attachment in messages posing as the latest (v1.4) release of Skype. Legitimate downloads of the software only came out last week, so the attack is timely. If users open the infected payload on a vulnerable Windows machine they will find their PCs transformed into zombie clients (theoretically at least) under the control of computer hackers.

Don’t trust patches or software distributed via email. That’s all there is to it.

Pete Lindstrom corrected my spelling and offered his own suggestion for a Universal Risk Measure:

There are two ways to respond to this problem - First, we can assert that we do have a unit of measurement - the % point, because risk is simply the likelihood that something bad will happen, and “bad” can be anything we want it to be. Secondly, we can simply suggest that it doesn’t really matter - any unit is fine, as long as everyone is clear on it. I have never heard anyone say that they couldn’t understand the risk because the unit of measurement was wrong.

I think I disagree with the % (percent, as opposed to its alternate usage, the modulo operator) as a Universal Unit of Measurement. Picking freqency or likelihood of occurance doesn’t do us much good since, unfortunately, there are only two levels of likelihood in most people’s mind when dealing with risk:

Definitely will happen (probability 1) — i.e. worms or viruses. In the eyes of The Business, things only move into this category after technology solutions exist, which I should be busy providing instead of wasting their time with meetings and trying to turn their project’s status to Yellow.
Never will happen (probability 0) — i.e. Systems intrusions by skilled attackers resulting a massive loss-of-data or a hurricane destroying New Orleans. (Note: this eventually gets turned into, “Definitely will happen,” but only after it’s too late to do anything about it. For further reading, see Code Red, SQL Slammer, or anything in Adam Shostack’s privacy breach category.)

Pete also gets to deal with more risk-aware people than I do if he’s never had anyone claim that the unit-of-measure kept them from understanding risk. Maybe that’s because he’s a consultant, so if people are talking to him, they understand the concept well enough to think he’s worth paying Real Money to discuss it with them. I, on the other hand, live in the corporate world and have to descend like some sort of office ninja on those whose projects catch the wrong eye or need connectivity to a third party.

As a result, I spend a lot more time trying to explain to people why we’re all sitting on the phone or in a room together, especially given that their default viewpoints are
1) The incident in question Never Will Happen
2) I’m probably about to turn their project status Yellow and thus must be opposed with all their mortal vigor

So maybe they’re being deliberately obtuse, but that fact doesn’t make my life any easier.

To make matters worse, I completely agree with Pete that security practitioners have a nasty habit of playing loose with the fact that, “‘bad’ can be anything we want it to be.”

I’m guilty of this myself, usually because someone asks the, “What’s the worst thing that could happen?” question. The answer, of course, is that their systems will vanish in a blaze of keystrokes right after ueber-l33t h4×0rs sneak through their probability-zero hole and ravage our internal digital landscape like the Mongol Horde in Kiev. This may be the l0w-likelihood, high-impact scenario, but since they asked, I tell them. This also provides a nice juxtaposition with the more likely scenarios such as a disgruntled insider, a lack of clear system ownership, excessively loose administrative access policies, or any number of other issues which they should have thought of ahead of time but didn’t bother to either ask me (or someone like me) or to read our standards documents.

Which brings me back to my original point. We’ve got two variables that we can potentially control, how likely is the event and how often does it happen. Unless they agree that the probability is 1, we’re screwed since anything times zero (the only other option in their mind) is zero, which makes the risk zero, and so they can now decide in their own mind that the Information Security risk has been managed, which means that their project is now back to Green and life is once again good.

I would have benefitted from a new AI System that’s being tested in the United Kingdom a couple weeks ago.

Computer scientists at the University of Aberdeen, UK, were asked to generate an “artificial weatherperson” by operators of offshore oil rigs, who wanted more clarity in their forecasts. The vocabulary used by different forecasters can be vague and highly variable, says Ehud Reiter, who led the Aberdeen team.

While this is simply an irritation to most of us, it can be a big headache for the offshore oil industry, where unexpected bad weather can damage equipment and threaten safety.

When Reiter and his team compared past weather bulletins with the raw forecast data on which they were based, they found a striking variability in the choice of words used by different forecasters. For instance, when they talked of “evening” weather, some meant conditions around 6 pm, while others meant much nearer midnight. “Late morning” could mean anywhere between 9 am and noon. The UK’s Met Office is also reviewing how effectively its forecasters communicate.

An imprecise weather forecast (”wave heights increasing through the afternoon”) recently resulted in myself and a friend getting caught out in 8 foot seas in a 23′ sailboat at lunchtime. We made it back in one piece, but it was the toughest sailing either of us had ever done. My buddy literally kissed the ground when we got back ashore. If the forecast had been, “wave heights increasing to 6-8 ft after 12pm,” we would never have left the harbor.

Of the many difficulties I face when trying to quantify risk, lack of a consistent nomenclature is one of the more significant. Currently, we can’t even agree in which units we should denominate the discussion.

To make matters worse, we’re expected to measure our estimates in different units depending on the audience. While this is implicitly an indicator of the audience’s agenda, it doesn’t change the fact that we need to know what that agenda is, then translate all of our arguments accordingly.

Management likes Dollar Value. Their concern is the impact this is going to have on the financial performance of (their project|their department|the company|their compensation).

IT Operations staff want to hear it in terms of outage length or staff hours required to support the mitigation strategy. They, in turn will run it through their own equations to turn the cost into dollars for their management.

Regulators and auditors want to talk about risk in terms of the effectiveness of the solution and in terms of preventative or detective controls. They could care less about cost.

The public and the press want to hear about risk in terms of lives lost and impacted. The bigger the numbers, the better. Which makes it strange that they haven’t picked up on Avian Flu more strongly than they have. I guess that “bigger is better” only goes so far.

The people left in the middle questing for some sort of Risk Rosetta Stone are people like me.

To make matters worse, risk means something different to everyone. This is more than just the demand for different denominations for different audiences. To most of the non-security/risk people I talk to, risk is “Project Risk.” That is to say, “What are the odds that this will affect my project timeline or resource estimations?” This is because their success or failure is generally determined by how close to on-time and on-budget they complete their project. Whether it puts the company’s information at risk is Somebody Else’s Problem.

In their minds, Risk Avoidance is ignoring what I have to say because once they know about the Information Security risks, they might have to alter their project plan which could turn their project from “Green” to the dreaded “Yellow” on the stoplight report/dashboard which is all that senior managment ever sees.

I’ve found that unless I get my concerns addressed before the project timeline and budget are set, they probably never will be. And even when they are included in the project’s requirements, I have to make sure they don’t get cut to keep the project Green on the dashboard. But it’s better than nothing.

Now in closing, I want all the Project Risk people to repeat after me, “Ignoring the Risk is equivilent to Accepting the Risk, not Avoiding the Risk!”

Not that it’ll help, since they are compensated based solely on how well they manage Project Risk, but it’s worth a shot.