From the BBC:

A French battleship sunk in 1917 by a German submarine has been discovered in remarkable condition on the floor of the Mediterranean Sea.

The Danton, with many of its gun turrets still intact, is sitting upright in over 1,000m of water.

It was found by the Fugro geosciences company during a survey for a gas pipeline between Algeria and Italy.

…

The final resting place is a few kilometres from where people have traditionally thought the ship met its end.

“The French Admiralty did argue with us for a while that it should have been several nautical miles away, but we reminded them that modern GPS methods are more accurate than the sextants they used in those days,” said Mr Hawkins.

In other words, the French Admiralty tried to pull the Who are you gonna believe, us or your lyin’ eyes? routine.

Keep this anecdote handy for the next time you’re dealing with metrics. Good metrics frequently discover that the Conventional Wisdom is flat-out wrong and that things are not what or where people thought they were. A little humor can go a long way toward defusing the situation, getting people out of a defensive posture and back on track to thinking productively about the problems at hand.

It doesn’t matter if you get the right answer or not unless you ask the right question.

Paul Krugman explains this concept scarily well using Friday’s unemployment reports.

Or, to quote myself, “it is entirely possible to be both absolutely accurate and utterly wrong.“

For those of it who don’t have a subscription to IEEE’s Security & Privacy newsletter, we can still get a PDF of Dan Geer & Dave Conway’s new essay/column/article, “Security is a subset of Reliability.”

Reliability measures the deviation between the system and the specification. Security involves a sub-space of reliability—only particular deviations—thus, security must be easier than reliability. Thumbs up. Hastening over the delicate premise that the specification is always accurate and up-to-date, we can roughly align security with the subset of reliability where the cost of deviation per unit time is very high. Thumbs down. This makes us wonder about measuring how risk tolerance scales and consequently where to point our thumbs.

Similarly, since I’ve been talking to lots of engineering types about security issues of late, I’ve been using the argument that “security issues are a subset of defects, especially post-release defects, which tend to have a higher-than-average severity.” This gets their attention since PRD’s are something they worry about, even if they consider it somehow “unfair” that they’re a category of defect that they aren’t very good at.

I’ll definitely be passing this article along to a few people I know.

Despite coming from a somewhat-unlikely source, this Boing-Boing post is a re-print from last year analyzing how press releases from the National Retail Association get turned into “news” is a good example of metrics (or, more likely, made up crap) get turned into “facts” by an uncritical media, specifically CNN:

[I] examined the retail numbers cited by the National Retail Federation about sales over Thanksgiving, and so-called Black Friday. I made the point that this news is fake news, coming from a press release generated by a retail trade organization and then spoon-fed to us by uncritical reporters. While the stories credit the source, the headlines give the impression that the retail industry wants, using numbers they provide. (Reporters like a story with specific numbers, no matter how contrived they are. Independent backup for the numbers is never provided.) There’s every reason for NRF to present numbers that favor their view that consumers will be buying more. It’s like asking the fox to count the eggs in the hen house and report on the health of the chickens.

A newspaper article observes an interesting correlating economic indicator:

The nation’s largest employer and retailer is a bellwether for many things, but theft may be its greatest contribution. Due to the sheer size of its stores – coupled with chronic short-staffing and no security staff – Wal-Mart tends to be ground-zero for shoplifting.

The evidence comes from the discarded packaging found in the deeper reaches of the stores. In normal times, shoplifters will grab CDs, DVDs, and smaller electronics items, strip them of packaging in the quieter aisles, then walk through security scanners undetected.

But over the past few months, workers are discovering that even thieves are having a hard go of it during this wretched economy. “Now I’m finding lots of things like food, diapers, tampons, over-the-counter pharmacy stuff like kid’s cough medicine and insulin,” says one employee.

Metrics are often not found where you expect them.

There has been a flurry of news stories of late about how high gas prices have people turning to scooters. You’re expected to listen to the “high gas prices drive scooter demand” angle, but the whole thing falls apart if you actually work the numbers. I like to think of this as “Metrics Judo”–turning someone’s own facts and figures back on them to get to the core of the argument.

As my old boss used to advise, “Before you get into any discussion of the issues, always make sure everyone agrees on the facts.”

For example, this morning I heard this story on NPR while driving to work. I was going to ride my scooter, but they’re predicting severe thunderstorms for the afternoon commute, which is no fun at all on two wheels, so I took my wife’s car. She’s not terribly pleased, but that’s life sometimes. If I had a public transit option, I’d take it in a second.

Getting back on topic, this story has a stronger safety component than most stories of its ilk–it actually stresses that scooters are dangerous, for a change, something that I have harped on in the past.

That part I liked.

But what I didn’t like was that the facts don’t actually support the story. First, the shop owner in SanFran said that his sales are up from ~6 scooters/month to his full shipment of 40/month. But he says he’s “thinking of” offering a promotion of “free gas all summer,” which he figures would cost him only $40/scooter. He also says that a scooter along with “all the safety gear” is only $3,000.**

If he’s selling out his inventory every month, why would he spend $1,600/month on a promotion? He’s supply-constrained, so the promo can’t increase revenue. The owner of a successful small business has to know that, or he won’t be a business owner for long. So, ignore everything but the cost per scooter–$40 or less than 10 gallons of gas per-scooter for the summer.

Then, we take that $40 cost over the course of the summer, compare it to the $3k-6k that people are spending for a scooter to save somewhere between $40 and $275* over the course of a “summer” (which I’ll define as May-Sept, or 5 months), depending on how heavily you want to weight the model in the scooter’s favor. That means that the average annual savings is between $100 and $700/year. This obviously doesn’t add up to the cost of the scooter, and it didn’t take anything but paying attention and some arithmetic to know that.

The only way that a scooter makes any sense at all is to do what I did–get rid of the car and buy the scooter instead. I did that and have been money ahead for years. The fact that I live in a highly-congested urban area where a scooter provides a tactical advantage in maneuvering through gridlock traffic and parking when I get to my destination is just gravy.

Now, turn this back to IT and/or Risk Management. How many times have we all been presented with data by some vendor which, with a little analysis, can easily be pulled apart and used to produce either the real value (rather than the one the marketing people want us to focus on) or, getting algebraic for a moment, allowed us to solve for the “real” data which, if we have to jump through these sorts of hoops, probably isn’t going to say what the vendor wants us to hear?

* Assume 60 mpg*10 gallons = 600 miles on a scooter, 30mpg*20 gallons = 600 miles in a car, so net fuel reduction of 10 gallons @ $4/gallon. Or, for a more fully loaded value, use $0.53/mile, the standard deduction for operating a vehicle, and now you’re looking at a cost avoidance of, at most, $278 ($318 - $40 for gas on the scooter, but no operating costs, so it’s still apples-to-oranges).

** He’s selling cheaper scooters than I ride. My scooter was $2,800, my helmet was about $250 on sale, I have two jackets, which were each about $200, my boots were over a $100 (leather, steel shank & toe, hard vibram soles), gloves another $50. A pair of armored trousers would be another $120 or so, which I’ll probably buy sooner than later now that I’m riding to work on a regular basis. Yes, I’m a bit of a safety nerd–I got of easily on learning that lesson the hard way.

I’ve been pretty busy lately, which has impacted, among other things, my time and energy for blogging. I don’t know when I expect this to materially improve, so I’m going to fire these thoughts out before I run off to my next meeting even though they may still be a bit partially-formed.

Lately, I’m feeling that there are two fundamental problems with risk and security metrics.

The first, which I’ve written about previously, is that they don’t scale the corporate ladder well. The second, and perhaps more serious from an industry perspective, is that the metrics which do scale the corporate ladder well don’t compare well across industries or even within industries. Thus, there seems to be a paradox here: the more business value a security metric represents, the less either generic or share-able it will be.

For example, take metrics related to policy compliance, one of my KPI’s. I assume that policy (or lack thereof) is an expression of or proxy for a company’s tolerated level of risk. Given that no two companies have the same policies (unless they both cribbed them whole-cloth from SANS), the risk measurement is going to be inherently different between companies. Throw in the fact that most companies won’t be willing to share this data in anything but a tightly-controlled forum, and you’ve got a real problem.

Nevertheless, I’d still be pretty happy if we could get general agreement (or even understanding) across so-called risk managers that, like it or not, policy effectively defines organizational risk acceptance. With that starting point, we might then actually be able to begin doing meaningful comparisons of different policy/control sets (e.g. does CObIT+SoGP produce better compliance (as measured through audit findings & exceptions) than CObIT+ISO-27001? *That* would be an interesting and worthy research project, IMHO), although the vested interest factor could definitely hurt the effort.

And one final piece of the puzzle (and this is probably too much to even dream of, but keystrokes are cheap) would be to then correlate these measures of relative compliance to operational metrics. While correlation is not causation, we still might then be able to begin using compliance as an attribute to describe our accepted level of risk rather than as an end unto itself.

Don’t assume that traditional measures are good measures. For an example, The Economist looks at GDP growth:

WHICH economy has enjoyed the best economic performance over the past five years: America’s or Japan’s? Most people will pick America. The popular perception is that America’s vibrant economy was sprinting ahead (albeit fuelled by credit and housing bubbles that have now painfully burst), whereas Japan crawled along at a snail’s pace. And it is true that America’s average annual real GDP growth of 2.9% was much faster than Japan’s 2.1%. However, the single best gauge of economic performance is not growth in GDP, but GDP per person, which is a rough guide to average living standards. It tells a completely different story.

(emphasis mine)

For example…

Using growth in GDP per head rather than crude GDP growth reveals a strikingly different picture of other countries’ economic health. For example, Australian politicians often boast that their economy has had one of the fastest growth rates among the major developed nations—an average of 3.3% over the past five years. But Australia has also had one of the biggest increases in population; its GDP per head has grown no faster than Japan’s over this period. Likewise, Spain has been one of the euro area’s star performers in terms of GDP growth, but over the past three years output per person has grown more slowly than in Germany, which like Japan, has a shrinking population.

Some emerging economies also look less impressive when growth is compared on a per-person basis. One of the supposedly booming BRIC countries, Brazil, has seen its GDP per head increase by only 2.3% per year since 2003, barely any faster than Japan’s. Russia, by contrast, enjoyed annual average growth in GDP per head of 7.4% because the population is falling faster than in any other large country (by 0.5% a year). Indians love to boast that their economy’s growth rate has almost caught up with China’s, but its population is also expanding much faster. Over the past five years, the 10.2% average increase in China’s income per head dwarfed India’s 6.8% gain.

So, if you’re a Finance Minister, you’re apparently going to go with the number that makes you look best (total growth) rather than the number that most accurately reflects the economic fortunes of your populace–and even that number is probably not as good as median per-capita growth per-head, especially as a measure of relative change. The Minister knows better (I hope), but presents the less-honest number and knows that the vast majority will never catch him at it.

I’ve been looking at my anti-virus metrics of late, and I’m thinking that I’ve been asking the wrong questions there. Basically, I’ve got two different sets of anti-virus metrics, coverage rates (% of machines with anti-virus deployed by region) and infection rates (% of machines with infections, again per-region).

But I noticed this morning that, depending on how I’m defining my population, we’re only seeing 1-2% of the identified infections. That is, itself, only 7% of my total system population, or 0.1% (1/10th of 1%) of my total population calling the help desk due to malware problems every month.

So I’ve been failing my own first question for security issue–is this a problem I have?

Amrit asks, “Is the cure costlier than the disease?” regarding desktop security agents. His story starts out familiarly enough:

When I was still an analyst I was part of the mobile workforce, coming into the office maybe once or twice a year. The company owned laptop I was provided ran 4 different security agents, plus several other agents for various systems management functions (asset, configuration, etc) and remote access. Since the majority of the time the company had no ability to manage these mobile systems they would enforce some fairly draconian security policies, such as locking down aspects of the OS, disallowing certain protocols and applications to traverse the network VPN, as well as configuring the various scan-based security technologies to scan the system on a recurring basis (OK so maybe these are all reasonable and I felt they were draconian because I suffer from a Nietzsche “super-employee” complex and believe myself to be above the normal security policies of other employees - coincidentally I stopped using the corporate supplied laptop and switched to a Mac) .

Here is the kicker, my machine suffered from significant performance problems. Not only did it now take a good 5+ minutes to restart, it was unusable during a scan - which meant I was unable to work several hours a week

This is the story of the life of the average “enterprise” worker. In a past life, we were effectively told, “you can’t add any more agents unless you take one of the existing ones away.” Today, I “only” suffer from two or three different security-related agents on my laptop, which is especially ironic given that I do much of my work inside a virtual machine running Ubuntu Linux.

Getting back to Amrit, though, he’s kind enough to provide a great Back-of-the-Envelope (BOTE) analysis of the costs of providing desktop “security” for a theoretical 5,000 person company.

How’s it stack up, and to what? Amrit uses the same data set I used for my DLP and Full Disk Encryption BOTE analysis, the ISF’s Annual Survey, which told him

“The Computer Security Institute conducted a survey of 538 computer security practitioners in corporations, government agencies, financial institutions, medical institutions, and universities in the United States. Their results revealed that 85 percent of respondents had detected computer security breaches within a twelve-month period. The 35 percent who listed a financial impact reported $377,828,700 in financial losses. Of these, many cited their Internet connection as the point of attack for hackers.”

I’m not going to give you the spoiler–you can go read it yourself–other than to say I wholeheartedly agree with his assumptions, his methodology, and his conclusion.