I’ve been thinking hard about what makes a good Key Performance Indicator. The wikipedia entry for KPI opens with:

Key Performance Indicators (KPI) are financial and non-financial metrics used to quantify objectives to reflect strategic performance of an organization.

That’s all well and good, but it creates a new problem: They don’t resonate with people working and thinking at an operational level. This means that they effectively fail the “So what?” test applied by in-the-trenches operations staff. And if they don’t believe the metric has value, then they won’t execute in such a manner as to improve it.

For a non-security example, consider Gross Domestic Product. This is total value of goods and services produced by an economy, typically a country. Economists along with government and financial leaders use this as The Number for measuring how well or poorly an economy is doing.

There’s only one problem with it: it’s largely useless to the Average Joe who’s getting up and going to work every day to actually produce the Product.

He’s more likely interested in a KPI like “dollars earned per hour/week/year” or paycheck size, or something otherwise downward-focused and inconsistent across the economy.

I know that no one told me this was going to be easy, but that’s cold comfort some days.

At this point, the volume of reporting about SocGen is getting deafening. My favorite came from a friend and former CISO (of a different global investment bank), who declared to me Saturday over a G&T, “They let themselves get taken by an utter muppet!”)

This introduction also sums it up pretty nicely:

By Friday, January 18, Jérôme Kerviel, a junior suit in the banking world, was on the hook for €50 billion - the equivalent of about half of all the gold and currency reserves held by France. The sum also exceeded the entire value of the bank at which he worked.

The 31-year-old trader at Société Générale, one of France’s most prestigious institutions, had secretly set up a series of deals that were going horribly wrong. So wrong that they threatened the survival of the bank and the health of global financial markets.

Yet senior executives at the bank, which initially claimed that it had no inkling of Kerviel’s activities, yesterday admitted that managers had missed several warning signs over many weeks that would have revealed the apparent fraud.

I’ll bet they did.

As the details come out, this story just gets more and more interesting. The Cheese had a nice excerpt from the Forbes article, which pointed out that

as long as traders had knowledge of back-office operations, the risks of abuse would always be there.

Now it’s been almost ten years since I lived in the world of High Finance, but back then I worked as a software developer implementing risk models for a bulge bracket investment bank, first for the market risk group and later on credit and collateralized trading risk management systems.

At my bank (not SocGen), the operational functions of the risk management group were sometimes used as a sort of bush league for traders who had a hard time staying under their risk limits–they either spent a year learning how to stay out of trouble and returned to trading, became permanent fixtures in the back office, or they left left the bank.

At the time, we had such a mish-mash of systems that we really had no good unified picture of what was going on, and we were better than any of our large counterparties. We knew what our positions were alleged to be, but had no good tie back to confirm the integrity (legitimacy) of the data. In retrospect, I won’t hesitate to admit that we were lucky we didn’t get hit with this sort of fraud.

Unfortunately, it sounds like thinks haven’t gotten much better over the past 15 years. In the fallout from the collapse of Barings’ Bank, we learned that two of the key metrics that Nick Leeson was reporting back to London, Profit & Losses (P&L) and Cashflow were wildly out of alignment. If they had had a unified view of things, the story he was telling to the head office would have summarized to, “We’re making huge profits, now send more cash.”

In this case, Kerviel was able to fabricate hedges which made it look like his risk was netted off, when in fact he was naked to the market. Did he simply create dummy trades and book them against the accounts for the other liquidity providers, where there are thousands of trades in both directions at any given point in time, counting on them to get lost in the noise?

I read something which indicated that his fraudulent hedges had to effectively be unwound and recreated every three days. Perhaps he was exploiting the window between expiration and settlement.

Or did he simply exploit broken deprovisioning and internal transfer processes to aggregate access as he moved between jobs in the middle office? Eventually, he might have had the ability to create trades at the necessary points in the workflow to make the risk look netted below his limit, then disappear them later before they flowed through to where they might have upset some other reconciliation or balance.

Or perhaps he well and truly “hacked,” as some news stories have said, the back-end, gaining unauthorized access and manipulating the systems directly from the operating system or database back-end.

As much as I’d love to know, I’m also really glad that I’m not standing close enough to the mess to have that good a view of what happened, because it’s probably not that the powers-that-be didn’t realize the fraud was possible, more that they didn’t think it was an unacceptable risk.

When you consider the multitude of mechanisms that he could have exploited, some of them internal weaknesses and some of them part of the structural nature of how capital markets work, you rapidly see that closing all the holes is impossible–there will always be some material level of residual risk.

As Stiennon said referring to his pen-testing days:

It would take about four days to get an insider’s feel for operations. And, in every case, I could discover ways to steal from the client company. In my opinion the only reason that most of these companies have *not* experienced a major theft is that people in general, and frankly IT staff in particular are trustworthy.

Companies must balance the risk of a loss against the cost of mitigating (or transferring or otherwise reducing the inherent risk). Evaluating and minimizing personnel risk through background checks on the front-end and ongoing supervision and management is another.

Risk Management helps the business determine how much control is enough to prevent a major loss or fraud, but not without cost. Thus, the business must also minimize the impact of risk management on operations, both direct cost (again) and externalities like increased transaction time or inefficiency for the business functions that the security protects.

Could SocGen have decreed that every trade must be explicitly pre-approved by multiple people before executing to prevent the risk of fraud? Sure, but it would have killed them as a competitor in the market even more surely than accepting the risk of a bad apple getting through their other processes.

Thinking about measuring the effectiveness of security training and awareness, I ran across a post by Tim Hartord, the Undercover Economist, provides an excellent reminder of why metrics can fail to properly assess the actual outcomes environmental or infrastructural improvement projects:

Social capital is important but elusive. It is an attempt by social scientists to expand the traditional list of economic assets beyond physical capital, such as computers or roads; and human capital, that is you and me and whatever tricks we’ve learned along the way. Economists have attempted to account for why, in the words of P.J. O’Rourke, “some parts of the Earth prosper and others suck”. But as long as those attempts have simply measured the roads and counted the number of IT graduates, they have failed. Social capital is supposed to explain why, and social scientists, including economists, embraced the concept with enthusiasm.

(emphasis mine)

This is the sort of thing that makes security awareness so hard. What we can measure is how many people attended the training and maybe even passed a test filled with example-sized problems. Then we ignore the fact that, pretty much regardless of subject area, they will forget 95% of what they learned in the next two weeks and revert to type.

Security awareness is a driver of getting buy-in for and adoption of security-related changes to processes and technology, but is not an end unto itself, regardless of what many seem to think (unfortunately, the whole “people, process, technology” mantra drives the wrong conclusions for some people).

While it’s important to do awareness training, the changes it’s driving should be understood and incorporated through changes to processes.

Having worked in China in the past, I was vaguely familiar with guanxi, but this article sums up the dilemma it produces for Western companies better than any other I’ve read.

Western company standards, and the codes of conduct in which these standards are codified, hold that the interests of the company come first. When a person becomes an employee of a company, he accepts an ethical obligation to put the company’s interests first. Because of this, an employee who uses his position to have his company purchase goods from the employee’s friend that are slightly higher in price or inferior in quality would be regarded as unethical, even blatantly corrupt. Though such things do indeed happen in the West, few would regard this type of action as ethical. When such incidents are discovered, the offenders often are dealt with quickly and harshly by the company hierarchy, and they frequently are even reported to the legal system.

Chinese business ethics, however, are built on the basis of guanxi, which places relationships above other considerations, including an employer’s code of conduct and even the law. The idea that taking a job with a company, particularly a non-Chinese company, cancels obligations toward people with whom someone has long-term relationships and to whom one owes much guanxi is seen not only as alien but also as the essence of immorality. Perhaps even more confounding to Western businesses, the obligations of guanxi can bridge time and distance, sometimes being invoked even when two people have not seen one another for many years.

This is a tricky issue, as guanxi is a knife that cuts both ways. If your employees have lots of guanxi, then they are more likely to be highly effective. If they do not, then the opposite is probably true. Either way, the increased costs or decreased quality must be viewed as a cost of doing business which must be accepted so long as it does not cross the lines of something like the Foreign and Corrupt Practices Act.

So what’s a manager at a Western firm to do? Basically, be realistic about the practice and simply provide some boundaries around it.

Before entering into any negotiation, they state clearly and up front the legal parameters within which they can act, already laid in stone by their superiors. This shapes their foreign counterparts’ expectations of what they can squeeze out of the negotiations and keeps them from overstepping Western legal boundaries in their demands. This does not always work, but it is increasingly being used and returning results.

And lastly, guanxi may not go by the same name in Western business, but anyone who thinks the practice doesn’t exist is sorely deluded. Sure, we like to think we have a more transparent transaction structure, but more often than not, that’s to prevent the parties from defrauding each other after the relationships have determined who gets a piece of the transaction or a seat at the table.

There was much discussion of “security-by-obscurity” last week. Then, this week, we get a world-class example of security-by-obscurity failing in the real world. I’ll use the Freedom To Tinker write-up, because it’s better than any of the press accounts–as is often the case when you have real experts, rather than journalists, doing the reporting:

The new Dutch transit card system, on which $2 billion has been spent, was recently shown by researchers to be insecure. Three attacks have been announced by separate research groups.

What failed? A combination of factors, not least of which was that the engineers of the system seem to have assumed that their encryption algorithm and key would remain a secret:

Karsten Nohl, “Starbug,” and Henryk Plötz announced an attack that involved opening up a Mifare Classic card and capturing a high-resolution image of the circuitry, which they then used to reverse-engineer the cryptographic algorithm. They didn’t publish the algorithm, but their work shows that a real attacker could get the algorithm too.

Unmasking of the algorithm should have been no problem, had the system been engineered well. Kerckhoffs’s Principle, one of the bedrock maxims of cryptography, says that security should never rely on keeping an algorithm secret. It’s okay to have a secret key, if the key is randomly chosen and can be changed when needed, but you should never bank on an algorithm remaining secret.

Read the post for the full details, but the next time someone tries to muddy the waters by arguing about what is or isn’t security-by-obscurity, take a look back at this case study of spectacular failure.

[Update: thoughts from John Quarterman below]
[Update2: notes from a friend with some insider info]
This tale of extortionists taking power plants off-line by attacking their computer systems is getting a lot of play right now, at least in the InfoSec press.

Criminals have launched online attacks that disrupted power equipment in several regions outside of the U.S., he said, without identifying the countries affected. The goal of the attacks was extortion, he said.

“We have information, from multiple regions outside the United States, of cyber intrusions into utilities, followed by extortion demands,” he said in a statement posted to the Web on Friday by the conference’s organizers, the SANS Institute. “In at least one case, the disruption caused a power outage affecting multiple cities. We do not know who executed these attacks or why, but all involved intrusions through the Internet.”

This whole thing seems light on fact and long on titillating detail to me personally–more the realm of Movie Plot Threats than an unacceptable risk. Even a casual (google news search for “power outage”) didn’t produce any hints–everything was either domestic to the U.S. or localized and the cause explained (worker electrocuted himself and his body shorted out the grid, etc.).

Nevertheless, people are coming out of the woodwork to defend the speaker as credible:

Having worked with Tom Donahue on these and related issues in the past, I regret to inform conspiracy theorists that he is virulently allergic to hyperbole. That he might be making these statements lightly are about as likely as any sane person playing Russian roulette with a semi-auto pistol.

But this leads to an interesting question, namely, Is the power grid vulnerable to a systems-level attack against it? We know that the United States’ 2003 blackout of the Midwest and Northeastern US and parts of Canada was caused by a cascading failure of SCADA systems essentially making the system unmanagable and taking it off-line.

So what is the level of risk? According to Alan Paller, Director of Research at NIST,

The prospect of cyberattacks crippling multicity regions appears to have prompted the government to make this information public. The issue “went from ‘we should be concerned about to this’ to ‘this is something we should fix now,’ ” said Paller. “That’s why, I think, the government decided to disclose this.”

Assuming that we use the 2003 outage as the potential impact, make some conservative assumptions that the state of security awareness and systems resilience has not improved materially since that incident, then we must assume one of two things: Either the threat has escalated or The Experts are playing FUD games with us.

We have an issue which has been known since at least 2003, when we suffered the first major incident, where suddenly all sorts of “credible” experts (or so we’re told). What are they hoping to accomplish by suddenly going public with a vulnerability that was effectively defined in the aftermath of the 2003 outage? Is this effectively a form of disclosure similar to public software vulnerability reporting as a tool of last resort to drive improvements in the electrical generation system?

Personally, I suspect an evolution in the Threat–a position that John Robb seems to have reached as well. He’s written extensively about the evolution of systems disruption attacks. Now imagine the synergy of the kidnapping-for-ransom/extortion business models so common in Latin and South America these days combined with the technical expertise that’s producing work like the Storm worm and an outdated, under-maintained IT infrastructure directly tied to real-world impact, and the threat seems increasingly credible.

Just because it sounds dramatic doesn’t mean it isn’t so. As the old saying reminds us, “Just because you’re paranoid doesn’t mean that no one is out to get you.”

So will I be researching options for going “off the grid” in response to this? No. Actually, I’ve already been looking at off-the-grid options for economic and environmental reasons, but this is something I might factor in to my ultimate decision.

[Update: An additional theory on motivations:]
As Adam pointed out, John Quarterman suggested on Dave Farber’s Interesting People mailing list that this might be a used to create a threat whose solution will turn out to be more monitoring of Internet Traffic. I already deleted the message, but had to agree that it might make sense, given the strong pro-wiretapping agenda of the agencies now propagating and vouching for one another, combined with the Urban Legend-/Weekly World News-grade facts that allegedly support the story.

[Update2: Notes from a friend with some insider info]
On the power grid thing. I have done a bit of a study of some control networks and there is quite a bit of risk out there. They have been taking old systems and just will-nilly adding them on to IP networks. this stuff used to all be dialup. They run 20 year old OS’s or worse. I do not know about all the current noise in the press. I am part of a team with [a friend] working in the industry to start making changes. There is a lot of crazy stuff. Operations and IT do not talk to each other….

It should come as no surprise, but Alex Hutton said something smart:

controls that work to prevent threat action by lowering the probability of action are extremely under-appreciated.

Everyone is so focused on reducing vulnerability or impact, which are inherently reactive (and a losing fight to boot. Review OODA for why.), rather than focusing on the various preventative options and applying those which give the best bang-for-buck, which is where we might actually have a chance to make a difference.

Of course, he also said

Regular readers will note that we believe controls basically give us some value to prevent/detect/respond to loss events. What we haven’t revealed is that obscurity (or other controls that prevent the probability of threat events) is one preventative measure that actually, in our models, seems to have a boatload of value. Yes, boatload is the qualitative label we use

I would like to take this moment to point out that the man who’s using “boatload” as a term of art took issue with me summarizing unacceptable risks as things for which I “turn projects yellow.” ;-)

Chicago, as you may or may not know, outlawed the serving or consumption of fois gras in 2006. True foodies, however, will not be denied their delicacy. While some restaurateurs have chosen to openly defy the ban, enjoying the publicity and challenging the law in court, others have chosen a more interesting (to me, at least) response.

From an invitation a friend just sent me:

The amazing chefs there will tantalize us with a 4 course Forbidden Foie Gras degustation menu complete with wine pairings. (I have done this with them before and it was amazing.) At this meal we will hand out 2 business cards to each person at the dinner with a logo and names of a few additional restaurants that will be participating in this little club. After this meal anyone with a card may present it at any of the named restaurants for a main course or appetizer that features Foie Gras, or as we are referring to it, redacted–ed.. You will need a card to get the secret serving at these restaurants.

While the subterfuge is as much for theatrics as security, I suspect that it is a sufficient countermeasure against anything but a warranted, targeted police raid of the establishment’s kitchen. It also, to an infoSec geek like myself at least, provides a nice example of non-IT use of two-factor authentication:

Something you Know — The name of the dish to ask the waiter about
Something you Have — The card identifying the presenter as someone “in the know” about the availability of fois gras on the menu

Bon Appetit!

Much of what I do these days is serve as a buffer between the people who actually do work and the committees that think they are ensuring the work gets done. As such, when Ian Grigg posits Hypothesis 4.2, there is nothing I can add except my wholehearted agreement:

Simplicity is proportional to the inverse of the number of designers. Or is it that complexity is proportional to the square of the number of designers?

Sad but true, if you look at the classical best of breed protocols like SSH and PGP, they delivered their best work when one person designed them. Even SSL was mostly secure to begin with, and it was only the introduction of PKI with its committees, models, digital signature laws and accountants that sent it into orbit around Pluto.

…

…It should be clear by now that committees are totally out of the question. They are like whirlpools, great spiralling sinks of talent, so paddle as fast as possible in the other direction.

Actually, I do have one thing to add. This pre-supposes that the individuals doing the design work are qualified to do so.

Lotteries, goes the saying, are a tax on people who are bad at math. How bad? So bad that that Britain’s National Lottery is able to market using data that I would like to think is an argument against playing the lottery.

From a suspicously press release-ish BBC Article:

Medway Towns has one top prize-winning ticket for every 6,119 adults who are eligible to play the Lottery, compared with Ilford in second place with one in every 6,839.

That’s not so bad, too many people probably think (The ones who don’t know about Base Rate Fallacy, which is to say, most of them. After all, their reasoning goes, that’s much better than stated odds from an old IHT article:

The chances of [winning the lottery] actually happening are slim — about one in 3 million, on average, according to industry figures

I don’t recall what the frequency of drawings is for the National Lottery, but even if there were a drawing every single day, the average elapsed time to win the lottery (assuming buying a ticket for every single drawing) would be every 16 years and 9 months. And during that time, you’d have spent over £6,000 (plus time value of money, ignoring the time involved to purchase a ticket and then see if it won) to do it. If you lived in Ilford, you’d be looking at waiting almost 19 years (on average) to see any money back, and the queue length just goes up and up from there.

Another thing which is not mentioned is the period of time over which that rate of winning accumulated. I get the impression that’s since the National Lottery was founded in the mid-1990’s. Without knowing the effective time period, we’re missing a key variable whose omission further distorts the data.

Of course, I still haven’t answered my original question. Fortunately, as I sought data on how much the average payout was (since all I can seem to get from public sources is averages), I found this page at the UK Treasurer’s Web site, which I realized actually tells me all I really needed to know:

For every £1 spent on the National Lottery…50p returned as prizes

At the risk of putting myself in tax peril for my math skills, I’ll argue from that fact that the tax rate on poor math skills (the part that the government doesn’t give back) is effectively 50%.