I was disappointed, though not at all surprised, to learn that The DHS has no Disaster Recovery Plan:

It’s disheartening. It’s incredible. But it’s not all that surprising. That’s how some business continuity experts and government officials reacted to the news that 15 out of 19 agencies under the Department of Homeland Security lack fully operational disaster recovery sites—a shortfall that could hinder DHS’s ability to carry out its mission during a service disruption or national emergency.

DHS’s ability to to carry out its mission during a disruption or emergency is hindered by a lot more than their lack of a DR plan.

From what I can tell, the best way to keep a building from catching fire would be put these clowns in charge of burning it down. They truly are The Gang That Couldn’t Shoot Straight.

I don’t actually own a crystal ball. It would have gotten broken during one of the many moves I’ve made in my life. Unfortunately, I don’t think I need one to predict that the legacy of Kelo v. City of New London rears its ugly head in New Orleans.

Kelo expanded Eminent Domain from Public Use with clear benefit for the public good to merely promising better commercial benefit. In the waterworld of New Orleans, his seems like a pretty obvious slam-dunk for the developers. After all, which brings more economic benefit, a bunch of houses filled with poor people or Gothic South-themed resort, gambling and entertainment complexes? Maybe they’ll even import some of the displaced residents to give it some “authentic local color.”

The last thing this country needs is more corporate-generated faux culture. I may not have any evidence that its coming, but I’m getting a bad feeling that it’s what New Orleans has to look forward to in its post-Katrina existence.

The media has reported over and over and over that people stranded in New Orleans have been shooting at helicopters (1,530 hits for that phrase over at google).

More likely, I suspect that people were firing in the air to draw attention to themselves. There are a couple of problems with this technique in practice, but it’s the only rational explanation I can come up, assuming that people on the ground were not, in fact, firing at helicopters.

I was taught long ago and far away that if you’re ever lost in the woods, one of the ways to draw attention to yourself is to fire three shots in the air, assuming you have firearms handy.

Now the average citizen of New Orleans might or might not have done much camping, but they probably had seen it done on TV and in the movies, where firing in the air is good for everything from signalling rescuers to breaking up a bar fight. And since nothing else was working, they probably figured it was as good as anything else they might try.

Unfortunately, if you’ve ever flown in a military helicopter, the key word to remember is loud. I’m talking louder than a Rush concert. It’s so loud, if you’re not wearing a helmet and mic, you have to cup your hands over a person’s ear and scream. And even then, you’ll probably have to repeat yourself.

From the ground, what you see are people firing guns into the air. Where the helicopters are. Ergo, they must be shooting at the helicopters.

Whether you learn firearms safety from the NRA or the military, rule #2 (after “always treat a weapon as if it’s loaded”) is that you only point a weapon at things you intend to shoot.

From inside the helicopter, you look down and see people waving and/or firing guns. You know that it can’t be to get your attention, because it’s way too loud inside the helicopter for that. You assume that they follow the same gun safety rules as you do. Ergo, they must be shooting at you.

Throw in some low expectations of the folks on the ground at it all seems to add up. But that doesn’t make it so.

TechDirt put me onto this USA Today story about why The Gap, Old Navy, and Banana Republic’s Web sites have been down of late

Already mired in a sales slump, Gap Inc. has closed its two most popular Internet stores so the clothing retailer can upgrade its online operations before the pivotal holiday shopping season.

Both Gap.com and Oldnavy.com have been closed for the past week, driving frustrated shoppers like Kira Storch of San Francisco to other Web sites to buy clothes.

…

Banana Republic, another chain owned by Gap, also closed its site two days last week before reopening Aug. 26.

Wotta buncha amateurs. If they don’t know enough about IT Operations or Ecommerce to realize what a huge, unnecessary mistake that was, I’d be terrified to trust them with my Personally Identifying Information or credit card number.

I know a number of ways to avoid an outage when replacing an ecommerce site. When I did this in the past, we simply replaced all of the old hardware, which paid for itself in reduced support and maintenance costs since we were able to cut the number of Web servers from over 100 to about 50 simply because the new gear was so much faster. As a result, the actual “dark” time when the site was a placeholder saying, “We’re upgrading! Check back in a little while,” was only a couple of hours from midnight until 2am while the necessary database changes were made.

I’ve also suffered the curse of not being able to afford the new hardware yet being tasked with complete upgrades. We were able to build up the new infrastructure in alternate directory trees, then perform the cutover by changing a symlink from the old to the new directories and re-loading Apache in a rolling reset. The downtime was only a few minutes while we made database changes, and even those had been scripted ahead of time, so we only needed enough time to dump the database, run the scripts, and rebuild some indexes. A little more effort, but no more money was required to do it this way. Worst case scenario was to dump the post-cutover transactions, fallback to the database backup, re-point the symlinks, and re-create the transactions.

So what does it say about their IT environment that they couldn’t or wouldn’t spend the money and lacked either the basic IT Resources or skills to avoid going dark for days? To me, it says they probably aren’t spending as much as they need to be on IT in general and Security in particular. Meeting the CISP requirements takes a lot of know-how and hard work. I guess they’re willing to accept the risk that Visa will find out and give them either a big fine or the Death Penalty.

Finally, since the topic is fresh on everyone’s mind this week, what does this say about Gap’s Disaster Recovery planning? To me, it hints quite strongly that they might not have much of one. Rather than going dark, they could have activated their backup site, upgraded the new site, then cut back over.

Maybe their Web site doesn’t produce enough revenue to justify a Hot Site, but I find that a little hard-to-believe, too. According to a friend of my wife who lives near a Gap fulfillment center, the usually-bustling center was “dead” during the outage, so they’re moving a fair amount of inventory through it.

Feel free to prove me wrong in comments, but I’ve never personally heard of a situation where it’s necessary to go dark for days on end for a planned change.

Before-and-after satellite imagery of New Orleans from globalsecurity.org. I have nothing more to add.

The most fundamental role of Government in society is to protect the well-being of its citizenry. The current Presidential administration has completely failed those most in need in this regard. That I cannot excuse in this disaster.

So here’s how it all breaks down for me:
1) NOLA was literally a disaster waiting to happen.
2) The current President chose to cut funding for the maintenance and improvement of countermeasures at a time when the threat (Hurricanes) is at an all-time high. This is akin to lowering the castle’s drawbridge with an army marching on the city.
3) DHS & FEMA, the two organizations charged with preventing and responding to disasters, are being run by inexperienced, incompetent political hacks. Michael Brown knew nothing about Disaster Management prior to taking over FEMA, nor does he seem to have learned anything during his tenure. FEMA’s budget has been gutted in the name of the War On Terror. These are all decisions made by the current administration and cannot be blamed on anyone else.
4) People have now DIED because the Federal Government chose not to respond in a timely manner, despite their mission being,

In the event of a…natural disaster or other large-scale emergency, the Department of Homeland Security will assume primary responsibility on March 1st for ensuring that emergency response professionals are prepared for any situation. This will entail providing a coordinated, comprehensive federal response to any large-scale crisis and mounting a swift and effective recovery effort.

Some of what has now happened in New Orleans may have been unavoidable, but the disaster as it has played out was *absolutely not necessary*. The utter failure of the Bush administration to perform any sort of Risk Management combined with willful ignorance with regards to disaster preparedness was utterly negligent.

So let me break this down from a Risk Management perspective. For anyone not familiar with the concepts, I’ll provide a very brief overview, then see how they apply in this situation.
(more…)