AT&T disaster recovery awes Homeland Security

To a government employee, for whom efficiency is something one hears about but is rarely able to achieve, the efficiency which the market can provide can seem like magic: mysterious and forever out of reach.

So it was on June 7, as 24 AT&T Network Disaster Recovery trailers rolled in to the parking lot at FedExField, normally the home of the Washington Redskins, and began setting up for a disaster recovery exercise, George Foresman, undersecretary for preparedness at the Department of Homeland Security, stared slack-jawed and commented that government needs “to really begin to understand how these [communications] networks — that were like magic — work.”

AT&T was in D.C. to conduct a disaster recovery exercise and test its preparedness to restore part of its network damaged by a natural or man-made disaster. The company conducts three or four such exercises each year to maintain and improve readiness. It even has its own hazmat team. And this exercise was a bit of a show for its government customers.

“We are about to go through what I like to call the preparedness revolution,” Foresman told Government Executive.

And none too soon. The government is woefully unprepared and poorly coordinated for the next disaster. Invariably it will be ordinary people, and companies like AT&T and Wal-Mart, who pick up the slack.

On September 11, 2001, when its lower Manhattan central office was destroyed in the World Trade Center terrorist attacks, and the company had to locate its two dozen trailers across the river in New Jersey, the company was still able to get the first lines into lower Manhattan within two days.

AT&T also responded to Hurricane Katrina with emergency vans and had its disaster trailers in reserve at the NASA Stennis Space Center. The company was able to provide emergency communications to law enforcement and relief workers across southeastern Louisiana and southern Mississippi during the first weeks after the hurricane.

AT&T’s Network Disaster Recovery program, in operation since 1990, has been called into service 21 times. The company says its goal is to respond to any disaster anywhere in the country within 72 hours. Its equipment, housed in four undisclosed locations around the country, consists of over 150 self-contained trailers and support vehicles containing communications equipment, fiber optic cable, and other equipment needed to restore a destroyed central office, along with another 250 trailers containing power generation, air conditioning, food, tents, first aid equipment and other logistical support. Everything is fully redundant, with each trailer able to generate its own power or connect to a second power generator trailer capable of supplying 500 kilowatts of electricity.

With this capacity, the company could respond to multiple simultaneous disasters, and it even conducted a separate disaster recovery exercise in Atlanta last September, while still responding to Hurricane Katrina.

The Network Disaster Recovery team are all volunteers drawn from within the ranks of AT&T and go through specialized training in disaster recovery, of which the disaster recovery exercises are a part.

Members of AT&T’s disaster response teams even are equipped with biohazard suits so they can, if called upon, conduct field operations in the aftermath of a “dirty” bomb detonation or anthrax attack.

But after Hurricane Katrina, the team found it still had much to learn, and that some additional materials ‘ namely, the parts to run a complete shower ‘ would be needed for future long-term exercises.

“Down in Katrina, we got a bit gamey after a while,” Desiato said. — Government Executive

In 2003, AT&T Network Disaster Recovery also set up the satellite telephone systems that U.S. soldiers serving in Iraq use to call home.

It’s not magic. It’s the market.

One thought on “AT&T disaster recovery awes Homeland Security

  • June 14, 2006 at 7:59 pm

    They have a really great NDR team there at AT&T. Everything from Fiber to HF radios, they can connect anybody they want to.


Comments are closed.