Hard-Won Experience: Lessons from America’s Biggest Disasters and Emergencies

Katrina. Sandy. The Boston Marathon bombing. 9/11.

Some emergencies are indelible on the national consciousness to the point that they need no introduction. These tragedies are looked back upon as touchstones in America’s quest to be as prepared as possible for the next disaster.

Perfection is an impossible goal, but the public officials who’ve been at the center of response and recovery efforts can’t help but reminisce on what went wrong and right, what works and what doesn’t, and how their experiences might be useful for others forced to live through a similar event.

The people who manage the public’s technology and computer systems often are unheralded. But in an increasingly interconnected and online world, the lessons they’ve learned will only loom larger when — not if — the next large-scale disaster strikes.

Government Technology talked with officials who’ve worked amid some of America’s biggest disasters and emergencies of recent years. They offer practical, real-world perspective that we all can learn from.

Rafael Mena, Orange County, Fla.

A day before the official start of this year’s Atlantic hurricane season, Rafael Mena was invited to the White House. On May 30, shortly after embattled Veterans Affairs Secretary Eric Shinseki resigned, Mena sat across a table from President Obama.

Mena and two others, a Virginia official and the U.S. Energy secretary, were there to discuss innovative approaches to disaster preparedness. As the CIO of Orange County, Fla., Mena has had more experience responding to hurricanes than many. In the decade since 2004, a year when Charley and two other hurricanes made landfall in the Orlando area, Mena has led several technology projects to prepare the county for the next megastorm.

In 2011, Orange County introduced an app called OCFL Alert. Developed in-house by the county, the Web and phone app gives people all the vital real-time data they need before and after a hurricane: where sandbags are being delivered, a map of shelters, the location of drinking water, and on and on. Thousands have come to depend on the app for the latest information.

“What makes our applications unique is they are tied to a back end — to case management systems. When you submit something to us, in real time, that information creates a work order or a ticket incident. It’s delivered to the proper department so they can start working on the issue,” Mena said. The user gets real-time updates. “It’s very interactive and it gives the citizen information and it keeps us accountable; it’s transparent.”

An encore is coming. Mena and his staff are working on another app that will help deploy volunteers to big groups like the American Red Cross or the small food bank around the corner. Volunteerism can become chaotic if it isn’t managed well.

Lessons Learned Don’t underestimate the importance of GIS. Apps with two-way communication are effective. During a crisis, get everyone in the same room if you can.

Public-facing apps are indispensable, but Mena said just as much time and attention must be given to the computer systems that are out of the limelight. A point of emphasis in Mena’s business continuity plan is ensuring the county’s GIS system stays up and running. Basic map layers for electricity, lakes, schools and essential buildings help officials see what’s happening even if they’re confined to an emergency operations center. “A picture tells a story well and allows for better decision-making. GIS is very important,” he said.

Another lesson learned that Mena suggests sounds almost too simple: Have the key decision-makers sit next to one another during a crisis. When an emergency strikes, get everyone — literally everyone possible — in the same room. Orlando’s sprawling hospitality industry, Disney and other hoteliers, are major players inside Orange County’s EOC because any type of evacuation in Florida could bring an influx of travelers who need lodging.

These are the kinds of lessons Mena shared with the president last spring. The commander in chief was engaged during the talk, Mena said.

“It was a great experience and he asked very good questions. It was very, very cool and I enjoyed it,” Mena said.

justin-holmes.jpg

Justin Holmes, Boston

When two perpetrators set off improvised bombs on Patriots’ Day the afternoon of April 15, 2013, near the finish line of the Boston Marathon, Justin Holmes and his team immediately jumped into action to alert citizens and keep them apprised of the latest developments.

It wasn’t a matter of creating a bunch of new social media channels and websites. For the most part, the city simply kept doing what it had been doing for years, only more of it. For the past four years Holmes has overseen Boston’s 24-hour phone hotline along with a number of mobile and website applications.

“The wrong time to stand up a social media presence is in the middle of a disaster,” Holmes said. It’s most effective when you have an audience that is already familiar with your brand and leveraging that during an emergency. “Having sustained, proactive and daily engagement with residents helps you in a time of emergency.”

Three were killed and 264 hurt in the marathon bombing. Residents were on edge and looking for any information on friends and family who might be potential victims. People also were sending in tips as law enforcement hunted the two suspects. Thousands of calls simultaneously flowed into Boston’s call center, and the volume remained unusually high for the following two weeks. The city leaned heavily on its scalable telephony to handle the surge of volume.

Lessons Learned Before an emergency, ensure that technology is scalable so you aren’t caught off guard. Ask for help from the private sector; companies probably will respond quickly. Be personal, not robotic, when communicating with the public.

Some of the Web tools the city had weren’t as ready for an emergency as Holmes and other Boston officials would’ve liked, and that problem has been fixed in the bombing’s aftermath. “Our website actually went down for a brief period during the marathon bombing because people were visiting cityofboston.gov in such higher numbers,” Holmes said. The importance of having scalable solutions on hand was a lesson learned.

Having the right technology in place was important, Holmes said, but just as valuable, if not more so, was establishing trusted relationships ahead of time. “The right people who are willing to work together on whatever challenge arises is infinitely more important than anything else,” he said. That also extends to partnerships with vendors that are willing to step in at a moment’s notice. Esri helped Boston develop a tool to manage snowstorm operations, and Twitter helped raise more than $5 million among 500,000 donors during a six-hour sponsored tweet campaign for One Fund Boston, which was formed to help people affected by the bombings.

Establishing the public’s trust online and on social media also has been critical to success. Holmes’ advice is to adopt a personal, human tone when communicating — every day and during crises: “Too often when we try to solve problems of scale, getting thousands of phone calls at once, the technologist might say to just put up a recorded message. Well, during an emergency, people are so concerned that having a human voice on the end of a phone in a call center context is important. They’re looking for comfort and assurance. A computer recording can’t give them that in the same way.”

His second pointer for social media use is being mindful to manage expectations. In an emergency, if someone is worried about their power being out, Holmes said the worst thing you can tell them is, “We’re going to get it back on as soon as we can. Hang tight. We’ll have the power back on in two hours.”

Instead, be honest. You probably don’t know exactly when the problem will be fixed. So tell them it could take a while and you will keep them posted.

steve-emmanuel.jpgSteve Emanuel, New Jersey

Superstorm Sandy was a once-in-a-century weather cataclysm of unimaginable proportions when it hit the Eastern Seaboard on Oct. 29, 2012. It hit New Jersey especially hard; coastal developments were wiped away en masse and more than 100 people were dead or missing. Sandy caused an estimated $65 billion in damage nationwide.

For days in advance, New Jersey CIO Steve Emanuel pre-planned and made lists of possible needs — cell sites, email and other emergency messaging platforms, air cards and smartphones were just a few. “One of the things we talked about was voting by mail because it was right around election time,” Emanuel said.

There also were more mundane considerations, like coordinating how relief trucks from across the U.S. would get through the interstate electronic toll system when driving to New Jersey.

Even so, when Sandy came to pass, Emanuel was caught somewhat off guard when his CIO colleagues, like David Behen from Michigan, Karen Robinson of Texas and Hawaii’s Sonny Bhagowalia called him and offered assistance.

“You probably could help,” Emanuel said at the time, “but I need to think about what the hell you can do for me.”

The experience had Emanuel brainstorming that perhaps it’s time to stand up a national database or website where CIOs in need could access a list of contacts, capabilities and resources. It would’ve been useful to know which states could’ve sent New Jersey an emergency supply of spare parts, switches and server infrastructure, or a dozen satellite phones.

Even having another state’s CIO call the telecom companies on his behalf to collect statistics on outages would have been helpful, Emanuel said.

“Something as simple as that, because at one point in time we thought we had 40 to 45 percent of wireless capacity down. We didn’t talk about where or who they were; we put a global number together so the governor would know why he’s not getting information.”

Lessons Learned Be prepared for help from the outside, including colleagues from other governments. Tap into the disaster recovery plans of the private sector when offered. Pilot projects can be put to use in the real world to fill a need.

In the aftermath, Emanuel also found that the private sector — Cisco, Microsoft, Verizon and others — shared valuable action plans that New Jersey would use to further improve its disaster recovery and business continuity. Still, thanks to the state’s preparations, he said New Jersey did well to avoid a major system outage and personnel stayed overnight in the state’s data center to ensure uptime.

“The bottom line is there are so many things that surround us that we react to that you don’t think about until that ‘gray sky’ day,” he said.

Emanuel also said pilot projects that were in progress played a role in New Jersey’s response efforts. For instance, the state had recently turned off a social media pilot that was aggregating data on specific discussion topics. New Jersey turned that back on during Sandy.

“Eventually that became one of the conduits for pleas for help from people. After the second week, they started dispatching responders based on social media,” Emanuel said.

hugh-miller.JPGHugh Miller, San Antonio

It’s easy to forget that Hurricane Katrina was a truly national-scale disaster. The Category 5 hurricane ravaged New Orleans, but the long-lasting effects extended well beyond the French Quarter into neighboring states.

San Antonio became a destination of last resort for an estimated 30,000 Katrina refugees, and the city quickly realized that an influx would be coming in search of shelter. San Antonio CIO Hugh Miller and a few of his staff members were put in charge of technology at an old Air Force building that would become a processing center for thousands left homeless from Katrina. Some came only wearing the clothes on their back.

It was a hasty operation by necessity, with routers, switches and wires strewn about between cubicles and rows of cots. The first phones were individual outbound wires before Miller’s staff later installed an IP PBX system to organize call routing.

The United Way, FEMA and other organizations got involved, and for a time everyone was doing their own thing. Technicians from the big telecom companies would show up and begin installing equipment without oversight. There were almost too many volunteers. Chaos ensued.

“Early on we realized someone needed to be in charge of these different types of work so that you didn’t have people stepping on top of each other,” said Miller. The city had to aggressively assert authority over IT matters.

Slowly the project settled in and Miller and city staff turned to the day-to-day work of installing big screen TVs, organizing food and a multitude of other functions. They even created databases of the displaced residents and disseminated the information nationwide because the evacuees’ loved ones didn’t know where they were.

“One of the big things that was prevalent was that there was truly not the level of collaboration that was needed,” Miller said. “Granted this was back in 2005, so there were a lot of things maturing — but there was not a clean, integrated way to process people.”

Lessons Learned Brace for some chaos; it might be inevitable. Establish authority and leadership early on. Know who your vendors are before the emergency.

The vendor community also stepped up. Dell is in Austin and AT&T has a huge footprint in San Antonio, and the presence of those companies was one reason the federal government chose San Antonio as an evacuation point, Miller said. He advised to know who your vendors are and have those key contacts in advance. San Antonio was fortunate to have that list already in place, Miller said, in part because of the high volume of regular purchases necessary for the nation’s seventh-largest city. Consequently, vendors reached out to offer assistance and the city already had extra computers and phones on hand because it was on a scheduled IT refresh cycle.

“We had a regular batch of systems that were ready, so we pulled from that stock and deployed it instantly,” Miller said.  

Photo of Justin Holmes via YouTube; photo of CIO Steve Emanuel by David Kidd; photo of CIO Hugh Miller by Steve Towns.