Downtown streets quickly filled with people scurrying around wondering what was happening and how to get home. Getting out of the parking garage became a 30-minute challenge in accident avoidance. Several commuters volunteered to direct traffic on busy street corners.
Cell phones either didn't work or were constantly busy. The real shock came when I heard a radio announcement that the entire northeastern United States, including New York City, was experiencing a blackout. No one knew the cause. My mind was flooded -- was this another 9/11? Could this be a terrorist attack? What was going on?
Assessing the Situation
As Emergency Management Coordinator for the Department of Information Technology (DIT), I reported to the State Emergency Operations Center (SEOC), which was on generator power. As I walked into the underground facility, I thanked God we ran three emergency exercises in the past five months to prepare for moments like these. On arrival, I learned the full scope of the outage in Michigan and surrounding states. My job was to coordinate actions with other departments and ensure the DIT provided computer and communication assistance needed during the emergency.
I immediately contacted our DIT emergency coordination center, which was activated at our backup data center location -- also running on generator power. Many of our technical staff and emergency contacts were on vacation, but after working through wrong and unanswered phone numbers, representatives from every section of the DIT were connected into our phone bridge. This line was buzzing with activity for the next five days.
An executive update was given by each organization every few hours. The governor walked around the room to hear each report and ask follow-up questions. I was impressed by her focus and hands-on approach to the crisis. I had at least half a dozen conversations with her during the blackout. During one of the briefings, I was surprised and encouraged when President Bush called the governor to promise federal support.
Over the next several days, our Public Service Commission (PSC) representative gave regular reports about the power outage's expected length in different areas. Maps on the walls showed which areas were still without power and which were still in a state of emergency. A pattern developed in which power was restored quicker than estimated, but in some cases the power was unstable and failed shortly after it was restored, hampering our computer restoration efforts.
The biggest issue was water. Many organizations, including the National Guard and the Red Cross, helped get water to southwest Michigan. Private companies donated water and others volunteered to truck it from one part of the state to another. Reports were also given by the Department of Community Health on hospital coverage and other health-related issues. The Department of Agriculture was active in resolving food spoilage issues and restaurant food safety.
Establishing Order
On Thursday night, the DIT faced numerous challenges and questions. Reports came from all over the state about whether services were up and running. Some computer servers went down when their Uninterruptible Power Supply (UPS) failed. The Executive Office wanted to update the state Web portal "Michigan.gov" with regular messages from the governor and the Public Service Commission, but our connectivity was down.
We worked much of Thursday night to get things working again.
Over the weekend, the DIT was involved with workarounds to get unemployment extension letters out, update benefit card credits (formerly food stamp allocations), and assist in coordination for many other business processes. At one point, the Department of Community Health couldn't get an urgent e-mail to the Center for Disease Control in Colorado. They thought their emergency center's generator would enable them to keep all computer services going in emergencies like these, only to find their e-mail server was in a different building without power. Situations like these continued to arise through the following Monday.
Power was expected back in Lansing around 4 a.m. Friday. Should state employees report? Since cooling for state buildings in Lansing was provided by the utilities via chilled water, not air conditioning, would the computer rooms have enough cooling to bring up servers in time? Finally the decision was made to have Lansing employees report, even if computer networks were unavailable. Through the dedicated efforts of employees, most computer services were available by 9 a.m. Friday morning in Lansing.
In most of Detroit, power was unavailable until Saturday morning. Work continued through Monday morning as the DIT went through the same processes in Detroit that we followed in Lansing the previous Friday.
Looking Back
So what lessons did we learn? On the positive side, our previous exercises helped us. Having a common incident tracking system at the SEOC and DIT emergency coordination center was priceless. It allowed everyone at our department command center to see all the actions, alerts, logs and issues available at the SEOC. We could share common event logs and track actions taken across the enterprise.
On the negative side, we learned behaviors change when real emergencies occur. Most staff went home to check on their families before coming back work. What if power had been off in all of Michigan? Would everyone have reported as quickly? We were amazed at how an extended loss of power affected so many other areas. What if the outage would have been a week or longer? As a result, we updated several parts of our emergency plan activation procedure.
Looking back at the blackout of 2003, I realize again how vulnerable we are to emergency situations. From hurricanes to power outages to terrorists acts, we can prepare for emergency situations, but we can't control events.
For Michigan's DIT, the power outage enabled us to gain a more positive "can do" reputation with our customers. Our relatively new department has now lived through an emergency with the agencies we serve, which helped build trust. Not only did the blackout strengthen the new relationship with our client agencies, it showed the tremendous accomplishments we can achieve through teamwork in our own agency.
Despite a hectic workload and an intense atmosphere, I stepped back a few times to watch how things were running in the SEOC. I was amazed by the calm dedication and lack of panic. I was proud of the response everyone provided, and especially that our prior planning paid off. As a department, I'm proud of the excellence and teamwork we showed.
