Disasters are inevitable and no amount of planning will account for every possible scenario, and that’s why improvisation skills are so valuable. Being able to improvise comes from experience and knowing not to panic. We can’t help you with the latter, and there is no true replacement for experience, but sometimes just being exposed to a few good ideas is enough to get a head start for when things do go awry.
So whether you’re responding to a disaster, fixing a security breach, bracing for a change in leadership, adapting to the introduction of a new technology or squeezing a penny for all it’s worth, we present to you the next best thing to experience. This is your CIO survival guide, based on the real stories of people who have been there, gotten their hands dirty and survived to share what they know.
There’s a security breach.
Quick — what do you do? Notify the press? Call your customers and apologize? Hide under your desk until it’s all over? Remember all that stuff earlier about how planning is useless? Forget all that. When it comes to cybersecurity, you need a plan, said Dan Lohrmann, Michigan’s former chief security officer and current chief strategist and chief security officer at Security Mentor.STEP 1: Have a plan.
If you don’t have a cybersecurity incident response plan in place when an attack happens, it multiplies the work to be done and makes an already stressful situation even more confusing, Lohrmann said. And he should know. Lohrmann weathered a few security incidents during his years with Michigan and the National Security Agency. Having a plan was always key to recovering cleanly.
You may need to contract with a forensics team at some point to figure out what happened, and creating that relationship before an incident occurs can ease the burden, he added.
Plan templates are easily found online, including Michigan’s. Each incident is different, so plans aren’t comprehensive, but they lay the ground rules and protocols to ensure recovery can go as smoothly as possible.
STEP 2: Get the facts straight.
The 2013 Target breach is a high-profile example of an organization that mismanaged its security incident. The number of exposed records grew larger with each announcement the company made, and just as in politics, it wasn’t so much what the company did that upset people, but how it handled the aftermath.
Not every security incident is a breach, and not all breaches are equally severe. Before your organization can appropriately respond to an incident, it needs to be clear what’s being responded to. It might take a few days before all the facts come in, and to get those facts, your organization may need outside help.
STEP 3: Call a friend.
Friends will come in several forms, Lohrmann said. Lean on your vendors, professional organizations and more experienced governments for guidance. “You are not going to be able to do this alone,” he said. “Think of yourself as the conductor in an orchestra. This is not a one-man team, Superman’s-going-to-take-on-Chicago type of thing.”
[click_to_tweet]"This is not a one-man team, Superman’s-going-to-take-on-Chicago type of thing."[/click_to_tweet]
Organizations like the Center for Internet Security and U.S. Computer Emergency Readiness Team have checklists, tools and programs to help organizations manage incidents.
For legal reasons, vendors may sometimes disappear when you need them most following an incident, said Lohrmann, which is why it’s a good idea to get the most out of them before there’s an issue. Like professional organizations, vendors provide many free tools, and depending on your relationship, they may be willing to help your organization prepare for a breach for free.
STEP 4: Don’t forget to communicate.
When there’s a problem, people tend to start pointing fingers, Lohrmann said, and besides not being very productive, right after a breach is not the time to place blame. One of the reasons for creating an incident response plan is to establish roles that make it clear who’s responsible for what, which serves partially to sidestep the blame game. Blaming the person responsible for a car wreck won’t unshatter the glass or help the victims who need medical attention. Communicate constructively; you can place blame later.
A disaster hit.
A cybersecurity breach can have huge legal and financial consequences, but there’s nothing that quite matches good old-fashioned death and destruction. Whether man-made or natural, disasters have a way of gripping everyone’s attention in a way that can make the problem worse. Oregon CIO Alex Pettit knows how to handle a disaster because he’s been through a few, including flood, hurricane, terrorism and tornado. The first step in a disaster situation, Pettit said, is to give your face a splash of cold water.STEP 1: Move on.
The Sept. 11 attacks elicited the typical human reaction to disaster, which is to freeze up and watch the scene slowly unfold from a safe distance. That’s a fine reaction for private citizens, but public servants have a job to do. The first step is to stop being a spectator, stop thinking about yourself and start thinking about what needs to happen next.
Everyone wants to talk about where they were when Kennedy was shot, Pettit said. “Well, that’s great, but it’s not about you,” he said. “It’s about Kennedy.”
STEP 2: Understand your situation.
Before you can solve the problem, you need to identify it. Sometimes, it might not even be clear there is a problem. On Sept. 11, 2001, Pettit was the chief technology officer of Denton, Texas, where he and his team stood 1,375 miles from the smoldering towers, mesmerized by the images on television. Despite the distance, Denton had a problem too.
Once city officials stopped staring at the TV, they realized they were broke. The city’s payroll funds were locked up in interest-bearing bonds that the attacks had made inaccessible. Recognizing that problem early allowed the city to make other arrangements and ensure employees still got paid, said Pettit.
STEP 3: Listen for requirements.
The third and most important role of IT during a disaster is to keep both the mind and ears open, a job easier said than done, Pettit explained.
In May 2013, a series of tornadoes hit central Oklahoma, grinding miles of civilization flat. When it came time for department heads to “report out,” as Pettit who was then the state CIO calls it, he didn’t immediately realize that the things he was hearing from his compatriots were cues for him to do his job.
One cabinet secretary mentioned that they had more bottled water than they could handle, but not enough hand sanitizer. Pettit called his staff and told them to set up an online registry, which was used to advertise to donors the goods the state needed most.
Another secretary said he needed pet crates to handle all the stray animals that were running around, but instead of finding crates, Pettit set up a Pinterest account and called it Pet Connect. They returned more than 500 pets to their owners that way.
The first secretary didn’t want hand sanitizer — he wanted an efficient system for getting needed supplies. The second secretary didn’t want animal crates — he just wanted pets to be reunited with their owners.
“When you’re supporting the first responders, a lot of times what they’re articulating that they want isn’t really what they want,” Pettit said. “What they’re really trying to do is some higher-level outcome, but they’re telling you how they think they need to achieve that outcome, and it’s very hard to articulate tools that they don’t even know exist or could exist.”
Emergency situations have a way of jolting people out of their normal modes of thinking, but being in the middle of a disaster doesn’t change what technology people are supposed to do, which is make tools. If someone asks for fish, build a fishing pole, a net or a device that lures fish straight into the oven. It’s the same job as usual: Listen for requirements, run it through the IT translator in your head and mobilize a solution.
A new technology is changing how everyone around you does business.
The smartphone is changing how most people live and work, and more disruptive technologies will follow. The first thing organizations should do, said Washington state CIO Michael Cockrill, is rip the Band-Aid clean off.STEP 1: Embrace change.
It’s going to happen sooner or later, so it’s best to just accept it, Cockrill said. Some organizations are trying to avoid allowing their employees to use their own mobile devices or to work remotely, but technology can embolden operations so utterly, and younger generations are so demanding of such arrangements, that it’s not practical to fight it. The gains to be had both in recruiting and productivity are simply too great to be overlooked, Cockrill said.
“If you’re asking a new college graduate to give up their smartphone, to be tethered to their desk, to only work between 9 and 5, you’re in danger of creating a bridge too far,” he said.
Preventing employees from working as the rest of the world works also creates a pernicious cultural rift not unlike what Plato describes in his Allegory of the Cave. Public-sector workers cannot fairly be expected to understand and serve citizens who inhabit a vibrant world so different from the shadowy cave wall of government.
STEP 2: Focus on the data, not the device.
There are ethics issues at the intersection of work and personal life, but if it weren’t for the security risks, most government offices would have adopted a bring-your-own-device policy by now. The best philosophy for treating the BYOD security issue is to forget about the phone and worry about protecting the data, Cockrill said.
Yesterday it was a computer, today it’s a smartphone and tomorrow it will be something that worries the security people even more. But it doesn’t matter much what the device is, because no one is trying to steal the device. They’re after the data, and so that’s where the focus should be.
STEP 3: Assume infection.
When it comes to cybersecurity for new technologies, many go wrong by concentrating too much effort in one place. Imagine you’re designing a facility to protect the Hope Diamond. You could put your entire budget into building the strongest, most impenetrable walls imaginable, but if someone thinks of a way around those, then there aren’t any guards or laser beams to stop the thief from pocketing the jewel and moonwalking out.
It would be irresponsible to not build any walls at all, or to not at least make an effort at building strong walls, but it’s more realistic to assume that at least a few people will get through the first line of defense. If you assume your network’s end points are infected, then it shouldn’t matter what form they take — smartphone, computer, Google Glass or otherwise — and your overarching security philosophy won’t inhibit your adoption of new technology.
Protecting a network is a lot like dating. Sure, there are some nasty viruses about, but you protect yourself the best you can and get out there to reap the rewards. Staying inside may be the safest approach, but you won’t meet your soulmate sitting on the couch.
A new governor is elected.
The CIO position is typically short lived to begin with, so when a new governor is elected, appointed CIOs usually start looking for a new gig. The following steps probably won’t save your job, but it’s worth a try. And it’s good general advice anyway that’s been tested by someone with experience both in politics and the CIO’s office.Virginia CIO Sam Nixon served the state legislature for 16 years, and last year he kept his job as CIO despite not only a change in governor, but a governor of a different political party too. Although Nixon recently decided to move to a new position with the Virginia State Corporation Commission, his experience shows it’s possible for CIOs to successfully transition to a new administration.
“In my mind, the key thing is that you cannot ignore the politics or the organizational dynamics associated with being a state CIO,” Nixon said. “There’s the fact you’re dealing with state networks that contain citizen data, and if we have something go wrong, whatever went wrong can and will be on the front page.”
STEP 1: Establish your worth.
Being a quiet champion is a noble role that history may or may not remember, but it’s also a role that people definitely won’t notice or appreciate today. A majority of CIOs are more IT manager than politician, but touting your achievements is an important part of the job, Nixon said.
In Nixon’s case, he carefully “but appropriately” took inventory of his accomplishments and ensured the decision-makers around him understood precisely why he was worth keeping. Proving oneself as an asset isn’t necessarily enough to keep your job, but it is the primary requisite.
STEP 2: Explain yourself.
Another reason the incoming governor didn’t seek a new CIO is because Nixon had a reputation for being fair and trustworthy.
“It’s all about being transparent and accountable and forthright,” he said. “And even when you have to deliver bad news, it’s important to deliver it in a deliberate and respectful way that people, even if they disagree with your decisions, can at least understand there was some rationale behind what you were trying to do and you weren’t just being arbitrary.”
Building “trust relationships” in this manner was a key component to keeping his job, Nixon said.
STEP 3: Make contacts.
Having been in state politics and IT his entire career, Nixon knows a lot of people on both sides of the political aisle. Having a good reputation alone is not enough for most governors to keep someone on board — they want direct assurance that the CIO they appoint will work well in their machine. When Nixon learned a new governor was incoming, he reached out to a few contacts close to the governor-elect so he could size up the situation and figure out what to do next.
STEP 4: Be proactive.
After learning the governor did not already have a new CIO lined up, Nixon requested to meet with the transition team. He shared information he thought the team needed to know, regardless of if he was to stay or not. He briefed the team on the organization’s technology, how it operated and the cybersecurity stance.
The meeting gave the team an opportunity to informally interview Nixon without the pressure of a real interview. Once Nixon saw that the door was still open for him to keep his job, that meeting facilitated an opportunity for him to walk through.
Your budget is about to get cut.
Take it from a banker: There are right and wrong ways to handle the hardship of a smaller budget. Joe Marcella worked in banking for nearly 30 years, and he’s been the CIO of Las Vegas for the past 18 years. The long-surviving CIO said he likes working in government because he can never get it quite right — it’s always challenging.So when the recession switched into full gear in 2008, and in Las Vegas especially, Marcella must have felt giddy because there are few things more challenging than being asked to maintain what you do but with way less funding. Not only did Marcella maintain services in his city, he may have come out the other side in better shape than anyone.
STEP 1: Focus on infrastructure.
Marcella said he recognized in 2006 that change was coming and began planning accordingly. He began from a simple recognition that infrastructure was his most important investment, both from strategic and practical standpoints.
Keeping IT infrastructure updated gave the city a platform to continue delivering crucial services as needed. At the same time, Marcella reduced the depth and number of services he provided, using performance-based budgeting and careful analysis of old assumptions to target the cuts. Changes like that became common throughout the city after the downturn.
Maintaining infrastructure while cutting back on nonessential services also positioned Marcella’s office for a quick comeback. Marcella said he wanted to hit the ground running as soon as the economy began to recover, and he knew that multiyear infrastructure rebuilding projects would have made that impossible. Instead of heading to City Council hat in hand to beg for millions of dollars to rebuild aging infrastructure, Marcella was instead able to showcase a well oiled platform that was ready to support new services.
Strategically the approach works in two ways. The first is that legislators could view the investments he was asking for as an opportunity to provide services to their constituents, rather than as a risk with a delayed payoff. Piling onto an existing infrastructure investment also makes sense from an efficiency standpoint, because infrastructure alone does nothing. If the infrastructure is already there, you may as well find ways to use it.
“Many of my colleagues across the United States are still standing in front of their councils or commissions or boards, getting security infrastructure, systems, software, because of application life cycle, approved,” Marcella said. “They’re still in a situation where they’re recovering their infrastructure when I’m not even concerned with that.”
STEP 2: Find opportunities in the cut.
Marcella also used the recession to do things he normally wouldn’t have been able to do, like restaff. Las Vegas’ IT organization is about 75 percent unionized, which made it difficult for Marcella to replace employees with new ones with the skills he needed, he said. The city’s shrunken budget gave Marcella license to reduce the IT department from 117 employees to 70. When he was able to restaff, he could do so according to the new services provisioned by the performance-based budgeting process.
STEP 3: Understand the change is temporary.
Marcella was able to avoid the Catch-22 of infrastructure investment because he did two things: He saw trouble ahead and knew it would be temporary. He admitted that running a centralized enterprise “saved their bacon” to an extent, but ultimately it was strategic planning and foresight that allowed Las Vegas to evade many of the struggles that continue to plague municipalities today. If you expect times of feast and famine, and play the long game, as Marcella did, overall performance will average high.
