The Performance Audit of Disaster Recovery and Business Continuity of IT Systems, conducted by the Michigan Office of the Auditor General, shows that while DTMB has made strides toward preserving data streams generated by state departments, it did not go as far as it should have in doing so, or in shaping policies around that effort.
In a series of responses to the audit, DTMB indicated agreement with the findings and spelled out changes that are already underway to improve data recovery plans (DRPs) and business continuity plans (BCPs) if data streams should be compromised or lost.
The Auditor General’s office told Government Technology that the examination wasn’t prompted by any single factor, incident or issue at DTMB. “We have touched on DR and BCP in individual system audits and felt it would be beneficial to take a look at, and provide recommendations related to this topic statewide so that multiple systems could be addressed at one time …,” the Auditor General’s relations officer said via email.
The Auditor General’s office said it considered the first four of eight findings in the audit, released in late December, to be “more severe,” and having the potential to impair management’s ability to effectively operate a program, or adversely affecting the judgement of an interested person concerning the program’s effectiveness.
“Findings 1 and 2 are significant because they relate to ensuring that DTMB has identified every system, service and underlying infrastructure needed to restore critical systems in the event of a disaster,” the Auditor General’s office said. "The more significant findings indicate a need for better planning, coordination, documentation and testing of the plans."
The first four findings state that the DTMB:
Did not fully plan to restore all critical infrastructure services and enterprise systems needed to restore other red card systems during a statewide IT disaster. DTMB had a partial plan for recovering the network – but the plan was actually stored on the network, auditors said, rendering it inaccessible if the network crashed.
DTMB also did not identify the state's intranet – through which many critical systems and services including Michigan’s Living Disaster Recovery Planning System are accessed – as a critical infrastructure service and fully plan for its restoration.
“As a result, the state may not be able to restore all critical infrastructure services and systems within the maximum recovery time of 24 hours for red card systems,” the auditors wrote.
The Auditor General’s office recommended DTMB fully plan to restore critical infrastructure services and enterprise systems necessary to restore other red card systems in the event of a statewide IT disaster. DTMB agreed with the recommendation and said in the audit it plans to do so, completing BCPs and DRPs for all critical infrastructure and enterprise systems.
Caleb Buhs, DTMB’s public information officer, spoke with Government Technology via email and telephone after explaining that an interview with DTMB Director and state CIO David Behen would be difficult to schedule before press time. He said via email that DTMB and state agencies do have plans in place to recover “key IT systems” and maintain business continuity if disaster were to strike.
Did not ensure the completeness and accuracy of the red card, which could keep recovery resources from going to the most critical systems and services first. Among the reasons why not, some agency and DTMB staffers disagreed about whether the red card should contain systems critical to the entire state, or all critical systems.
The Auditor General’s office recommended DTMB ensure the completeness and accuracy of the red card to make sure recovery is devoted to the most critical systems and services first. DTMB agreed, and said it will assist agencies in completing a business impact analysis and partnership agreements to identify critical applications. DTMB said it will also help agencies understand the importance of complete DRPs and BCPs through “BCP/DR 101 training.”
“Having that up-to-date is of utmost importance,” Buhs said. “This is another area where we’ll work with agencies to ensure they’re constantly reviewing the systems they have on their red card, and so we at DTMB can make sure those systems are backed up.”
DTMB and state agencies didn’t always coordinate DRP and BCP preparation; plans weren’t always created, and did not adequately address recovery of both the business process and the information system. DTMB had assigned only five staffers to coordinate DR planning, and only one staffer to coordinate BCP for roughly 1,700 IT applications.
Auditors recommended DTMB work with agencies to coordinate preparation of DRPs and BCPs to make sure the plans exist and have the information needed to be effective in a disaster.
DTMB agreed and noted that last year it began a departmentwide initiative to address internal control weaknesses in the state IT operations. To date, it has completed or substantially completed more than 80 percent of DRPs for DTMB’s 34 red card applications. DRPs for 75 percent of other departments’ red card applications have also been completed or substantially completed.
Did not implement a review process to ensure DRPs and BGCPs contained the necessary elements for effective disaster recovery, such as annual reviews, prioritization of recovery order for IT system components and an updated list of servers.
Auditors recommended DTMB implement such a review process, and the agency agreed. DTMB said in the audit it has implemented a review process, created a schedule for DRP and BCP training, and revised the training to ensure all necessary elements are documented.
DTMB began a “comprehensive review” of response and recovery plans before the audit, which has already led to significant improvements, Buhs said, including creating a central repository for DR plans, retaining virtual and hard copies of plans, and maintaining a remote data center 50 miles from the current center – an additional safeguard in the event of a regional disaster.
“We began this process a year ago in some cases. Because it’s such an ongoing and ever-changing thing, I think it’s difficult to put an end date on it,” he told Government Technology.
The department has also instituted an initiative to move more data to the cloud, and in January established a new critical ranking metric to ensure the agency always has an accurate picture of recovery requirements.
Buhs said that the need to strengthen and evolve emergency planning is always continuing as technology advances, and he characterized DTMB’s efforts to do so as “ongoing.”
The agency has been aware of DR and business continuity requirements, Buhs said, but the issue “is that they haven’t been consistently developed across all agencies.”
“That’s where a large part of the audit’s rub was, I guess,” Buhs added, noting that DTMB had begun to improve DR and scrutinize red card systems prior to the audit’s release.
After more than a decade of centralizing Michigan IT, Buhs said the state is “still cleaning up some things,” effecting culture change and upgrading legacy systems.
“We have 30-year-old systems, for whatever reason," he said. "Bringing those up to current standards is something that does take some time."
In its last four findings, auditors found that DTMB did not ensure DR servers were in place for all red card systems; that DTMB together with state agencies had not granted and maintained appropriate access to DRPs; that DTMB and state agencies did not fully utilize a central repository and backup storage location for DRPs and BCPs to ensure plans are available in a disaster; and that DTMB and state agencies did not use effective version control on DR and business continuity plans.
Auditors tested 20 of 74 state-hosted red card systems and found three of the 20, or 15 percent, did not have DR servers.
Buhs said the issue is a “high priority” for the agency, and remediation has already begun.
“A lot of it comes down to resources and funding, and we’ve made it a priority,” he said, adding that he didn’t have an exact dollar figure for what it would cost to add servers.
Appropriate access to DRPs, Buhs said, is an issue somewhat linked to the hiring and departure of employees, and is one DTMB is “constantly working on.”
State agencies generally do a good job with DR, Buhs said, but there are still gains to be made.
“There’s a long way to go to move away from paper,” Buhs said, noting that Michigan has adopted a mobile first strategy for anything online – “not just mobile friendly, but mobile first.”
As Michigan State Police brick-and-mortar offices have closed, he noted, the state has “basically made the state police cruisers mobile offices.”
“They spend more time on the road," Buhs said, "and less time sitting at a desk."
