Virginia Outsourcing Improves Disaster Recovery, Officials Say

Eyes are peeled on the progress of Virginia's extensive outsourcing of IT to Northrop Grumman. The state recently boasted success with the partnership's first disaster-recovery test. The commonwealth managed to restore "mission-critical" IT functions within 24 hours, compared to the roughly 72 hours it took previously.

The test coordinated 135 employees from 19 of the state's more than 70 agencies. The Virginia Information Technologies Agency (VITA) selected participating agencies, such as the Department of Social Services and the Department of Taxation, based on whether they had functions needing immediate backup during a disaster.

Virginia officials said the improvement stems from the state creating its own backup data center within quick driving distance of its main facility in Chesterfield, Va.
The new facility strengthens Virginia's ability to prepare for disasters and practice recovery techniques. In previous disaster drills, IT workers transported backup tapes to a back-up data center the state rented in Pennsylvania. After arriving, those workers burned time loading numerous tapes before resuming IT operations. Every hour was critical because the state's contract allotted only 48 to 72 hours for the test, said Fred Duball, director of the Service Management Organization of VITA.

"If we ran out of 'test window' using the old vendor, we had to de-install our stuff and leave," Duball said. The state also eliminated the risk of the vendor facility already being occupied during a disaster test.

Naturally the state now use its own backup data center for as long as it wants, and firing up that facility is faster than plugging into a rented one. "We no longer have to run out to Pennsylvania with tapes," Duball said. "The transactions occur automatically across the wire."

While Virginia only needed 24 hours to restore "mission-critical" functions, it spent four days on the overall test. With no duration limit on using its new backup data center, the state had extra time to test functions it previously couldn't.

VITA also credits the faster IT-backup process to technology upgrades, which included disk replication and "virtual tape," a system that mimics a traditional tape backup by using disks instead.

VITA plans to run these tests at least once a year and employs 10 to 15 people to prepare for them.

"It's not like you plan for three months and you're done," Duball said. "Things are always changing. Business requirements are always changing. Technology is always upgraded. If I update something in my primary data center, I've got to do the same in my backup data center."

VITA and Northrop Grumman conducted the test without disrupting normal state operations, said Duball. "We did all of this behind the scenes."