As city planners look toward the inevitable next big blow, they’re thinking about infrastructure. What happens when emergency phone lines no longer function or when the data center goes down? To meet that challenge, Nags Head is teaming up with other municipalities to create inter-city backup arrangements.
“[If] we should have a storm and the area has to be evacuated, essential personnel generally would be required to stay here. But [if] we have a very severe storm, essential personnel would be evacuated, and this arrangement gives us a place to set up shop,” said Allen Massey, IT coordinator of Nags Head.
The arrangement he refers to involves Cary, a city of 146,000 people that’s much farther inland. For call services in particular, Cary is Nags Head’s fallback position.
Cary boasts strong municipal infrastructure, said Bill Stice, director of the city’s Technology Services Department. His office operates a 911 system, emergency radio network and two fiber-connected data centers. Cary also makes a place for Nags Head’s nonemergency call lines, in case of a disaster.
“They have a need for people to take nonemergency calls, but if there is a hurricane or major event, they don’t want to keep their staff in the local area because of the dangers,” Stice said. Nags Head paid to install six phone lines in Cary’s building, with the agreement that space would be provided in the facility for its staff to answer calls.
In 15 years, Nags Head hasn’t needed to take advantage of that offer. Still, planners in several municipalities see it as a model for something bigger: the inter-city backup of entire data centers.
Rack Room
Over the past few years, Cary has built a relationship with Jacksonville, N.C., hosting a data center rack for the purpose of disaster recovery. “We give them the space, we don’t charge them for the power or HVAC,” Stice said. “They pay the connection cost from Jacksonville to Cary, using our facility to handle a storage device and backup server.” Cary also could serve as a 911 backup center if needed.Cary is developing a similar relationship with Kernersville, N.C., located two hours west. Stice said that as surplus equipment shifts out of his data center, he’ll make room for Kernersville’s needs, presuming the city will return the favor to others that need it.
That sense of municipal reciprocity lies at the heart of many such backup arrangements, where instead of money changing hands, planners exchange mutual good will and support.
“We are basically just doing each other favors at this point,” Massey said. “Local government obviously does not have the resources that the federal government does, we don’t have those kind of budgets, and yet we all have the same concerns.”
Stice said the model has grown from the kinds of mutual-aid initiatives seen in the North Carolina Local Government Information Systems Association, whose members pledge to share 911 calling and other systems in case of an emergency. “If we all did this, all of us would have backups at essentially a lower cost than it would cost to have a third party do it,” he said. “You hope there will be a domino effect.”
In the West
While backup agreements unfold locally in North Carolina, across the U.S., whole states pursue a similar course.In Montana, CIO Ron Baldwin is confident in his recovery arrangements. His main data center in Helena is backed up by a facility in Miles City, nearly 400 miles away. But he also makes room for the neighbors, leasing out space in his data center since 2012 for Oregon to secure its backup needs.
In an emergency, “processing can be switched over to our data center more or less in real time, depending on exactly what Oregon needs and wants,” Baldwin said. “We can provide all the capabilities, from a backup site to services and equipment that can be brought up within a specified time.”
Oregon benefits primarily from state-of-the-art floor space, but Montana also has a suite of other capabilities to offer. “For example, if they need some technical hands to do something specific to their equipment, they can call us and we will physically go to that equipment,” Baldwin said.
This goes beyond the strictly favor-for-a-friend approach to reciprocity. Montana bills its neighbor for these added services, which it can do thanks to an intergovernmental services contract implemented in 2013, a tool that Baldwin said is invaluable in making this arrangement work.
Under the contract’s terms, the two states can buy from each other’s catalogs without having to jump through hoops, making purchases from a broad list of categories without needing special permissions.
“Without that, we would’ve had to put some very specific items into the data center agreement. It would have become not just an equipment lease, but an equipment and services lease,” Baldwin said. The arrangement’s streamlined nature is especially helpful given that services in the backup center may well need to be purchased on the fly, in the case of a major emergency.
“Here you have two states, one with need and one with capacity, and now both have laws that allow for this sharing,” Baldwin said. “With that they can enter into agreements that are very well priced and are harmonized between the two states.”
While Montana may not need backup services for itself, Baldwin can see such a purchasing tool as an aid to his IT operation in several arenas. “If we would want to share mainframe capacity, well, mainframes are very expensive,” he said. “If this would allow us to utilize services on one another’s mainframes, this could be a cost-effective way to do that. It isn’t something we have done yet, but we have talked about it.”
Pieces in Place
As Baldwin considers future ways to purchase IT backup services, Massey in Nags Head continues to expand his cost-free reciprocity arrangements. He’s exploring a possible data center arrangement with nearby Kill Devil Hills. Though it’s a small town, Kill Devil Hills might be able to back him up, thanks to ever-decreasing server needs.“Our physical space requirements have shrunk greatly in the last four or five years as we’ve virtualized nearly all of our servers,” Massey said. “Space requirements for servers are probably a tenth or a fifteenth of what they were.”
The technology behind these backup agreements is rather straightforward, Baldwin said. “Just like we built the interstate highway system, there is now a fiber-optic system between states and across the United States.”
More challenging, however, are the political mechanics.
“Cities are often in competition with each other: for the latest factory, the latest office complex. So there is always a challenge in trying to overcome that inherent competitiveness and go to the point of mutual sharing,” Stice said. “Our county here has been very good about creating that environment, about setting the tone for that, and it has caught on among the municipalities.”
