The NSA collects recordings of 3 billion phone calls each day according to some reports, and is collecting most emails, voicemail messages, and social media communications. And evidently, running a data center capable of storing that much information explodes sometimes. In the past 13 months, there have been reports of 10 electrical surges that have melted metal and blown out circuit boards and cooling components.
Groups of anti-NSA protestors routinely gather on the highway near the data center, picketing and keeping an eye on the data center. They are permitted to do so because they adopted the stretch of road adjacent to the data center through the state’s adopt-a-highway program. With anti-NSA sentiment high and protestors milling around the facility, sabotage doesn't seem an unlikely explanation for the electrical explosions. But, it’s probably just bad design, IT analyst Rob Enderle said.
With a project this size, especially a government project, Enderle said, it’s to be expected that there will be major design faults, “probably because the project was underbid.” The NSA's website puts the cost of the data center at $1.5 billion, but with overruns and its as-yet unfinished status, that could change. “The costs with that have to be legendary,” Enderle said. “It’s just massive and would clearly have major budget overruns, so the amount of scrutiny and pressure to keep the cost down on that would be significant.”
The pressure on the NSA to keep the budget down is probably one of the main reasons electrical explosions persist, he said. In such a huge project, the chance for miscommunication between computer hardware people and power supply hardware people is high, combined with the fact that the work they’re doing is unprecedented, he added. You can only find a design flaw on a data center this large after you build it, he said, because it’s never been done before.
Enderle saw a similar thing happen while working for IBM when it operated one of the world’s largest data centers in San Jose, Calif. “In that case, nobody modeled what happened to the water cooling system if we catastrophically lost power and we had a power surge that took out the major transformers, which shut down the power, and caused the water cooling system to shut down, and all the mains broke, and there we were in the biggest building east of the Mississippi, up to our knees in water. Everything was destroyed. It was a nightmare.”
Everything in the IBM data center in San Jose was triple redundant, he said, so no one thought to consider the water mains. Similar unforeseen problems are probably occurring at the NSA, he said – it’s just what happens with big government projects.
Down the Rabbit Hole
Some reports suggest that the NSA’s data center operations are so secretive that the staff may build all their hardware themselves, to avoid leaks by vendors. Others say that the data center is kept pitch black for security purposes, as workers traverse the facility with lights on their helmets, like miners, a practice Google has also reportedly adopted in its data centers.Exactly what secrecy safeguards the NSA is taking are unknown to him, Enderle said, but it wouldn’t be unusual for federal government agencies to take such precautions, particularly when it comes to projects like those exposed by Snowden. Taking pictures is probably out of the question and there’s also likely a very high level of monitoring for anyone inside, he said.
“I have a hard time believing they actually had people working with power in pitch blackness because that’s a good way to get people electrocuted,” he said. But with the high level of security and secrecy around NSA data centers, just about anything could happen inside and the public probably wouldn’t find out, according to Enderle, adding that some things that happen during large government buildouts aren’t always intelligent, either.
Enderle recalled his time working with NASA for the Saturn lunar rockets, and though many equate NASA with competence and success, many have written books giving their policies and internal politics mixed reviews. In one test process, Enderle said, NASA had three parts being tested and all three parts failed. And then NASA would install one of the failed parts with the expectation that the failsafe would pick up functionality when it failed. “Why would you ever install something that was tested and failed?” he asked, blaming the government policy that emanated from someone who was ordered to keep costs down.
The new NSA facility is expected to consume a continuous 65 megawatts of electricity, enough to power a small city of 20,000 residents.
