As media outlets and the public at large pound on the digital front door for the latest poll numbers, results portals across the country face the strain of hundreds of thousands — if not millions — of hits. Some falter and are overwhelmed by the attention and come crashing down; others come to the game prepared, having learned from past follies.
Though 2014 wasn’t exactly what you’d call a big-ticket election — with no presidential candidates on the ballot — states across the country experienced issues with their election reporting websites. Whether the problems were due to overwhelmingly high Web traffic or just technical difficulties, several states had to step back and rethink their online reporting strategies.
With 2016 expected to be a veritable title fight between the headline grabbers like Hillary Clinton and Donald Trump, Government Technology caught up with several states who have seen issues in the past and moved to confront them head on.
In Virginia, problems with the state’s election reporting website started in 2008, when online traffic to the site caused an outage.
“In 2008, Virginia became a hotly contested state for the first time in a presidential election in decades, and our election night reporting site saw an unprecedented amount of traffic," said Virginia Department of Elections CIO Matthew Davis. "We actually had a firewall failure because the firewall sitting outside of our network couldn’t handle the traffic coming in to our site from across the nation.”
Then in 2014, a hotly contested state election brought enough traffic to cause a website failure officials likened to a denial of service (DoS) attack. "Our site just gave up within the first couple of minutes after 7 p.m. that night," he said. "It just could not handle the influx of traffic.”
According to Davis, Virginia’s election reporting system saw a jump from the normal 39,000 election night visits to more than 300,000 visits, resulting in 25 million page views within the website.
These ongoing service problems eventually led the state to switch service vendors from a Microsoft Azure environment to a cloud-based Akamai solution. Under the new system, Davis said site visitors are directed to 2,000 servers worldwide.
“We decided to drop back and punt again with a different solution," he said. "We launched that solution and it worked well.... Because we have elections almost every month in Virginia, we have lots of chances to test things."
The popularity of personal connected devices is also adding to the strain on election results websites, Davis said. In the not so distant past, election parties were the go-to for up-to-the minute election night tallies. Back then, one computer or a news broadcast did all the work; today, the same room full of people are on their mobile devices individually hitting key websites.
With more people than ever before watching by-the-minute results on their mobile devices, Davis said, systems are put through their paces.
To make sure viewers have access to the current information in the event of a failure, he said a static HTML website is refreshed and published across the system every 10 minutes. While a static site does not offer many of the more interactive features, Davis said the simplicity keeps the systems from enduring unnecessary strain and keeps vital information visible to the public.
“So basically we have a static website replicated on over 2,000 servers around the world. It has turned out to be extremely stable just because a static site in and of itself is going to be more stable, but unlike 2008, where we had a static site, we don’t have to worry about firewalls giving out…” he said. “Everybody wants interactive sites with maps and things like that, but those take server resources..."
And when you have a spike, especially an unexpected spike, they just cause problems.
Across the country in Montana, the secretary of state’s election website also faltered early on in the 2014 onslaught because of unbalanced server loads.
Secretary of State Linda McCulloch told Government Technology the setback forced her office to reevaluate the relationship with its service provider — the state of Montana.
“Essentially the load-balancing function that distributes traffic to multiple servers was not configured correctly by our service provider. So this resulted in only one server handling all of the traffic,” she said. “As far as the traffic, it’s huge. In the 2014 general election, we saw 61,000 sessions, almost 39,000 users and 481,000 page views. In 2012, we saw about 170,000 session, 87,000 users and over 1.2 million page views.”
After sorting out the server balance issues of 2014, McCulloch said the state began looking into a new agreement with another service vendor in 2016. The secretary of state said the new system environment will provide some much needed flexibility to the state reporting.
“The new environment will allow us to handle a large volume of traffic and expand server capacity instantly as needed,” she said.
As for what to expect from 2016, McCulloch said she believes voter turnout and website traffic will rival that of the 2008 and 2012 presidential elections. Her advice to other states is simple: Load test and run through all of the possible scenarios in advance.
“I can’t stress that enough,” she said.
For Arizona, election night reporting system failures in 2012 and 2014 led the state to dump its vendor for an in-house alternative. Communications Director Matt Roberts with the Arizona Secretary of State’s office said there was never a clear answer from service vendor SOE about what caused the state systems to falter.
“We never really got a complete analysis or understanding as to what caused what we were frustrated with,” he said. “I’d love to say it was only traffic, but I don’t think it was just traffic. From what we understood from SOE, there were failures or multiple failures within the server system that created the delays.”
The reporting system's issues meant the state was not able to publish results for a more than a half-hour period on election night.
“Secretary [Michele] Reagan made the decision to develop our [Election Night Reporting] system in-house with our own IT people and staff to develop a system that will cost less…but perform better," he said. "We have more control obviously without a third-party vendor."
In addition to relying on internal talent to address any potential problems, Roberts said the new system will feature a more streamlined layout and will be backed up with a simple WordPress-like publishing portal in the event of unexpected disruptions.
“If we get to that [point], that would certainly be the worst of all scenarios, but we are planning in case something does go wrong,” he said. “If something did go wrong that required us to display those types of results on the stripped down platform, we’ll have much greater problems — because that would mean all of our systems were down and the statewide system that aggregates the results would also be down.”
The new system is still under development, but is expected to be ready ahead of the November 2016 elections. As for his advice to other states who may not have taken a hard look at their own reporting systems, Roberts said they should plan for the worst, but hope for the best.
“You sink or swim really quickly within an hour and a half on election night,” he said. “That first hour on election night is where all of your efforts and work will either pay off or you’ll be answering a lot of questions from the media and your constituents and those in the political environment.”
