The Sequoia voting system in Palm Beach County, harshly criticized and already old in 2007 when the county paid $5.5 million to keep it, has for years come under fire for not reliably doing one or the other — or both.
The aging system made headlines again last week, when high-speed vote counters appeared to overheat. That delayed vote counting in the nationally watched Florida recount.
Palm Beach County Supervisor of Elections Susan Bucher Thursday reiterated her belief that equipment malfunctions are at fault for a failure to finish a machine recount in four races by a state-mandated deadline. The county's equipment is so outmoded she didn't have time to even start the recount of nearly 600,000 ballots in two of the statewide races.
Dominion Voting Systems, the Nebraska-based owner of the Sequoia system, balked at her comments, saying its equipment may not be to blame.
While it's too early to pinpoint what happened and why, the machines were being run at four times the volume they would be expected to handle, company Vice President of Government Affairs Kay Stimson said.
"Palm Beach County has been running this equipment in a way that stresses it and that is highly unique," she said. "It has also been running these machines in a mode that we have never seen before."
John Brakey, an Arizona-based voting transparency advocate who has pushed for adoption of newer election systems, pointed out that "These machines usually run eight hours a day," not day and night without pause.
"Everyone did the best they could," said Brakey, "but it was an insane recount to do so much so fast."
Vote swaps and double counts
Bucher already has declared her intention of asking the secretary of state to yank the equipment's certification, which means it could no longer be sold in Florida.Not that anyone is buying: Palm Beach is the last Florida county to be using Sequoia.
Dominion sells newer systems here.
But when Dominion bought Sequoia Voting Systems in 2010, it also inherited that company's namesake product — and the criticism that has dogged it for more than a decade.
In 2008, the Sequoia system played a murky role in vote-switching in a local judge's race. It double-counted more than 10,000 votes in Indian River County. In Wellington in 2012, it shifted votes, reporting that two losing council candidates had won.
Also in 2008, it ran into trouble in New Jersey, where 37 touchscreen machines counted more Republican votes than Republican voters in some precincts, and more Democratic votes than Democratic voters in others. It stumbled in Washington, D.C., where a touchscreen version added 1,500 write-in votes.
In Berkeley, Calif., teams of computer scientists long ago concluded the system was both flawed and vulnerable, a finding that prompted the state to yank the system's certification for sale in that state.
The criticism largely involves older versions of Sequoia. Much as a house is repaired and expanded over the years, though, multiple add-ons and updates have been created to make Sequoia more accurate and safer.
But the system certified for sale in Florida hasn't been changed since 2015, according to state records, and those changes did not affect the underlying software or critical components involving most voters.
And it's not immediately clear whether updates and patches could entirely remedy the issues found by the California scientists.
Some could easily be fixed. Others were more concerning, such as "architectural decisions that were made very, very deeply in the system's design," University of Pennsylvania computer scientist Matt Blaze told The Palm Beach Post in 2012.
California screamin'
In a room at the University of California, Berkeley in 2007, Blaze, Ka-Ping Yee and a team of about seven other computer scientists were putting Sequoia to the test, poring over 800,000 lines of source code.Source code is the basis of any computer program; the programming instructions for how a system works. It's jealously guarded by elections systems companies: Sequoia once threatened to sue the state of New Jersey if it allowed Princeton University computer scientists to examine the source code.
At Berkeley, security was tight. The source code was locked in a safe every night. Janitors were not allowed to enter. The room had its own alarm system.
But the findings were made public, because the rare look at source code was part of a broader review of voting systems ordered by California's secretary of state.
"I anticipated that we would find problems, because the source code had not been reviewed independently outside of Sequoia," recalled Yee.
"I was shocked, however, at the number and nature and the pervasiveness of the problems we found."
The team reported that vote-counting software recorded 50,000 votes from a "precinct" with 100 voters. The system accepted "negative" numbers of votes. Every cryptographic algorithm — complex math designed to keep information secret — "was either obsolete, known to be weak or ... had obvious flaws," the team's report stated.
One team was able to break into the machines seven times. Five of 15 instances of poor encryption were found in high-speed ballot-readers, like those counting absentee ballots at election headquarters.
"Virtually every important software security mechanism is vulnerable," they wrote.
California's secretary of state declared the Sequoia system to be defective and inadequate — and ultimately pulled state certification for that version of Sequoia to be sold there.
Five months later, Palm Beach County paid $5.5 million for a slightly newer version of the same system.
No worse than any other system
Dominion Voting Systems would eventually buy Sequoia. But in 2007, it was a Dominion salesperson who warned Palm Beach County commissioners that Sequoia was "obsolete" and filled with old technology.There had been plenty of discussion about whether then-Supervisor of Elections Arthur Anderson should extend Sequoia's existing Palm Beach contract with a no-bid contract.
Susan Bucher, who had yet to run for supervisor of elections, quietly lobbied against purchasing it. California's blistering report was online and had been for months.
The county bought it anyway.
Anderson had not been impressed with Dominion's argument that the system was old. In a less-than-ringing endorsement, Anderson explained to The Post that Sequoia was no worse than any other elections system.
The next year, it was at the center of a vote-counting controversy.
Ping-pong recount for judgeship
As judge's races go, the 2008 match of Palm Beach County Circuit Judge Richard Wennet against attorney Bill Abramson wasn't expected to gin up much heat outside of the legal community.That was before vote-counting machines started adding and subtracting votes for both.
On election night, Abramson claimed victory with a 17-vote lead.
But at a church on Military Trail outside of Delray Beach, the cartridge that was supposed to record the 110 votes cast at that precinct had recorded no votes at all. At an elementary school in Lake Worth, 60 votes were cast but just 35 had been recorded. At a church on Spencer Drive in West Palm Beach, 43 ballots had been cast but only 11 had been counted. In all, workers found 198 uncounted ballots.
There was a recount. Abramson came up the loser this time.
A court ordered a second recount. Abramson came up the winner.
Human error played a role. Roughly 3,500 ballots had been misplaced.
Then-Assistant Palm Beach County Administrator Brad Merriman, tapped to oversee the office after the first recount, said it might never be possible to determine whether other problems were man-made or machine error.
But he pointed out that roughly 102,700 ballots were run through the machines several times, and "volume is always an issue with these machines."
Switching seats in Wellington
Like the 2008 judicial race, Wellington's hotly contested 2012 municipal contest appeared unlikely to trigger interest beyond city limits.And under Dominion, the Sequoia system had been upgraded, bolstering accuracy and security.
But on election night, winners were found in all the wrong places.
Shauna Hostetler was elected to Seat 1. John Greene lost.
Al Paglia won Seat 4. Matt Willhite lost.
Bob Margolis pulled 58 percent of the vote in the mayor's race, a clear win.
Six days later an audit turned the results upside down.
John Greene won Seat 1. Hostetler lost.
Matt Willhite won Seat 4. Paglia lost.
Margolis won with 50.6 percent of the vote, not 58 percent.
Bucher responded with a press release that began with: "Technology fails."
Dominion said the software did nothing to stop a "shortcoming" and also allowed the error to go undetected.
But Dominion balked at the idea that a bug in the system actually caused the error.
Bucher's sharp criticism of the company's error did not extend to immediately getting rid of the election system. Six years later, Sequoia was still counting votes for Palm Beach County; this time, running day and night to meet a state deadline for ballot counting in the heated 2018 midterms.
When tabulators stopped, Bucher cited overheating and decried that equipment as old, echoing a complaint Dominion had raised 11 years earlier when it did not own the system.
"There's no denying that this is an older system," Dominion's Stimson said; however, "It's really not their age, it's how they are being used," she said.
"It is outside the normal processes."
Palm Beach County far from alone
Sequoia is not the only Florida voting system to face criticism.At the 2018 Defcon convention, one of the largest hacker gatherings in the country, hackers easily broke into and discovered multiple vulnerabilities in a version of Miami-Dade's vote tabulator.
In 2012, almost half of Florida's votes were counted by tabulators that could malfunction in as little as two hours and start adding votes. The manufacturer issued a nationwide bulletin warning that it needed to be carefully cleaned to avoid adding "phantom" votes.
In a Broward County precinct in 2004, under certain circumstances, equipment counted about 32,000 votes and then started counting backward.
In Hillsborough County in 2008, a computer server crashed when early voting results were fed into it. In 2010, memory cards there failed. A spokesman for the supervisor of elections said the company admitted error.
If accuracy and security seem to attract recurring criticism, it's partly because voting systems are complex. They include much more than source code, or precinct ballot counters and the large central counters that tabulate precinct results.
There are also memory cards and cards to activate them, USB sticks, off-the-shelf software, special seals and scanner encryption. They all have to work, sometimes in tandem.
They have to work with humans, who have their own built-in potential for error.
And good software code can be hard — very hard — to write. Even Microsoft, which has deeper pockets and small armies of coders no election vendor enjoys, routinely sends out "patches" to improve its best-known products and resolve unforeseen glitches.
"It's difficult to impossible to make a software that is 100 percent perfect," said Yee. But it is possible to create software where errors are detected, he said, a crucial component in catching mistakes and creating confidence in voting systems.
Ion Sancho, former Leon County elections supervisor, agrees that trying to make a bullet-proof system is not the answer. "Protecting the front end to make sure no one can penetrate your systems is an exercise in futility," he said.
Sancho should know. He gained international notoriety in 2005 when he invited Finnish computer programmer Harri Hursti to try to break into his election system.
It was easy, the equivalent of "A house with an unlockable revolving door," Hursti later wrote.
What if everyone could count votes?
Sancho and some other elections officials, however, are encouraged by another type of voting technology.The older Sequoia system does not have the capability, but multiple Florida counties are now using newer voting systems in which ballots are photographed and can be stored. Brakey, the Arizona activist, has successfully argued in Ohio, and sued in Alabama, that the photographs are public records and should be retained.
Because they are anonymous, the ballot images can be made public, including publishing them online. That means candidates can do their own audit of any race, and any precinct. It gives voters the data for their own 100 percent recount.
Anyone, anywhere, can count all the votes.
"You could have a completely transparent system," Brakey said.
It's a system that also can spot elections system problems quickly and cheaply. In Humboldt County, Calif., scanned, publicly available ballot images revealed 197 votes dropped by machines and prompted an examination of the voting system.
Now, Brakey is in Broward County, where he says he is considering suing Florida's secretary of state to make sure counties do not immediately destroy the ballot images created by the newer systems. It's a first step toward making them accessible, and, he says, getting beyond the distrust created when voting systems don't work as expected.
Bucher, meanwhile, is wading through recounts, and with fewer machine problems.
"The machines have been working very well," she said. "I just think we can't operate them 24/7."
And, said Bucher, "We look very much forward to assembling new equipment for our next elections cycle."
Staff writer Joe Capozzi contributed to this story.
©2018 The Palm Beach Post (West Palm Beach, Fla.). Distributed by Tribune Content Agency, LLC.
