This point has been made (and repeated) many times by well-known people in various ways and circumstances across different generations.
Remember the line: “A crisis is a terrible thing to waste,” from Rahm Emanuel and Paul Romer? Or, "If You Have a Lemon, Make a Lemonade," from Julius Rosenwald and Dale Carnegie?
Well no matter how you slice it, the Iowa Democratic caucus mobile computer application (app) that was placed into action this past week qualifies as a lemon.
So how can we make lemonade out of this Iowa caucus app “debacle” situation?
Putting the politics, conspiracy theories, election ramifications and inevitable finger-pointing exercises aside, what happened to cause such an issue? Equally important, what didn’t happen? Another related question for all of us includes: What is the true cost when an app fails?
Online retailers have precise models that can analyze revenue loss down to the seconds when an app crashes. Fast food chains can tally lost sales. Marketing and PR firms can quantify brand damage impact from clunky design. Even app hacks and identity theft costs can be quantified after a security incident.
But in the case of this Iowa caucus app, we may not know the answer to that cost question for years, due to huge number of ramifications and possible future outcomes. Some are even wondering if the Iowa caucuses will ever be the same or maintain the “first in the nation” status in the 2024 elections.
A Quick Iowa Caucus App Primer
But backing up a bit, NPR reported in January that the Iowa's Democratic Party was confident in the new app and also the backup processes in place for the February caucuses. Security was a top concern.
Troy Price, the chairman of the state party said, “We as the party have taken this very seriously, and we know how important it is for us to make sure that our process is secure and that we protect the integrity of the process. …”
However, on Feb. 4, the AP reported that the caucus app that was supposed to speed up reporting was blamed for delays that left results unknown for many hours.
“Glitches with a new mobile app Monday caused confusion, and some caucus organizers were forced to call in results for the state party to record manually, introducing delays and the possibility of human error. Iowa Democratic Party Chairman Troy Price said the delays were not the result of a breach and party systems were secure.”
On Feb. 5, NBC News reported, “The smartphone app that caused a significant delay in reporting Iowa caucus results suffered from technical and design flaws and appeared to have been rushed into use, according to cybersecurity experts who examined a version of the app that was made public.
The app became the subject of widespread scrutiny after the Iowa Democratic Party said problems with reporting the caucus results were due partly to "coding issues" with the app, which was being used for the first time.
The app was supposed to be the "preferred" method for caucus chairs to submit results, but only about a quarter did so, said Gerard Niemira, the CEO of Shadow Inc., which developed the app for the Iowa Democratic Party. The majority of chairs opted to call in, jamming understaffed hotlines. …”
On Feb. 6, NBC News reported that, “The Iowa Democratic caucus results are rife with potential errors and inconsistencies that could affect the outcome of the election…
The apparent mistakes — spotted in at least dozens of the state's 1,711 precincts — call into question the accuracy of the outcome of Iowa's first-in-the-nation caucus, which was held Monday night. …”
On Feb. 7, NPR reported that after the Iowa app problems, Nevada Democrats will not use the app for their caucuses.
In this New York Times opinion piece called, The App That Broke the Iowa Caucus, Charlie Warzel called this episode a “massive failure” in the Democrats' attempt to win the Internet to beat Trump.
What Went Wrong with the App?
But I want (to try) to take politics out of this discussion. Yes, the context of the Democratic caucuses in Iowa is political at the core — with all the world watching election results closely. Nevertheless, despite the stage that was set for the bad app publicity, there are certainly potential lessons learned for all of us — if we want to learn them.
TechCrunch reported this: “'Honestly, there is no need to attribute conspiracy or call shenanigans on what happened with the new app during the Iowa caucuses,’ Dan McFall, chief executive at app testing company Mobile Labs, told me in an email. ‘It’s a tale that we have seen with our enterprise customers for years: A new application was pushed hard to a specific high profile deadline. Mobility is much harder than people realize, so initial release was likely delayed, and to make the deadline, they cut the process of comprehensive testing and then chaos ensues.’”
Others say it was concern for security and a late a late patch that caused the chaos. This quote is from the Des Moines Register:
“The reporting app that is getting a large share of the blame for the chaos surrounding Monday's Democratic caucus results was working until the national party required the installation of a security patch less than 48 hours before the first-in-the-nation contest, a recent member of the Iowa Democratic Central Committee said Thursday.
The update is believed by some Iowa Democratic Party staffers to be the reason for a mismatch between the app’s coding and the state party’s computerized verification system that caused omissions in the results, said John McCormally, a former state Democratic staffer who was a member of the party's central committee until last year. …
Doug Jacobson, director of the Iowa State University Information Assurance Center, reviewed the app Thursday at the request of the Register. He said he believes a link that Iowa Democrats provided to caucus officials to download the smartphone app was on a site that had low security and was vulnerable to tampering. And he said instructions provided to users to help them troubleshoot problems were poorly written and hard to follow, setting the stage ‘for various possible cascading failures.’”
Others say a lack of training and group testing was to blame for the problems. “The volunteers running the caucuses were not trained on how to use the new app that powered the entire process. The app wasn't included in the chair training that everyone was required to take," Zach Simonson, the Democratic Party chair in Wapello County, told The New York Times.
Training and technology experts agreed that not training people during a technology transition is almost always a fatal error. Corinne Jones, president of CJC Human Resource Services, said that skipping the training step during a transition is an expensive mistake."
One more: The New Yorker Magazine wrote this article with a variety of concerns about Shadow, the company that built the app.
Nothing to See Here? — Most App Issues Don’t Get So Scrutinized
No doubt, 99.9 percent of public- and private-sector app launch situations do not get nearly this amount of attention and “Monday morning QB” analysis. The global impact and one-of-a-kind interest in the Iowa Democratic caucuses in 2020 were so much higher than we typically see for governments or businesses.
It (at least initially) appears that a strong concern for security led to last-minute patches that caused quality issues that may have undermined the wider app functionality. As we have repeated numerous times in this blog, people, process and technology must be addressed from start to finish throughout the life cycle of any app development and deployment.
And this reality cuts two ways. One would think that with so much at stake, all of these basic app development and security testing steps would have been double and triple-checked in Iowa. But it did not happen with the needed level of expertise.
The sad truth is that various rollout problems occur more often than most people realize in government and business app development and rollout situations all over the world. Despite some critics who claim that an app was not even needed in this situation to count Iowa caucus votes, almost everyone wants “an app for that,” just as in this State Farm commercial.
I hope this Iowa caucus app becomes a case study we can all learn from. The lessons from this failure can “offer lemonade” if public and private sector executives take notice and modify their app development processes to ensure quality is job #1 in all phases of the app life cycle (from design to upgraded versions).
Vox reported that experts are worried the Iowa debacle could threaten the legitimacy of the entire election process:
“The timing for a debacle like this could not be worse,” election law expert Rick Hasen told Vox’s Sean Illing in a recent interview. “If people lose trust in the process, the very basis of democracy is undermined.”
As much as Iowa’s problems can be chalked up to incompetence rather than anything more nefarious, it thrust the prospect of election interference into voters’ minds. It certainly caused multiple New Hampshire voters to wonder whether America’s voting system was secure enough.
“Things keep going wrong with the election process. It’s disconcerting and unnerving,” said Walpole voter Mary Armbruster. “It doesn’t feel secure.”
“What a shame — out of the gate, to start like that,” said voter Kathy Frick of Keene. “Everybody’s so jittery about someone coming in and hacking.”
Closing Thoughts
No, this Iowa caucus app failure situation was not the "Cyber 9/11" or "Cyber Pearl Harbor," which many have predicted in some cybersecurity circles. Most technology and security pros don't even consider it a cyberincident.
Rather, it was an event that virtually no one expected, and fear of cyberincidents from 2016 events may have actually influenced the problems created — or perhaps not. Regardless, it was a set of massive mistakes that included some combination of people, process and technology failures with significant ramifications that go far beyond the $60K that was paid for the app development.
One irony is that a lack of testing and training for the app not only created performance issues and quality-control issues for the caucus reporting processes (by some accounts), but it also undermined the faith in wider election security in other states — even though security was a major focus in the app development in the first place.
Meanwhile, there are thousands of other apps being developed, tested and deployed globally in the public and private sector this year. Some impact personal safety, human lives, emergency management, sensitive data and more. Other apps do not, but could provide a backdoor to more critical apps or data. Regardless, tech executives and development teams should take note of mistakes made in this situation.
Will CEOs, CIOs and CSOs do anything differently based upon what they’ve read and learned from this Iowa app situation? I am not suggesting that all apps need the same level of review, but surely experts knew the importance of what was at stake in these Iowa caucuses.
Will anything change in other essential business app rollouts?
One final thought: At the urging of a friend, I recently watched the HBO mini-series called Chernobyl. (The series is simply amazing and well worth watching.) Despite the horrific Chernobyl events and lives lost, thankfully, some change did occur as a result. When the story ends, the other nuclear reactors in Russia are fixed, and processes and training of staff improved, along with overall safety of their nuclear power plants.
Although nowhere near as consequential as Chernobyl, I expect someone, somewhere, is already discussing the movie script/rights for this Iowa Democratic caucus app story. I expect to see more “people, process and technology” details emerging about what happened with the app — well before the 2024 elections.
But the initial question remains: What will we (CxOs, tech pros, government staff, business people, app developers, code testers, investors and cyber experts) learn — and do differently — as a result of what just happened with the Iowa Democratic caucus app?
