The "ransomware" virus was the worst cyber attack in the city's history, encrypting 160,000 city files and demanding up to $33 million in the virtual currency Bitcoin to unlock them.
Only swift action from the city's information technology staff to literally "pull the plug" on the government's network saved the city from catastrophic data loss and financial cost, according to the investigation.
"In 25 years, that's the worst disaster I've ever encountered. ... It was an end-of-life event from the IT perspective," City IT Director Herminio Rodriguez told city investigators.
The February 2016 attack was contained in a matter of hours and IT staff's all-nighter restored the system by the next morning. It was only first publicly disclosed six months later, in late August, when city officials acknowledged the breach and emphasized no city employees' or residents' data were taken during the attack and all of the affected files were retrieved from the city's extensive backup archive.
But the incident itself was only the beginning of a bizarre series of events that included an Islamic State propaganda video, Russian hacking, the FBI and, ultimately, a Sarasota Police Department criminal investigation into city staff's handling of the virus after the attack.
An ensuing city administrative review of those events details several apparent miscommunications between SPD and city staff and a tense standoff about the investigation, ending with a State Attorney's Office decision not to prosecute city IT employees over alleged evidence tampering.
Although that review was completed last November, it was acknowledged and released publicly for the first time this week at the request of the Herald-Tribune amid an ongoing public records inquiry about the incident by the Florida Police Benevolent Association, city records indicate.
The report is the first time the city has disclosed the specifics of the attack and addressed in detail its own security responses to the breach.
The attack
The most severe cyber attack ever on the city of Sarasota started with an "innocuous looking email" with the subject line "scanned invoice" received about mid-morning on Feb. 25, 2016, a Thursday.A civilian SPD employee attempted to open the attached document, activating the malicious software that locks files and demands a ransom payment in return, according to the report. When the document didn't open, the employee forwarded it to the police and city IT service desks.
It would take less than four hours from the receipt of that email on the city email server for the virus, aptly referred to as "Locky," to spread throughout the system. Texts and phone calls from worried city staff members poured into Rodriguez and Information Security System Analyst Jackie Hemmerich, according to the report.
"In short order, Rodriguez and Hemmerich were forced to, literally, pull the plug on the entire City network in order to halt the encryption," the report states. "By then, Locky had corrupted 10-12 terabytes of data — some 160,000 files."
Although the software demanded ransom payments in Bitcoin, an untraceable digital currency, the city did not pay. Instead, the IT security team was able to work all night to purge the viral emails, restart system applications and restore all of the city's data from its backup system, limiting the outage for city servers to only one day.
The incident exposed critical vulnerabilities to the city's IT system.
Just two weeks before the attack, Rodriguez requested to upgrade the city's anti-virus software from the basic Microsoft product included for free, but was turned down because the new service was too expensive, according to the report. The city immediately upgraded to that software, called Sophos, in the aftermath of the attack.
The city also has since limited so-called "super admin" rights, in which some IT staff can access all servers and local computers, which helped spread the virus from the help desk to the rest of City Hall, the internal review concluded.
In March and April, SPD and city IT staff further investigated the attack together and compiled a report, which included a copy of the original email and virus on a thumb drive.
Crucially, they did not notify any other law enforcement of the attack at the time, a decision that would be scrutinized when the investigation escalated later in the summer.
"As our president of the United States says, this could be some guy sitting on a bed somewhere; we certainly didn't know for certain what came from where at the time," City Manager Tom Barwin recalled this week. "What we did learn, and this is now our standard response, is if and when we get hit with any attempt like this, we immediately go to the authorities.
"At the time it wasn't even a conscious sort of reaction, it was, 'Oh, boy another spam hit, we need to toughen our firewall.' We've now become a model of what to do."
ISIS and Russia
The ransomware attack would be the first of several alarming apparent cyber threats in the same summer that hacking and other similar attacks would headline international news and the presidential election.The first came during a "surprising visit" to City Hall from an FBI agent, who told Rodriguez that federal intelligence officials had noticed a photo of the city's email system in an Islamic State propaganda video, according to the internal investigation. That email system is made publicly available online for citizens or journalists to easily view many city officials' emails, which are public records under Florida's open-government laws.
The second came on June 14, when an unusual records request was submitted for Rodriguez's entire email inbox. The request came from an account using "Yandex.com," a Russian-based server, and would have included thousands of emails with proprietary security information that could be used to infiltrate the city's system, according to the investigation.
"'I built everything the City has' in terms of system security, 'and that's all in my mailbox,'" Rodriguez told city investigators.
The city did not fulfill the request. Instead, officials advised the massive request would require a $16,000 deposit and another $16,000 upon receipt — which is allowed based on how long the records would take to review and potentially redact — and the requester then abandoned the matter, according to the report.
Both curious instances came amid increasing ransomware attacks worldwide and, it is now publicly known, while national intelligence officials were investigating Russian attempts to hack local government and elections offices during the presidential campaign. Fear of both has only grown more intense this year and the massive international ransomware attack known as WannaCry rocked the world this summer.
Criminal investigation
SPD escalated the investigation into the ransomware attack on the city at the end of July, inviting FBI investigators to examine the virus and attempt to trace its origin.But that investigation quickly turned back onto city employees themselves after an unusual and ill-timed sequence of events that pitted city and police officials against one another.
The sequence began after SPD officials learned, apparently for the first time, that the initial attack began with a police employee's email, according to the city review.
On Aug. 2, SPD detectives and IT employees were in communication with a slew of city administrators and IT leaders about working together on the investigation and sharing information, emails showed.
As part of that, one SPD employee submitted a formal public records request to the City Auditor and Clerk's Office in an attempt to obtain the original email that would have transmitted the virus from SPD to City Hall. But the final request was broad, spanning months and about 5,000 emails.
The city's public records liaison, Karen McGowan, relayed the request to the city's IT department but did not disclose the requester was part of the SPD investigation. Per state law, a requester cannot be required to give his or her name to file a public records request, but neither does it prohibit the city from disclosing the requester's name, the report noted.
Rodriguez and Hemmerich were suspicious of the large request related to the virus following the Russian-based request and asked McGowan to identify who was seeking the records, but she refused, according to the report. They began to review the request and responded it would require a deposit, still not knowing it was a SPD request.
The following day, SPD's Capt. Corrine Stannish told Rodriguez to disregard the previous day's records request and asked personally for a much more specific set of emails around the time of the attack. SPD IT staff also found the original email on its own in a "deleted emails" folder and planned to show it to the FBI, according to the report.
In fulfilling Stannish's request, city IT staff realized there were still instances of the virus on city servers that could inadvertently be reactivated or accidentally released as part of fulfilling a public records request, Rodriguez and Hemmerich recalled during the administrative review. They immediately directed the purge of the viral emails, knowing that a copy of the original virus and email was still stored on the thumb drive made after the attack, they said.
But SPD leaders either did not know of or understand the copy and were shocked by the deletions.
"Captain Stannish found it incredible that emails related to the Locky virus, which SPD had planned to turn over to the FBI, could have been destroyed the day after she requested the Locky emails in her email to Rodriguez," the report said. "At this point, the joint SPD/FBI effort to gather evidence related to a cyber-crime morphed into an internal investigation of two City employees for the crime of evidence tampering."
No charges
The adversarial turn in the investigation pitted the two arms of the same local government against one another.The timing of the requests and the purge raised "red flags" for SPD investigators, Stannish said.
"'What it boils down to,' she said, is that 'when we started pressing, there seemed to be a motivation to hide mistakes,'" the report says.
But federal investigators ultimately were able to use the copy of the virus on the thumb drive in their review of the incident, which was not technically a formal investigation but only part of broader information gathering on such attacks, according to a final review and city officials.
While the unfolding events led SPD officials to believe "they were being accused of causing the Locky virus," city employees felt threatened by the sudden criminal investigation into their conduct, both sides relayed during the review.
"Everybody knows that mere deletion of an email doesn't delete it from the system," Hemmerich said. "When we gave them (SPD and FBI) the file, it should've stopped right there."
SPD ultimately asked the State Attorney's Office to review whether the deletions constituted evidence tampering, but Assistant State Attorney Art Jackman determined there was not sufficient evidence of intent to tamper, according to the report.
"In the end, the FBI appears to have been satisfied, the state attorney reviewed the matter and found nobody was tampering with any evidence and then we did our own review internally," Barwin said this week. "I think the report sort of shows that was the case. The finding was that IT acted responsibly and timely. SPD, once they learned of this, went after it aggressively.
"In the end, everybody acted appropriately, even if it got a little tense, which happens sometimes."
Security changes
The final internal investigation report issued harsh words for both city and SPD staff in the aftermath of the attack.The report found both sides at fault for not communicating "in any substantive way" after the virus was contained and ordered the city review its protocols for responding to such incidents, handling digital evidence and sharing records without a formal request that a citizen would be expected to submit.
The city also has upgraded its firewalls in addition to the improved security software and now has cyber-liability insurance coverage, which became effective Oct. 1.
The internal investigation concludes with an emphasis that it was not designed to assign any fault for the series of events that took place and that it recommended no employees be disciplined.
"The writers of this report have avoided second-guessing and casting blame, and have focused instead on positive recommendations for improving the safety of the City's IT system and for improving relations between departments," the report concludes. "We have endeavored to present this material in an unfiltered way ... None of the employees we interviewed appeared to have acted in bad faith or with improper motives, but were instead doing their best to act in the City's interest."
Public disclosure
But the new revelations about the attack raise questions about the city's public response to what its own leaders describe as a historic and potentially catastrophic intrusion.That an attack ever occurred only became public during the same week as the SPD investigation into the city's conduct in late August.
It was first suggested by a brief report by a local television station and later confirmed by the Herald-Tribune with City Manager Tom Barwin on Aug. 24 — a day before the State Attorney's Office chose not to pursue charges against Rodriguez and Hemmerich.
Barwin and spokeswomen for SPD and the FBI confirmed at the time the ransomware attack was under investigation but did not say that city staff members were being investigated.
Virtually no details about the scope of the attack and the ensuing investigations were made available at the time and no officials referred to the suspicious incidents regarding the Islamic State or Russian-based request. Barwin said it had not been disclosed earlier to avoid alerting would-be threats that the city had ever been vulnerable.
Barwin asked the city conduct an administrative review of the entire situation on Aug. 26 and assigned it to Human Resources Director Stacie Mason. The city did not publicly acknowledge this review, nor did it release the final report when it was completed on Nov. 15.
The final report was only acknowledged Thursday after a reporter raised questions about a detailed request for documents related to the incident by the Florida Police Benevolent Association, the union representing members of the Police Department.
That request for many of the emails detailed in the investigative report has not been completed yet.
The reason the administrative review was not disclosed last fall is itself an anomaly, city spokeswoman Jan Thornburg said. She was out of town when the incident was first reported by local media and did not receive follow-up inquiries from reporters, so the story simply "died down," she said Friday morning.
When the review was complete, Barwin was satisfied with its conclusion and briefed city commissioners in their private, one-on-one sessions with the city manager each week. None of the commissioners asked the report be discussed publicly, Thornburg said.
"After all of it, let's have an administrative review to see what was done right, what could be better in the future and how we could guide ourselves in this new era in which hacking and ransomware is a new reality we've got to deal with," Barwin said. "If you digest the whole report from my perspective, trying to administer and manage the city, there is a lot of good guidance that came out of that and a lot of lessons that were learned. It was internal operations, administration and figuring out how we're going to deal moving forward with ransomware and hacking based on the experience we had."
©2017 Sarasota Herald-Tribune, Fla. Distributed by Tribune Content Agency, LLC.
