Many of their questions were met with reluctance from administration officials to publicly discuss details. They said they wanted to protect confidentiality and didn’t want “adversaries” to know or to share personnel matters.
“I understand you all have a lot of questions and we want to do our best today to answer them in doing so though we must be careful to protect the details of policies, processes and controls, said Neil Weaver, secretary of the Office of Administration that oversees information technology operations for commonwealth agencies. “The disclosure of such could compromise our cybersecurity posture and be exploited by bad actors to launch cyber attacks.”
The Senate Communications & Technology and State Government committees held the informational hearing to learn more about the data loss caused by what Weaver called “an incredible serious human error that I don’t take lightly.”
Sen. Tracy Pennycuick, R-Montgomery County, who chairs the communications & technology committee, said afterward the hearing was the first of many hearings they were going to have to look into the matter.
“I honestly think we’re just trying to assess where we are and how big the problem is, and identify the shortcomings and the loopholes we need to resolve,” she said.
Weaver said his agency is conducting an internal investigation and hired Seattle, Wash.-based Layer Aleph. The company specializes in IT incident responses and can assist with making recommendations to update its policies, processes and controls in hopes of avoiding a future occurrence.
About the incident, Weaver said a database administrator was performing routine server maintenance on Jan. 3 when the error occurred.
It disrupted multiple agencies and lost data from two applications used by the Pennsylvania State Police and one by the State Employees’ Retirement System. He said the employee self-reported the error but he declined to say whether that employee was fired. Administration officials, however, have confirmed one employee was fired as a result of the incident.
“We took immediate action internally but I can’t get into the personnel side,” Weaver said.
Despite being asked in several different ways, Weaver and Chief Information Office Amaya Capellán declined to say if the employee was working remotely. They said the employee’s location had no impact on the incident.
They declined to say if the employee was involved in helping to recover the data. They wouldn’t disclose the location of their servers or if the state police and SERS data were stored on the same server. They also declined to discuss their policy for backing up data, saying only they have “regular back ups.” They also refused to say whether encryption keys are typically stored with the encrypted data.
“I can’t comment on how we keep encryption keys in open forum. It’s not something we want to do or want adversaries to receive,” said Chief Information Security Office Jim Sipe.
Asked if a policy violation led to the data loss, Capellán said, “The core of this was human error and there was an employee who was doing their job as a database administrator, who had access that was in line with the job that they had to do, and there was an error that perpetrated this incident and that’s the root cause of this.”
The error affected 77 out of the state’s over 6,300 servers that it manages. Administration officials didn’t say how many agencies’ data was on those servers but agreed to supply that list to the senators.
Over the course of four days after the deletion, and working round the clock, the IT team was able to restore data to 76 of the 77 servers. On Jan. 7, Weaver said, “we realized the issue that was in front of us.”
Capellán said, “Until Jan. 7, the team had no reason to believe that we would encounter any difficulties restoring data.”
When they got to the 77th server and couldn’t restore the data, Capellán said she was notified and then she notified Weaver. It was then escalated to the governor’s office and to Shapiro the next day. Weaver said the affected agencies were informed on the evening of Jan. 7.
Pennycuick asked why the General Assembly wasn’t informed at that time.
“When this happened, we needed to make sure we knew the breadth of it and needed to make sure that we didn’t come to you with a problem without a solution,” Weaver said. “We didn’t want to put it out in the public domain without having all the answers to the solution.”
Pennycuick responded, “I don’t think we’re the public domain but I think it would have been nicer had you brought us in earlier.”
The lost state police data included information that tracks evidence for its crime labs, some of which is now being re-entered manually by state police personnel.
According to the state police’s written testimony, all physical evidence remains in the crime labs’ possession and is secure. However, evidence submitted to the labs between June 15 and Jan. 3, along with any that was not fully processed since June 15, is being re-entered manually into an application that manages and tracks records and generates laboratory and statistical reports. To date, it says the application to submit evidence to labs remains unavailable.
“As a result of this incident, aa criminal investigation was initiated by the PSP Bureau of Criminal Investigation into the circumstances of this event as required by [Criminal Justice Information Services] regulations,” according to the state police testimony. “The results of that investigation are pending. In addition to the criminal investigation, a root cause analysis is required to maintain laboratory accreditation.
SERS reported no pension data was lost but that members logging on to its system would have to verify their identity to sign on and create new four-digit PINs.
“It could have been much worse,” SERS written testimony states. The agency’s IT team got its online and member employer services reopened for use by Jan. 16.
Sen. Mike Regan, R-York County, asked if state police, who conducted a separate investigation into the deletion, were given access to the physical servers and the employees involved in the deletion. Weaver said the state police contacted his office and all access was handed over upon their request. Regan asked if he had any information into the state police investigation into the employees. Weaver referred that question to the state police. State police’s written testimony indicated that investigation is still pending.
“I would request that we convene another meeting where we can empanel the state police to appear before us,” Regan said.
Afterward, Sen. Kristin Phillips Hill, R-York County, said there needed to be additional hearings to address not only last month’s data loss, but “overall systemic issues that we have observed.”
She and other lawmakers expressed a desire to pass legislation to put some guardrails to ensure data stored on state government computers is safe and secure.
“I think we absolutely will need to legislate some procedures and policies,” Pennycuick said.
Weaver said he understands the frustration, but told lawmakers that legislation isn’t needed with him at the helm.
“I have the full support of the governor to make the changes that we need to make and we’re very aware of where there are issues. We’re very aware of where there’s shortcomings and we’re going to take care of that,” he said. “You have my word.”
© 2024 Advance Local Media LLC. Distributed by Tribune Content Agency, LLC.
