And it has to do with a cut cable. Vermont works with third-party vendor Tyler Technologies for its web hosting, which the company primarily runs for the state out of a data center based in the Washington, D.C., area. Nailor said his understanding is Vermont sites and services were offlined after a fiber connecting into that data center was severed. The incident began at 1 a.m. Wednesday, and it took until about 12:30 p.m. for sites and services to be restored.
The incident “was very impactful,” Nailor said. “They [Tyler] do all of our primary web hosting, so that was about 110 or so sites [downed] ... plus, they are our primary provider of online services for areas like the DMV.”
This incident echoes an outage that hit state sites and services in April. That event, too, was caused by a cut fiber serving the same Tyler data center. At the time, Nailor had called such an incident “unheard of.”
During Wednesday’s incident, the state’s monitoring tools detected the Internet outage around 1 a.m. Staff then became aware of the event at 5 a.m. and began trying to decipher the cause and troubleshoot, Nailor said. Around 8 a.m. Tyler Technologies notified the state as to what had happened. Restoration efforts managed to bring around 95 percent of the websites back by noon, and just about all were restored by 12:30 p.m.
Tyler Technologies said the incident impacted other organizations as well. It had previously said the same about the April incident.
“On June 28, 2023, our network vendor confirmed an outage at a primary data center in Virginia, impacting us as well as other organizations,” the company wrote in an emailed statement. “Our secondary data centers were engaged and we worked with our state partner to restore access."
Tyler also has a secondary data center in Texas. During the April incident, Vermont hadn’t been able to immediately switch over to using that data center, however. Instead, the state and Tyler had to work together to make domain name system (DNS) changes so state websites would direct to content at the Texas center.
Learning from that experience, Vermont had since set up a way for traffic to automatically reroute to the Texas site should the main data center go down.
But despite this new measure, something else went wrong.
“This time, Tyler experienced issues with their failover site being operational,” Nailor said, adding that he did not know the specifics of that issue. “While we addressed how we can make sure we routed traffic to their failover site in Texas, they had issues internally there.”
Tyler Technologies did not answer questions about what had stopped the resiliency measures adopted after the April incident from working to prevent this latest outage for Vermont.
Vermont websites were back up because Tyler had figured out a way to route traffic from their Texas data center to the D.C. site through a backhaul connection, Nailor said. That’s a temporary fix, and next steps would involve restoring normal traffic patterns.
"We are in contact with the network vendor regarding restoration of primary data center service," Tyler Technologies said in its statement.
The June outage caught the state as it is in the midst of implementing the resiliency efforts inspired by the April outage. After that earlier incident, the state had focused on some short-term efforts and lessons learned, and then planned to hold a disaster recovery (DR) test with Tyler Technologies on July 9. As a midterm goal, Vermont intended to shift out of Tyler’s data center and migrate to cloud hosting with AWS.
In light of this latest outage, however, the state now plans to accelerate its transition to cloud hosting. It may delay the planned DR test to make more time to focus on this migration.
“We’re going to really emphasize the migration to AWS, but as a part of that … my expectation is we would have routines for testing scenarios of outages, to make sure we uncover all the little links in the chain that could be a problem,” Nailor said.
"We are working closely with the state of Vermont to supplement resiliency measures, including migrating websites and services to the cloud," Tyler Technologies said in an email.
