And so the agency moved forward with the project, launching the new site during January 2019. The decision ended up being disastrous, as the agency neglected to do one vital thing: gather feedback from users, many of whom visit the site every day.
Not only was there no user testing for the new portal, no banners or teasers were created for the old site to inform citizens that a massive change was on the way.
“We didn’t let people know that this was coming,” said Daniel Catley, Virginia Lottery’s digital customer experience manager.
The fallout was pronounced. Before the new site’s launch, Virginia Lottery saw an average user satisfaction score of about 80 (out of 100). After the launch, the organization observed a “Thelma and Louise style dive off the cliff” with the scores, Catley said. For mobile users, scores landed around 50. For Web users, even worse: 24.
Users faced several problems. Many could no longer find the information that they had always checked on a daily or weekly basis. Some faced more technical challenges if they used a particular browser, such as Internet Explorer. Because of the responsive design, the old mobile version of the site went away, leading citizens to busted links.
Sunny Beach, a senior usability analyst for software company Verint, has been a consultant for the Virginia Lottery for roughly nine years. Beach said Verint was able to spot a few potential usability issues before the new site went up, but the decision to launch had been made.
“You always want to include the voice of the customer, understanding why they’re coming there in the first place and what they’re trying to do,” Beach said.
The Turnaround
Today, user satisfaction scores for the mobile and Web versions of the Virginia Lottery site fall between 70 and 80, Catley said.How did that happen? The recovery started in spring 2019, when Virginia Lottery began regularly gathering user feedback through tests and surveys. Catley said improvements have been implemented in two-week sprints as part of an agile SCRUM process.
Lainey Naglic, digital product owner for the Virginia Lottery, has played an important role in the recovery. When the new site launched, Naglic had just joined the agency, but for the post-launch recovery, Naglic has served as the liaison between the larger organization and the Web team, prioritizing the features that need immediate attention.
The first large-scale improvement involved changing where winning lottery numbers appear on the home page and how these numbers can be accessed on other pages. Another significant adjustment was giving users more time to enter their tickets before the site automatically signs them out.
All the changes have been informed by data, and some tweaks are part of a proactive method where issues are eliminated before user complaints become frequent.
“The biggest lesson for us is really allowing the data to drive this,” Naglic said. “We’re always trying to pick up on trends so we can address them before they become a problem.”
Along the way, Verint has helped Virginia Lottery measure customer experience with different tools.
“The data Virginia Lottery collects with us is fairly specific,” said Andrea Fuhrel-Forbis, client success manager with Verint. “They have a lot of questions about what people were trying to accomplish on the site, as well as how they got around the site, how well the site is organized. … After the relaunch, we were able to track the changes on these types of questions and really specify what the problems were.”
Catley and Naglic emphasized that continual improvement has been essential to the site’s eventual success. Right now, Virginia Lottery is addressing navigation, but after that work is done, user surveys will likely reveal another set of changes.
“There’s always something new, and we always listen,” Naglic said. “How can we fit this in with the other [larger] initiatives?”
Beach said one of the lessons from Virginia Lottery’s experience is to, when possible, introduce tweaks to users in an incremental way. Don’t go all-in on the latest craze or on what one person wants.
“When you have everything change overnight, there’s a lot of frustration,” Beach said.
Catley also mentioned that Virginia Lottery utilized a partner it had previously worked with for the site redesign. While leveraging an existing vendor or contract can help accelerate a project, an agency must still exercise discretion. Here, the partner in question had difficulty implementing some of the site’s core functionalities. As a result, Catley said Virginia Lottery would use some stricter screening criteria when it comes to getting help from an outside source in the future.
