3 Key Ingredients for Making the Most of State Data Work

Despite the boom in data science, government projects that involve large data sets and powerful data tools still have a surprisingly high failure rate. Government agencies are increasingly seeking ways to use the data they already collect to achieve their measurement, evaluation and learning goals, but they often do not have the capacity or the right mix of staff to carry out data projects effectively.

The Center for Data Insights at MDRC, a nonprofit, nonpartisan research organization, recently partnered with state agencies to develop and execute a variety of data projects. We learned that barriers to success are not primarily about technical issues or analytic methods. Rather, data projects need three essential ingredients to be successful: people, perseverance and project scoping.

PEOPLE

Successful data projects are more about people than data — both the people served by the agency’s programs and the people on the data project team. Projects should adopt a human-centered design approach, which focuses the data analysis design on the people most affected by the projects. It requires staff to rethink and transform the way project design and analysis are usually deployed and may require consulting with the people the program intends to serve. Equally as important, project teams also should be interdisciplinary to incorporate people of diverse skill sets beyond data expertise so that important perspectives that can make or break the project are captured.

For example, MDRC worked with the New York State Office of Temporary and Disability Assistance to explore factors associated with long-term cash assistance receipt. The agency gathered an 11-person cross-functional team that included researchers, programmers, employment experts and operational staff members who worked regularly with local offices. Team members who did not have technical expertise provided content expertise and contextual information that were instrumental for both data quality assurance and interpretation of the analysis. The collaborative process prompted the technical staff to ask questions such as “How can different local offices use this analytical information in a practical way?” as they conducted their data analysis.

PERSEVERANCE

Perseverance is essential to success in any data project. Teams using new data techniques often go through a “hype cycle” in which high expectations for exciting results from a planned data analysis are frustrated by an analytic challenge. Successful teams persevere and adjust their original plans as needed.

The Colorado Department of Human Services was exploring the use of supportive payments, which are additional cash payments that can be used for the basic needs of the most vulnerable families who participate in the Temporary Assistance for Needy Families (TANF) program. They first wanted to know how the timeliness of certain types of supportive payments were related to employment outcomes, but the way the data had been recorded and tracked did not allow them to analyze data by payment type. Once they adjusted their research question to investigate the relationship between payment receipt and employment, they found selection bias issues that led to misleading findings about supportive payments. The team then tried several different ways to reduce the bias before identifying the approach that more accurately estimated the positive contribution of supportive payments to employment outcomes.

The team was able to draw insights from their analysis because they had the determination to persevere through data quality, selection and precision challenges. They also plan to continue both supplementing their analyses and improving their agency’s data collection and documentation practices to better support Colorado families who rely on TANF services.

PROJECT SCOPING

Project scoping is a way to set boundaries on your project by defining specific goals, deliverables and timelines. Designers should make room to be agile as they determine the scope of their data projects. The idea is to start small and then use what you learn to build more complex and nuanced analyses.

For example, the Washington Student Achievement Council (WSAC), the agency that oversees higher education in the state of Washington, wanted to learn about whether the United Way of King County’s Bridge to Finish campaign, which provides services to college students who may be at risk of insecurities for food, housing or other basic needs, could help students persist and earn a degree. The project scope began with a simple task: specify demographic and service use characteristics of students that may be associated with academic persistence and determine if these characteristics are measurable with the available data. This allowed the team to focus on the questions that were answerable based on data quality and completeness: Did the program recruit and serve students from historically marginalized groups? Was the program model flexible enough to address students’ most pressing needs?

If instead the project had been scoped to begin with more complexity, like building a predictive risk model to identify students who might not persist or complete college, the project would have been stymied because of insufficient data and an incomplete analytical tool. For the Bridge to Finish campaign, the simpler approach at the outset, with a project scope that was flexible enough to change as data challenges emerged, ended up leading to findings that were much more useful and actionable.

Setting up data projects for success is not primarily about data itself. Instead, it is about people who are planning designing, and pushing through challenges together. Projects that are scoped effectively and that encourage project teams to persevere through challenges yield better results, richer findings and ultimately help government agencies fulfill their missions.

Edith Yang is a senior associate with the Center for Data Insights at MDRC, a nonprofit, nonpartisan research organization.