As assistant director of psychometrics at Accessible Teaching, Learning and Assessment Systems (ATLAS), a research center at the University of Kansas, Thompson’s work focuses on the measurement of student knowledge, skills and abilities.
More specifically, he studies the use of diagnostic classification models (DCMs), which can assess student proficiency on any number of educational skills. Thompson said DCMs are potentially valuable, because psychometricians can translate them for teachers into fast, fine-grained feedback on student strengths and weaknesses, which can guide instruction to improve student learning.
However, DCMs are difficult for most psychometricians to use, mainly due to a lack of user-friendly software, Thompson said. He received a grant from the Institute of Education Sciences (IES) to develop software that could simplify the use of these statistical models.
Having never before published an externally shared software package, Thompson said he wanted to gain some practical experience first — and that’s where Taylor Swift comes into play. Thompson, a self-proclaimed Swiftie, built and published a software package called taylor, with data sets on the pop star’s lyrics, songs and album release history, as a low-stakes trial run for the IES grant.
“It was really just a way for me to learn,” Thompson said. “I had built a couple packages before but not anything super complicated or externally shared, and I really wanted just like a playground where I could try new things, see what worked, and go through the process of publishing a software package.”
Of course, like most things pertaining to Swift, taylor took off — at least in the online data science community, which used it for practice — and Thompson said it ended up being one of the most popular projects he’s worked on.
With taylor complete, Thompson took his newfound skills into the work of building and publishing the IES-funded software package for psychometricians. Called measr, the package automates the process of estimating and evaluating DCMs.
To start working with measr, student assessment professionals must have a data file containing several hundred student responses to a particular set of questions. The questions can be broad or specific and pertain to whatever skills the educator or researcher would like to assess. The user must then load the file into measr, which analyzes the data to estimate, or set the parameters for, student proficiency on each item.
“Once you have your model parameters, any new data that comes in, you can just feed it into the model and get updated estimates of whether the student has mastered the skills or not,” Thompson said.
Prior to the development of measr, setting parameters for a DCM required deep diagnostic experience and extensive coding skills, Thompson said. His hope is that more researchers and psychometricians will start using DCMs now that the assessment process can be automated.
“These models are great, more people should use them, and measr is my first step in trying to make it so that people can use them,” Thompson said. “Because I do think they are just really beneficial for helping understand where students are at and where they can take their learning next.”
He recently received a second IES grant to continue developing measr, but said the free open-source software package is available now and ready to use. In fact, Thompson is taking advantage of measr as lead psychometrician on a four-year pilot project led by the Missouri Department of Elementary and Secondary Education in partnership with ATLAS. Dubbed the Pathways for Instructionally Embedded Assessment, and funded by the U.S. Department of Education, the pilot uses measr to evaluate DCMs designed to assess math proficiency among fifth graders throughout the entire school year.
