Leanlab has been matching school districts with ed-tech developers to conduct this type of field research for about 10 years, according to founder and CEO Katie Boody Adorno. In response to the recent explosion of AI-enabled tools for teachers and students, much of the nonprofit’s codesign research is now focused on AI technology.
“What we’ve been doing over the last year is working with a group of AI companies, as well as our school and educator partners, to test and trial these solutions in real, authentic learning settings, to learn more about what’s occurring when the rubber meets the road of AI implementation,” Boody Adorno said in the webinar.
The event gave an overview of what researchers, ed-tech companies and educators discovered in six classroom studies involving five AI-powered ed-tech tools, conducted from January to August. Speakers at the webinar did not use the specific names of the tools and companies.
Leanlab Principal Researcher Lynne Harden said the expectations of educators and the intentions of developers involved in these studies were fairly well aligned: save educators time, support differentiated instruction, enhance student engagement and help teachers with lesson design, delivery and feedback.
“However, when we took these tools for a spin in authentic classroom environments, we frequently uncovered technical issues that often prevented easy and streamlined use of the tools, as well as some shortcomings in terms of their ability to differentiate content,” Harden said. “Additionally, we discovered some perhaps more interesting potential barriers to use, one being that the tools did not always integrate well into educator workflows, and actually, intriguingly, many educators struggled to fully trust the accuracy and quality of AI outputs.”
CONTENT PROBLEMS
In one study, educators flagged the fact that an AI tool was taking up to an hour to generate lesson plans. They relayed this information to the developers so they could prioritize a fix, Harden said. Educators encountered a similar problem with an AI tool designed to generate content for students, with an added twist: Not only did the content take too long to generate, but once created, it was not complex enough to keep students engaged for more than a few minutes.
“In this case, the developers were able to improve the content loading time and increase student engagement time with more complex content,” Harden said. “Teacher and student participants were actually able to see and experience these changes even before the study was over.”
Another important insight centered on AI content differentiation, which is supposed to give teachers and students the ability to tailor educational material to meet various skill levels. Educators noted that two AI tools designed to provide content differentiation were not useful to students on the lower end of the skill spectrum, because the tools assumed a certain level of mastery, Harden said.
“If you think about what we’ve learned, in terms of some of these tools are not actually helping those kids who are struggling the most, that’s a pretty significant disconnect between what students need and what teachers need and what these tools are able to do,” she said in the webinar.
Leanlab took this feedback to the developers and made recommendations for improvements, such as adding more scaffolding and support within the tool for students with lower skill levels, allowing students to slow down the pace of audio and video content, and giving teachers the ability to adjust the reading level or difficulty of the lesson.
BUILDING UP TRUST
One of the most important findings from the classroom studies, Harden said, was that educators do not yet trust the accuracy and consistency of AI-powered ed-tech tools, and that every time they encounter unpredictable output, it reinforces their skepticism.
The process of conducting codesign research can help address this issue, according to Andy Midgette, director of ed-tech partnerships at Leanlab. It’s also the key to creating a product that students and teachers can actually use, he said, which is a must before any kind of widespread outcome testing can begin.
“The rapid-cycle evaluation allows for that quick iteration of the tool that not only strengthens the tool itself but also strengthens the trust we’ve been discussing this whole time, by taking that feedback directly from users and making those changes in real time,” Midgette said in the webinar. “Once those iterations have been built in an operationally stable product, then we can conduct more feasibility studies to strengthen implementations, and from there we can prepare for more rigorous correlational and quantitative studies to show the impact these tools are having.”
