Jeff Livingston, a longtime executive in educational publishing and founder of the nonprofit Center for Education Market Dynamics, said the market has rewarded superficial engagement over effectiveness by “selling clicks and views.” He said some ed-tech tools have been funded and sold with minimal evidence of effectiveness and having only been tested in narrow, unrepresentative environments, like Palo Alto, Calif.
Kristen DiCerbo, chief learning officer at Khan Academy, said educators and students sometimes struggle even with ed-tech tools that have already demonstrated their quality.
“We can show you the data — this much practice leads to this much increase in student learning outcomes — but we're getting 10 percent of kids to that level of usage,” she said. “There’s so much out there in the space that I think it's hard to decide what to use, when to use, [and] how do you implement it in your classroom in a way that gets those outcomes.”
SMARTER PROCUREMENT
Panelists emphasized the role of administrators, who often handle ed-tech procurement, in shaping what kinds of educational tools succeed. Livingston said purchasing decisions often reward vendors for sales, rather than outcomes.
Outcome-based contracts can add an incentive for quality where none existed before, Livingston said. They include clear metrics for desired progress and corresponding bonuses awarded for achieving it.
Richard Culatta, CEO of ISTE+ASCD, said educators and administrators need better tools and guidance to make informed decisions. His organization partnered with six others last year to produce a list of five quality indicators for ed-tech tools: safe, evidence-based, inclusive, usable and interoperable. A corresponding product index compiles trustworthy evaluations of ed-tech tools based on these quality indicators.
“That is the sort of market tool that in almost every other industry exists,” Culatta said. “It has not yet existed in the education space.”
ISTE is taking additional steps in planning its annual conference this summer to support education professionals in making wise purchasing decisions. The organization is now offering priority space and time on the expo floor to vendors that demonstrate efficacy. By next year, tools that don’t meet certain standards won’t be allowed on the expo floor at all. ISTE will also offer information slips with guidance on questions attendees can ask vendors to assess quality.
“We're talking a lot about building better products. We also need to build better buyers,” Culatta said.
Guidance like this can help administrators discern strong evidence of effectiveness from graphics or phrases that sound nice but ultimately mean nothing — phrases like “six months of learning growth,” DiCerbo said.
It can also steer administrators away from traditional standards that are not as helpful as they may seem, like randomized controlled trials (RCTs). DiCerbo said that while RCTs are often considered the gold standard in experimentation, they don’t always reflect the complexity of classroom implementation. With Khan Academy's AI-powered teaching assistant Khanmigo, she said, her company spent the first year understanding how students and teachers were using the product so they could determine what would be helpful to study in a trial, rather than diving right into a study.
Culatta said that RCTs often fail to account for context or rapid product iteration, particularly with tools that rely on AI.
“They are intentionally designed to measure something that is inherently stable,” he said. “A lot of those apps are just prompts sitting on top of an LLM, and so any time one of the LLMs updates, bam, all these new features. So, whatever you were studying eight months ago when you started this study doesn't even matter.”
Panelists said ed-tech companies should build tools that show how students and teachers are using them, not just whether they’re being used at all. By collecting that kind of data in real time, they said, developers can know and show what’s working instead of waiting months or years for formal studies that may not reflect real classroom conditions.
COLLABORATIVE DESIGN
Similarly, instead of relying on RCTs or other measures to assess quality after a product is created, panelists recommended seeing quality as the result of a process that involves working with classroom teachers throughout design. At Khan Academy, DiCerbo said, she spends time in classrooms in New Jersey and New York observing how tools are used and fine-tunes them based on what she sees.
Culatta encouraged companies to stop designing for “generic users” and instead focus on specific individuals with real classroom experiences. If companies can’t call on specific users to inform their decisions, he recommended shadowing the intended audience. At ISTE+ASCD, he said, staff spend one day a year shadowing teachers.
“Most of the people on my team worked in education for years,” he said, but they still find the experience valuable. “Stuff changes so fast.”
DiCerbo said working with teachers helps vendors in the long run from a business standpoint, because the product is better and the experience builds relationships.
“When educators find what they want, they will tell other people,” Livingston said.
