Opinion: Governments Need to Come Together on Ed-Tech Evaluation

Governments worldwide face a critical challenge in selecting the best educational technology tools from an overwhelming array of options. On one hand, supporting the ed-tech industry is crucial for fostering innovation, creating jobs and enhancing digital literacy from a young age. On the other, there is an urgent need to protect children from potential harm, such as the misuse of their data and the distraction from valuable instructional time. As teachers grapple with an overabundance of ed-tech tools, policymakers must find a way to discern what truly works in education while safeguarding student well-being.

MULTIPLE QUALITY EVALUATION CRITERIA

Policymakers are asking two essential questions: What constitutes quality in ed tech, and how can this quality be ensured and identified? They tend to mention multiple requirements: ed-tech tools need to be pedagogically sound, interoperable within a school's technology ecosystem, protective and responsible with children's data, and provide added educational value. Additionally, these tools need evidence from research studies verifying their effectiveness. They should also be inclusive, bring marginalized groups to the center, and have a strong foundation in learner-centered research. A 2023 report commissioned by the UK's Department for Education found that there are 75 different evaluation frameworks developed for ed tech in K-12 alone.

Amidst discussions of certifications, evaluation frameworks, standards and regulatory structures, active efforts are ongoing to identify common frameworks and guidance for ed-tech developers. For example, the European Union is working on a common evaluation framework for digital content, and the U.S. Department of Education in July set out rules for ed-tech companies in their guide Designing for Education with Artificial Intelligence.

Each of them examines various domains of impact that ed tech can have on learners, which we summarize as the so-called 5Es of ed-tech impact: efficacy, effectiveness, ethics, equity and environment. As shown in Table 1 below, the most common frameworks meet the five Es to different extents, depending on their specific focuses.

Emerging evaluation frameworks developed collaboratively by various groups, such as the UNICEF EdTech4Good Curation Framework, examines various domains of impact that ed tech can have on learners, which we summarize as the 5Es of ed-tech impact: efficacy (Does the solution work?), effectiveness (Could the solution work?), ethics (How does the solution work?), equity (Who does the solution work for?) and environment (Is the solution sustainable?).

• Efficacy involves controlled experiments, prioritizing randomized controlled trials to ensure features of the technology are consistent with published research.
  
• Effectiveness pertains to cost-effectiveness and real-world classroom studies, incorporating teachers’ direct input, aligning the tools with pedagogy and curriculum standards, and making sure their results can be replicated in different contexts.
  
• Ethics encompasses data ethics, safe and interoperable data handling, and responsible data storage according to rules and regulations.
  
• Equity focuses on inclusive and participatory design, ensuring marginalized groups are included and that universal design principles are respected.
  
• Environment addresses the local environment and sustainable practices (e.g., using resources and preserving the natural environment) of the ed-tech organization, as well as the broader contribution to planetary health and eco-friendly policies.

Guides and frameworks can only offer recommendations, leaving schools and procurement teams with the challenging task of deciding for themselves which companies adhere to these recommendations and to what extent. This is partly why the discussion around what constitutes quality in ed tech also spurred innovation in quality certifications: In 2024, numerous ed-tech organizations such as Digital Promise, EdTech Impact and LeanLab either launched or re-launched certifications to clarify which companies meet specific guidelines or standards, although various certifications defined those differently. Aligning these different certifications and establishing a system of equivalence among them is crucial for helping teachers navigate the saturated ed-tech market, and for the ed-tech providers who need to comply with different certification requirements in different countries.

INTERNATIONAL CERTIFICATION OF ED-TECH QUALITY

In the U.S., to facilitate teachers’ choice of ed-tech tools, seven certification providers — ISTE, 1EdTech, CAST, CoSN, Digital Promise, InnovateEDU and SETDA — joined forces in July to collate different certification standards onto a joint platform.

On the international level, the nonprofit Eduevidence.org has developed a research-driven approach to ed-tech evaluation that combines various certifications, both U.S.-based and international. The system employs a weight-of-evidence method, featuring a tiered structure with Bronze, Silver, and Gold certifications. A tiered system, whether through different levels of certifications or adding up points, allows decision-makers to easily examine the evidence supporting ed-tech companies. Eduevidence.org's system can give users — whether teachers or government officials, depending on the autonomy of the education system — a clear understanding of how these standards relate to each other and to the different requirements of quality.

National governments need to supplement a tiered system of quality goals with a minimal quality requirement, a red line grounded in regulatory or legal frameworks. This ensures that only compliant tools are evaluated for their quality, providing a foundational safeguard before higher levels of certification are considered.

RED LINES

The availability of generative AI to ed-tech developers and the keen interest from ed-tech VCs in AI-driven tools have revived crucial questions: What are the red lines these tools should not cross, ensuring they are verified and checked for quality before reaching classrooms? A UNESCO GEM report highlighted these concerns last year, prompting many governments to realize that unchecked ed tech can not only fail to add value to learning, but actively harm it. In response, some governments took drastic measures, such as banning smartphones (France) and questioning the value of digital books (Sweden) or Chromebooks (Denmark). However, researchers in Queensland found in March 2024 that mobile phone bans were not a viable solution and may inadvertently remove potentially beneficial content.

The alternative is a nuanced approach, allowing school leaders to make choices based on objectively verifiable quality criteria. This is where a database of recommended tools and national catalogues becomes essential, allowing teachers to see how tools handle data, support children with special needs and make personalized learning more efficient.

There are various ways for policymakers or education leaders to set up repositories or lists of recommended resources. Competitions, where judges use specific criteria, and requests for submissions are popular methods. Philanthropic organizations in education increasingly use these approaches, focusing on how ed-tech tools can address entrenched issues like learning disparities and access to educational content in underserved areas. For instance, the Tools Competition has identified 50 winners who address these pressing challenges and adhere to learning sciences, effectively creating a de facto catalogue of recommended solutions.

The Gates Foundation is another major player, supporting AI-enhanced tools for teaching and learning mathematics in K-12. Additionally, UNICEF's Global Learning Innovation Hub, through their Blue Unicorn program and the EdTech4Good Curation Framework, is working to identify and promote the best ed-tech tools worldwide. There are also various academic summaries, reports, and reviews of ed-tech certifications available online.

QUALITY ASSURANCE AND ONGOING IMPROVEMENTS ARE KEY

Thus, policymakers must simultaneously pursue two objectives when evaluating ed-tech quality and implementing systems for doing so: identifying what ed tech should avoid, and highlighting what ed tech could achieve. An effective ed-tech ecosystem integrates quality assurance with continuous improvement, featuring certifications alongside consolidated benchmarks and regulatory frameworks. Government orchestration is essential. Despite ongoing efforts to unify the current patchwork of approaches, the challenge remains for governments to achieve full international coherence in the evaluation and implementation of ed-tech systems. A successful ed-tech ecosystem combines quality assurance with ongoing improvements, using certifications, benchmarks and regulatory frameworks. Government leadership is crucial in this process. Although efforts to unify the various existing approaches are ongoing, the main challenge for governments is to achieve international consistency in identifying and implementing high-quality ed-tech systems.


Natalia Kucirkova is a professor of Early Childhood Education and Development at the University of Stavanger in Norway.

Per Henning Uppstad is an assistant professor at the University of Stavanger in Norway.

Richard Holeton is a writer, education consultant and assistant vice provost for learning environments, emeritus, at Stanford University in California.

Lin Dan is an associate professor in the Department of Psychology at The Education University of Hong Kong in China.

All authors are voluntary advisory board members for the International Certification of Evidence of Impact in Education (Eduevidence.org), an international research effort to align different ways of documenting evidence and impact in education.