Logo image
Back
Assessing AI capabilities with education tests
Book chapter   Open access

Assessing AI capabilities with education tests

Mila Staneva, Abel Baret, Ángel Aso-Mollar, Joseph Blass, Salvador Carrión Ponz, Vincent Conitzer, Ulises Cortes, Pradeep Dasigi, Angel de Paula, Carlos Galindo, …
AI and the Future of Skills, Volume 2, pp.40-64
Educational Research and Innovation, OECD Publishing
11/16/2023
DOI: 10.1787/bbdeb1e0-en
url
https://doi.org/10.1787/bbdeb1e0-enView
Published (Version of record)See pages 40-64 Open Access

Abstract

This chapter introduces three exploratory studies that assessed the capabilities of artificial intelligence (AI) through standardised education tests designed for humans. The first two studies, conducted in 2016 and 2021/22, asked experts to evaluate AI’s performance on the literacy and numeracy tests of the OECD’s Survey of Adult Skills (PIAAC). The third study collected expert judgements of whether AI can solve science questions from the OECD's Programme for International Student Assessment (PISA). The studies aimed to refine the assessment framework for eliciting expert knowledge on AI using established educational assessments. They explored different test formats, response methodologies and rating instructions, along with two distinct assessment approaches. A “behavioural approach” used in the PIAAC studies emphasised smaller expert groups engaging in discussions, and a "mathematical approach" adopted in the PISA study relied more heavily on quantitative data from a larger expert pool. This chapter presents the results of the studies and discusses the advantages and disadvantages of their methodological approaches.

Details

Metrics

3 Record Views
Logo image