Dissertation
Neural network methods for application in educational measurement
University of Iowa
Doctor of Philosophy (PhD), University of Iowa
Summer 2021
DOI: 10.17077/etd.005860
Abstract
In educational measurement, Item Response Theory (IRT) provides a means of quantifying student knowledge. Specifically, IRT models the probability of a student answering a particular item correctly as a function of the student’s continuous-valued latent abilities Θ (e.g. add, subtract, multiply, divide) and parameters associated with the item (e.g. difficulty). Given a group of students’ binary responses (correct/incorrect) to an assessment, parameter estimation techniques are used to infer the students’ abilities Θ to better evaluate student performance. But as the number of students, items, and dimension of Θ increases, traditional parameter estimation methods which rely on numerical integration or MCMC techniques become infeasible.
In this thesis, a novel modification to a variational autoencoder (VAE), an unsupervised learning method, is presented which incorporates multiple pieces of domain knowledge into the VAE architecture. Specifically, an expert-annotated Q-matrix detailing the association between items and abilities is used to constrain the trainable weights in the VAE decoder. This directly links the generative posterior distribution of a VAE to IRT models, and allows interpretation of trainable weights as item parameters and a hidden neural layer as student ability estimates. The use of a neural network in this application allows for efficient estimation of parameters in high-dimensional datasets.
An additional alteration to the VAE encoder allows for modeling of correlated, rather than independent, latent abilities. This architecture is also of interest to areas outside of education where the probability distribution of the VAE latent code is known. The proposed method, titled ML2P-VAE, achieves accuracy similar to traditional parameter estimation methods on smaller datasets and provides quality estimates on educational assessments with a large number of students, items, and latent abilities, where traditional methods struggle.
Modifications used in ML2P-VAE are extended to integrate IRT into deep knowledge tracing models, which use time-dependent neural networks such as long short-term memory networks and Transformers. In an online environment where many items are available, the task of knowledge tracing is to track a student’s learning dynamically as they progress through an assessment. This task has received more attention recently due to the emergence of online learning and AI tutoring systems, where following student’s knowledge acquisition can help facilitate automated curriculum in real time. The inclusion of IRT in the knowledge tracing framework presents a trade-off between explainability and accuracy, while remaining competitive with state-of-the-art methods.
Details
- Title: Subtitle
- Neural network methods for application in educational measurement
- Creators
- Geoffrey Converse
- Contributors
- Suely Oliveira (Advisor)Mariana Curi (Committee Member)David Stewart (Committee Member)Jonathan Templin (Committee Member)Colleen Mitchell (Committee Member)
- Resource Type
- Dissertation
- Degree Awarded
- Doctor of Philosophy (PhD), University of Iowa
- Degree in
- Applied Mathematical and Computational Sciences
- Date degree season
- Summer 2021
- DOI
- 10.17077/etd.005860
- Publisher
- University of Iowa
- Number of pages
- xiii, 146 pages
- Copyright
- Copyright 2021 Geoffrey Converse
- Language
- English
- Description illustrations
- color illustrations
- Description bibliographic
- Includes bibliographical references (pages 138-146).
- Public Abstract (ETD)
In this work, multiple neural network techniques are presented for use in educational measurement applications. In educational measurement, the goal is to quantify student learning based on assessment performance. Item Response Theory (IRT) models the probability of students answering individual questions correctly as a function of the particular student’s set of latent abilities (e.g. add, subtract, multiply, divide), as well as parameters pertaining to the specific question (e.g. difficulty).
A common task is to measure a population of students’ knowledge, given their responses to an assessment. Though many methods exist for this purpose, measurement becomes difficult or infeasible as the number of students, questions, and latent abilities increase. This issue has become more problematic in the age of Big Data, as electronic learning environments increase in functionality and popularity and educational assessments grow in scale and scope.
In this thesis, a novel neural network architecture is presented which estimates question and student parameters from high-dimensional assessment data with ease. The modifications made to existing deep learning methods provide a direct link to IRT and have clear use in applications such as AI tutoring systems. The proposed machine learning architecture is relevant from perspectives outside of education as well, as it introduces interpretability into otherwise black-box deep learning models.
- Academic Unit
- Interdisciplinary Graduate Program in Applied Mathematical & Computational Sciences
- Record Identifier
- 9984124172002771
Metrics
114 File views/ downloads
233 Record Views