Dissertation
Applications of item difficulty modeling in statewide assessment
University of Iowa
Doctor of Philosophy (PhD), University of Iowa
Spring 2026
DOI: 10.25820/etd.008347
Abstract
This dissertation operationalizes an item difficulty modeling (IDM) framework for the Iowa Statewide Assessment of Student Progress (ISASP), using supervised machine learning to predict item difficulty (IRT b-parameter) and discrimination (IRT a-parameter) for Mathematics, Reading, and Science items at Grades 5, 8, and 10. Seven models were evaluated: Multiple Linear Regression, Ridge Regression, Lasso Regression, Elastic Net, Classification and Regression Trees, Random Forest, and XGBoost. Models were trained on item-level linguistic complexity measures, readability indices, structural metadata, content domain indicators, cognitive complexity classifications, and empirical item difficulty, and compared using test-set RMSE as the primary evaluation criterion.
No single model dominated across all subject-grade-parameter combinations. For item difficulty, regularized linear models, particularly Lasso and Elastic Net, achieved the lowest prediction error for Reading and Science, while XGBoost performed best for Mathematics difficulty at Grades 5 and 8. Overall predictive accuracy was modest, with the best difficulty models achieving test-set $R^2$ values of 0.21 to 0.36 for Mathematics and below 0.20 for Reading and Science. Discrimination prediction was substantially weaker, with $R^2$ values near zero in several conditions across all three subjects.
Variable importance analyses revealed distinct predictor profiles for difficulty and discrimination. For Mathematics difficulty, Depth of Knowledge Level 3 was the single most important predictor across all grade levels, followed by stem-level linguistic complexity features. For Reading difficulty, passage-level readability measures, particularly the SMOG score, dominated at lower grades, with a broader feature set contributing at Grade 10. For Science difficulty, content domain membership and the presence of visual stimuli were the primary predictors. Across all subjects, empirical item difficulty emerged as the dominant predictor of discrimination, with linguistic and structural features contributing negligible additional signal.
These findings have direct implications for three IDM applications. Mathematics difficulty models meet the minimum $R^2$ threshold identified by prior simulation research for use as weakly informative priors in Bayesian IRT calibration, suggesting practical utility for reducing field-testing sample requirements. Models for Reading and Science do not yet meet thresholds for item pretesting replacement or prescreening. Incorporating cognitive theory-driven features and LLM-based feature extraction represent the most promising directions for improving IDM accuracy in future work.
Details
- Title: Subtitle
- Applications of item difficulty modeling in statewide assessment
- Creators
- Mubarak O. Mojoyinola
- Contributors
- Catherine Welch (Advisor)Stephen Dunbar (Committee Member)Anthony Fina (Committee Member)Sanvesh Srivastava (Committee Member)
- Resource Type
- Dissertation
- Degree Awarded
- Doctor of Philosophy (PhD), University of Iowa
- Degree in
- Psychological and Quantitative Foundations (Educational Measurement and Statistics)
- Date degree season
- Spring 2026
- DOI
- 10.25820/etd.008347
- Publisher
- University of Iowa
- Number of pages
- xiii, 116 pages
- Copyright
- Copyright 2026 Mubarak O. Mojoyinola
- Language
- English
- Date submitted
- 04/27/2026
- Description illustrations
- tables, graphs
- Description bibliographic
- Includes bibliographical references (pages 106-116).
- Public Abstract (ETD)
When students take standardized tests, every question on the test has been carefully analyzed to understand how difficult it is and how well it can distinguish between students who know the material and those who do not. Figuring out these properties traditionally requires giving the questions to large numbers of students in field tests before the questions can be used operationally, a process that is time-consuming, expensive, and sometimes risky if questions are exposed. This dissertation investigates whether it is possible to predict these properties directly from the characteristics of the questions themselves, such as how complex the language is, what content area the question covers, and how cognitively demanding the question is.
Using questions from the Iowa Statewide Assessment of Student Progress (ISASP) in Mathematics, Reading, and Science across Grades 5, 8, and 10, this study trained and compared seven different machine learning models to see which could most accurately predict question difficulty and question discrimination from item features alone. The results showed that no single model worked best in every situation. For Mathematics, a model called XGBoost worked best, and the most useful feature was the cognitive complexity level assigned to the question. For Reading, simpler statistical models worked best, and the reading level of the passage accompanying the question was the most important feature. For Science, the content domain and whether the question included a visual like a chart or diagram mattered most.
Predicting how well a question discriminates between high and low performers turned out to be much harder than predicting difficulty. In most cases, the only reliable predictor of discrimination was the difficulty of the question itself, not any of the text or content features.
The practical takeaway is that the Mathematics models developed in this study are accurate enough to potentially reduce the number of students needed for field testing, which could make test development faster and less costly. Models for Reading and Science need further development before they can be used in operational settings.
- Academic Unit
- Psychological and Quantitative Foundations
- Record Identifier
- 9985177174102771
Metrics
1 Record Views