Dissertation
Text-to-speech technology as a designated support on statewide reading assessment: impact on item performance and item functioning
University of Iowa
Doctor of Philosophy (PhD), University of Iowa
Spring 2025
DOI: 10.25820/etd.007947
Abstract
The purpose of this study was to examine the effect of access to text-to-speech (TTS) technology on a statewide assessment of reading comprehension at the item level. The Iowa Statewide Assessment of Student Progress (ISASP) serves as the yearly test of student achievement for the state of Iowa, fulfilling the federal requirement that each state administer such an assessment for the purpose of accountability. The ISASP is a computer-based test, and various tools are built into the assessment platform. One such tool, the TTS tool, is allowed beginning in grade 6 as a designated feature, meaning that school districts make the decision to grant access to each of their students. By examining the impact of access to this tool at the item level, this study addressed concerns about a threat to construct validity that may arise from the use of such a tool on a reading comprehension assessment.
To understand how access to the TTS tool impacted the reading comprehension assessment, data from the Spring 2024 administration of the ISASP for grades 6, 7, and 8 was utilized. First, a matching procedure was followed to create a sample of students with and without access to TTS that were comparable on English Language Learner Status, Individual Education Plan status, and the prior year’s reading scores. Using this matched sample, dichotomously scored multiple-choice items from the grades 6, 7, and 8 reading assessment were analyzed for differential item functioning (DIF). To complete the DIF analysis, the Mantel Haenszel (MH) and Logistic Regression (LR) procedures were employed to detect both uniform and nonuniform DIF that might be present.
In total, 162 items were examined, with 52 items from the grade 6 assessment, 54 from the grade 7 assessment, and 56 from the grade 8 assessment. Results from comparing the overall proportion of students answering each item correctly from the two groups showed that there was a statistically significant difference in proportion for 17 items, with only 4 of these items having an effect size greater than .2 or the threshold for a small effect size for Cohen’s h. Results from the MH procedure showed only two items, both from the grade 8 assessment having negligible DIF and only one item, from the grade 7 assessment, having substantial DIF. The LR procedure showed 13 items with nonuniform DIF, with seven items from the grade 6 assessment, five from the grade 7 assessment, and one from the grade 8 assessment. Taking the results from the two DIF detection methods together, 8.7 percent of the 162 items examined showed evidence of DIF.
Items exhibiting DIF were examined further, and no patterns were found in the items' language, structure, or domain. The small number of items exhibiting DIF and the lack of patterns in item characteristics among those items led to the conclusion that having access to TTS did not change the construct of the reading assessment. These findings indicate that providing some students access to the TTS tool did not threaten the assessment's validity. These findings provide validity evidence for the assessment and the decision to have the TTS tool as a designated feature rather than an accommodation. In addition, this study serves as an example of how DIF analysis can be used to gather validity evidence under different administrative conditions, particularly those related to tools built into computer-based assessments.
Details
- Title: Subtitle
- Text-to-speech technology as a designated support on statewide reading assessment: impact on item performance and item functioning
- Creators
- Annette Vernon
- Contributors
- Catherine Welch (Advisor)Stephen Dunbar (Committee Member)David Cassels Johnson (Committee Member)Anthony Fina (Committee Member)
- Resource Type
- Dissertation
- Degree Awarded
- Doctor of Philosophy (PhD), University of Iowa
- Degree in
- Psychological and Quantitative Foundations (Educational Measurement and Statistics)
- Date degree season
- Spring 2025
- DOI
- 10.25820/etd.007947
- Publisher
- University of Iowa
- Number of pages
- xiv, 140 pages
- Copyright
- Copyright 2025 Annette Vernon
- Language
- English
- Date submitted
- 03/14/2025
- Description illustrations
- Illustrations, tables, graphs, charts
- Description bibliographic
- Includes bibliographical references (pages 84-93).
- Public Abstract (ETD)
The use of computer-based tests for statewide accountability assessments has allowed for tools to be built into assessment platforms to serve as aids for students completing the assessment. These tools can be granted to all or some students depending on the tool's status as a universal feature, designated feature, or accommodation. This study examines the use of one such tool, the text-to-speech (TTS) tool, and its impact on items from a middle school reading assessment. The TTS tool was classified as a designated feature, meaning districts decide which students are granted access to the tool. This study examined data from the Spring 2024 administration of the Iowa Assessment of Student Progress (ISASP), using a matching procedure to create a sample of students with and without access to the TTS tool who were comparable on their reading scores from the prior year, English Language Learner status, and Individual Education Plan status. Methods to detect differential item functioning (DIF) were employed, which included the Mantel Haenszel procedure and the Logistic Regression Procedure. Results indicated that only 8.7 percent of the items examined from the reading assessment for the three middle school grades showed evidence of DIF. Findings suggest that allowing school districts to decide which students have access to the TTS tool does not threaten the validity of the reading assessment, and scores remain comparable regardless of whether access to TTS was granted.
- Academic Unit
- Psychological and Quantitative Foundations
- Record Identifier
- 9984831023702771
Metrics
2 File views/ downloads
2 Record Views