Conference paper
Scoring with Nominal Missing-Response Parameters: Its Effects on Achievement Levels Set on the National Assessment of Educational Progress (NAEP)
Distributed by ERIC Clearinghouse
AERA Annual
04/22/1995
Abstract
This study examines the effects on the National Assessment of Educational Progress (NAEP) achievement levels of using item response theory (IRT) models that have nominal missing-response parameters. It compares cutpoints based on item parameters that were fitted using two different models. The first set of cutpoints were based on parameters for the two- and three-parameter logistic model, and the second set of cutpoints were based on R. Bock's (1972) nominal model. Data are from the 1992 NAEP in mathematics and reading for grade 12. For reading, data included 1,966 responses to a block of items, and for mathematics, data included 2,192 responses. Other data were the item-by-item ratings by each panelist who participated in the Achievement Levels Setting process for NAEP 1992. For each subject, the cutpoints set using the probability curves obtained by the different models were compared. The percent of students scoring at each level were also compared. For reading, the logistic model, when fitted to the data, converged in 25 iterations and yielded a marginal reliability of 0.67 with maximum information of 4.3 at theta equals -0.5. The nominal model converged in 88 iterations and yielded a marginal reliability of 0.85 with a maximum information of 14.2 at theta equals 0. For mathematics, the logistic model converged in 184 iterations and yielded a marginal reliability of 0.61 with maximum information of 23.9 at theta equals 1.5. The nominal model, with mathematics data, converged in 46 iterations, and yielded a marginal reliability of 0.62, with maximum information of 7.3 at theta equals -0.2. When the percentages of students scoring at or above each cutpoint were compared, none scored at the Advanced level using the nominal model. These preliminary results suggest the direction of future studies, but cannot be generalized to the NAEP assessment program. An appendix contains achievement levels descriptions. (Contains three tables, six figures, and five references.) (SLD)
Details
- Title: Subtitle
- Scoring with Nominal Missing-Response Parameters: Its Effects on Achievement Levels Set on the National Assessment of Educational Progress (NAEP)
- Creators
- Luz BaySusan Cooper LoomisTianyou Wang
- Resource Type
- Conference paper
- Conference
- AERA Annual
- Publisher
- Distributed by ERIC Clearinghouse
- Number of pages
- 43 pages
- Comment
- Paper presented at the annual meeting of the American Educational Research Association 22 April 1995.
- Language
- English
- Date published
- 04/22/1995
- Academic Unit
- Center for Advanced Studies in Measurement and Assessment
- Record Identifier
- 9984631938602771
Metrics
7 Record Views