Working paper
ACT's NAEP Redesign Project: Assessment Design Is the Key to Useful and Stable Assessment Results. Working Paper Series
Working Paper, Vol.No. 97-30
American College Testing Program
10/1997
Abstract
This report presents an investigation by the American College Testing Program (ACT) of an alternative design for the National Assessment of Educational Progress (NAEP). The proposed design greatly simplifies the data collection and analysis procedures needed to produce assessment results and has the potential to produce results that are more timely and easier to interpret. The plan calls for developing individual NAEP forms, where each individual form represents, as closely as possible, the assessment questions from the domain of knowledge being measured by an NAEP construct. Sets of these forms could be administered, in random order, to students in the schools. This would replace the balanced incomplete block design (BIB) currently used. Assessments constructed under the BIB design do not closely represent, at least for the 1996 science assessment, the content framework. Enhanced procedures are also suggested for developing precise content and statistical specifications for individual forms and procedures for pretesting items. The basic scores that ACT suggests using for producing group assessment results are calculated by weighting item scores from multiple-choice and constructed-response items, where the weights are determined, a priori, by content specialists. These weights should relate more closely to the weighting intended by content specialists than do current NAEP weights. Scaling, equating, and score distribution estimation methods are described that rely on less stringent psychometric and statistical assumptions than do current procedures. Issues in sampling that include sample size requirements, sample design, and estimating standard errors are also examined, as are procedures for reporting score distributions that reflect group performance on content domains. The feasibility of using such domain scores to measure trends and to facilitate setting NAEP standards is explored. ACT recommends focusing on the design of assessments and the data collection methods rather than on complex analysis procedures. (Contains 18 tables, 9 figures, and 41 references.) (SLD)
Details
- Title: Subtitle
- ACT's NAEP Redesign Project: Assessment Design Is the Key to Useful and Stable Assessment Results. Working Paper Series
- Creators
- Luz BayLee ChenBradley A HansonJay HappelMichael J KolenTimothy MillerMary PommerichJames SconingTianyou WangCatherine Welch
- Resource Type
- Working paper
- Publication Details
- Working Paper, Vol.No. 97-30
- Publisher
- American College Testing Program
- Number of pages
- 122 pages
- Language
- English
- Date published
- 10/1997
- Academic Unit
- Center for Advanced Studies in Measurement and Assessment; Psychological and Quantitative Foundations
- Record Identifier
- 9984631939802771
Metrics
7 Record Views