Extended Multivariate Generalizability Theory With Complex Design Structures

Robert L. Brennan; Stella Y. Kim; Won-Chan Lee

doi:10.1177/00131644211049746

Back

Extended Multivariate Generalizability Theory With Complex Design Structures

Journal article

Open access

Peer reviewed

Extended Multivariate Generalizability Theory With Complex Design Structures

Robert L. Brennan, Stella Y. Kim and Won-Chan Lee

Educational and psychological measurement, Vol.82(4), pp.617-642

08/2022

DOI: 10.1177/00131644211049746

PMCID: PMC9228696

Files and links (1)

url

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9228696View

Published (Version of record) Open Access

Abstract

This article extends multivariate generalizability theory (MGT) to tests with different random-effects designs for each level of a fixed facet. There are numerous situations in which the design of a test and the resulting data structure are not definable by a single design. One example is mixed-format tests that are composed of multiple-choice and free-response items, with the latter involving variability attributable to both items and raters. In this case, two distinct designs are needed to fully characterize the design and capture potential sources of error associated with each item format. Another example involves tests containing both testlets and one or more stand-alone sets of items. Testlet effects need to be taken into account for the testlet-based items, but not the stand-alone sets of items. This article presents an extension of MGT that faithfully models such complex test designs, along with two real-data examples. Among other things, these examples illustrate that estimates of error variance, error-tolerance ratios, and reliability-like coefficients can be biased if there is a mismatch between the user-specified universe of generalization and the complex nature of the test.

Mathematics

Mathematics, Interdisciplinary Applications

Physical Sciences

Psychology

Psychology, Educational

Psychology, Mathematical

Science & Technology

Social Sciences

Details

Title: Subtitle: Extended Multivariate Generalizability Theory With Complex Design Structures
Creators: Robert L. Brennan - University of Iowa
Stella Y. Kim - Univ N Carolina, Charlotte, NC 28223 USA
Won-Chan Lee - University of Iowa
Resource Type: Journal article
Publication Details: Educational and psychological measurement, Vol.82(4), pp.617-642
Publisher: SAGE Publications
DOI: 10.1177/00131644211049746
PMCID: PMC9228696
ISSN: 0013-1644
eISSN: 1552-3888
Number of pages: 26
Language: English
Date published: 08/2022
Academic Unit: Psychological and Quantitative Foundations
Record Identifier: 9984371295102771

Metrics

1 Record Views

1 Times Cited - Web of Science