Using Structural Equation Modeling to Reproduce and Extend ANOVA-Based Generalizability Theory Analyses for Psychological Assessments

Walter P. Vispoel; Hyeryung Lee; Tingting Chen; Hyeri Hong

doi:10.3390/psych5020019

Back

Using Structural Equation Modeling to Reproduce and Extend ANOVA-Based Generalizability Theory Analyses for Psychological Assessments

Journal article

Open access

Peer reviewed

Using Structural Equation Modeling to Reproduce and Extend ANOVA-Based Generalizability Theory Analyses for Psychological Assessments

Walter P. Vispoel, Hyeryung Lee, Tingting Chen and Hyeri Hong

Psych, Vol.5(2), pp.249-273

04/13/2023

DOI: 10.3390/psych5020019

Files and links (1)

url

https://doi.org/10.3390/psych5020019View

Published (Version of record) Open Access

Abstract

Generalizability theory provides a comprehensive framework for determining how multiple sources of measurement error affect scores from psychological assessments and using that information to improve those assessments. Although generalizability theory designs have traditionally been analyzed using analyses of variance (ANOVA) procedures, the same analyses can be replicated and extended using structural equation models. We collected multi-occasion data from inventories measuring numerous dimensions of personality, self-concept, and socially desirable responding to compare variance components, generalizability coefficients, dependability coefficients, and proportions of universe score and measurement error variance using structural equation modeling versus ANOVA techniques. We further applied structural equation modeling techniques to continuous latent response variable metrics and derived Monte Carlo-based confidence intervals for those indices on both observed score and continuous latent response variable metrics. Results for observed scores estimated using structural equation modeling and ANOVA procedures seldom varied. Differences in reliability between raw score and continuous latent response variable metrics were much greater for scales with dichotomous responses, thereby highlighting the value of doing analyses on both metrics to evaluate gains that might be achieved by increasing response options. We provide detailed guidelines for applying the demonstrated techniques using structural equation modeling and ANOVA-based statistical software.

Details

Title: Subtitle: Using Structural Equation Modeling to Reproduce and Extend ANOVA-Based Generalizability Theory Analyses for Psychological Assessments
Creators: Walter P. Vispoel - University of Iowa
Hyeryung Lee - University of Iowa
Tingting Chen - University of Iowa
Hyeri Hong - California State University System
Resource Type: Journal article
Publication Details: Psych, Vol.5(2), pp.249-273
DOI: 10.3390/psych5020019
ISSN: 2624-8611
eISSN: 2624-8611
Grant note: name: Iowa Measurement Research Foundation
Language: English
Date published: 04/13/2023
Academic Unit: Psychological and Quantitative Foundations
Record Identifier: 9984408649202771

Metrics

13 Record Views