Reliability of Difference Scores Obtained From Nested Data Within a Multivariate Generalizability Theory Framework

Rabia Karatoprak Ersen; Won-Chan Lee; Donald B. Yarbrough

doi:10.1177/00131644261451746

Back

Reliability of Difference Scores Obtained From Nested Data Within a Multivariate Generalizability Theory Framework

Journal article

Open access

Peer reviewed

Reliability of Difference Scores Obtained From Nested Data Within a Multivariate Generalizability Theory Framework

Rabia Karatoprak Ersen, Won-Chan Lee and Donald B. Yarbrough

Educational and psychological measurement

07/07/2026

DOI: 10.1177/00131644261451746

PMID: 42428570

Files and links (1)

url

https://doi.org/10.1177/00131644261451746View

Published (Version of record) Open Access

Abstract

The purpose of this study is to examine the reliability and dependability of difference scores computed as the change between a pretest and a posttest administered to assess the effectiveness of an intervention. The data-collection design involved a nested structure, with persons (p) nested within groups (g), and groups nested within sites (s). Multivariate generalizability theory was employed to estimate the reliability and dependability of difference scores at the levels of persons, groups, and sites. The G study designs included (p•:g•:s•)xi• , (p•:s•)xi• , (p•:g•)xi• , g•xi• , p•xi• , and s•xi• , with pretest and posttest serving as the two levels of the multivariate facet. In the designs with sites as the object of measurement, (P•:G•:s•)xI• , (P•:s•)xI• , and s•xI• , omitting groups within sites or persons within groups led to an underestimation of error variances and inflated generalizability and dependability coefficients. The relative error correlations increased, and the absolute error and universe score correlations decreased across s•xI• , (P•:s•)xI• , and (P•:G•:s•)xI• . Across all designs, generalizability and dependability coefficients were similar in magnitude, primarily due to the relatively small variance in items. Compared across different objects of measurement, both the generalizability and dependability coefficients were highest when the object of measurement was persons, and lowest when it was sites.

Original

Details

Title: Subtitle: Reliability of Difference Scores Obtained From Nested Data Within a Multivariate Generalizability Theory Framework
Creators: Rabia Karatoprak Ersen - GESIS—Leibniz Institute for the Social Sciences, Cologne, Germany The University of Iowa, Iowa City, USA
Won-Chan Lee - University of Iowa
Donald B. Yarbrough - University of Iowa
Resource Type: Journal article
Publication Details: Educational and psychological measurement
DOI: 10.1177/00131644261451746
PMID: 42428570
NLM abbreviation: Educ Psychol Meas
ISSN: 0013-1644
eISSN: 1552-3888
Publisher: SAGE Publications
Grant note: The authors received no financial support for the research, authorship, and/or publication of this article.
Language: English
Electronic publication date: 07/07/2026
Academic Unit: Psychological and Quantitative Foundations
Record Identifier: 9985180941602771

Metrics

1 Record Views