The development and use of adapted criterion-referenced reliability indices for composite measures with subscale-level cut scores

In educational and psychological assessments, decisions made from composite measures with embedded subscale scores are frequently based on composite-level criteria only. However, research indicates that it is of great importance to understand subscale score patterns within composites and the extent to which scores as referenced to subscale benchmarks coincide across replications.

Existing criterion-referenced reliability indices fall within two major categories: those evaluating the consistency of overall deviations of observed scores from a cut score, and those evaluating the consistency of examinee classification into the same category across replications of the assessment procedure. These indices are either intended for cut scores for individual embedded subscales or strictly for a composite formed from subscale scores. In this study, adapted indices were derived based on existing ones within each category, specifically, (a) Livingston’s and 𝜙(𝜆) coefficients, and (b) classification consistency and accuracy.

Both original and adapted indices were applied to a personality measure (Conscientiousness composite and facet subscale scores from the BFI-2) and compared in magnitude and patterns of change at different cut scores. Results revealed that both original and adapted Livingston’s and 𝜙(𝜆) coefficients were likewise affected by cut score locations in terms of their distances from the mean, with higher values associated with wider distances. Adapted classification consistency and accuracy were more sensitive to score density at the cut score locations. Adapted classification indices specific to subscale-level cut scores exhibited differences from original indices mainly due to their distinctive definitions of “satisfactory” performance within a composite measure and the different combinations of subscale scores considered satisfactory or non-satisfactory.