Comparison of DIF methods for the Student Experience in the Research University survey: a validity and methodological study

Student experience surveys are used to measure student attitudes as well as to initiate conversations for institutional change. Validity evidence to support the interpretations of these surveys’ results, however, is lacking. The purpose of this study was to compare Differential Item Functioning (DIF) methods on Likert-scaled survey data as one facet of validity evidence for the Student Experiences in the Research University (SERU) survey. Specifically, DIF analyses were performed to provide evidence for the comparison of survey scores across administrations (years), and between student demographic groups (race, gender, first-generation student status). In addition, new methods of DIF analysis based on aggregation techniques were applied to both SERU survey data and simulated survey data and compared to a more commonly used DIF method.

This study provides SERU survey developers with information on the validity evidence supporting certain subgroup score comparisons as well as information on potentially biased items that may need to be reassessed. Furthermore, both SERU survey developers and developers of similar student experience surveys can use this dissertation as a template for collecting internal structure validity evidence by way of DIF. Lastly, education and psychometrics researchers can use the results of this study to assess the behavior of DIF aggregation methods on survey data.