Dissertation
Item type and survey mode comparability: an analysis of measurement invariance between item response types and survey modes
University of Iowa
Doctor of Philosophy (PhD), University of Iowa
Spring 2023
DOI: 10.25820/etd.007202
Abstract
Prior research highlights the benefits of multimode surveys and best practices for item-by-item (IBI) and matrix-type survey items. Some researchers have explored whether mode differences for online and paper surveys persist for these survey item types. However, no studies discuss measurement invariance when both item types and online modes are used in the same study. This study uses a two-way repeated-measures design to explore whether matrix and IBI-type items can be compared across online and paper survey modes. Personality data for extraversion and conscientiousness traits yielded from four survey conditions are used in this study: 1) paper mode, IBI-type items, 2) paper mode, matrix-type items, 3) online mode, IBI-type items, and 4) online mode, matrix-type items.
A 2-parameter graded response model (GRM) using marginal maximum likelihood (MML) with the Gauss-Hermite quadrature rule for integral approximations was used to estimate item parameters. Analysis of item characteristic curves (ICCs) and item information curves (IICs) showed small differences in a_i and b_i parameters where the IBI paper format yielded a higher probability of endorsing more extreme responses at lower levels of conscientiousness compared to online matrix-type items. However, study results showed high internal consistency as measured by coefficient alpha values, no evidence of straightlining for any condition, and no evidence of differential item functioning (DIF) explored with an ordinal logistic regression for any items across conditions. These findings provide evidence of measurement invariance for comparisons of 5-point Likert-type, IBI and matrix-type items shown in groups of five across online and paper modes.
Details
- Title: Subtitle
- Item type and survey mode comparability: an analysis of measurement invariance between item response types and survey modes
- Creators
- Kayla Lynn Jackson
- Contributors
- Brandon LeBeau (Advisor)Katharine Broton (Committee Member)Liz Hollingworth (Committee Member)Jonathan Templin (Committee Member)
- Resource Type
- Dissertation
- Degree Awarded
- Doctor of Philosophy (PhD), University of Iowa
- Degree in
- Psychological and Quantitative Foundations (Educational Measurement and Statistics)
- Date degree season
- Spring 2023
- Publisher
- University of Iowa
- DOI
- 10.25820/etd.007202
- Number of pages
- viii, 83 pages
- Copyright
- Copyright 2023 Kayla Jackson
- Language
- English
- Date submitted
- 04/25/2023
- Date approved
- 05/01/2023
- Description illustrations
- Illustrations, tables, graphs, charts
- Description bibliographic
- Includes bibliographical references (pages 77-83).
- Public Abstract (ETD)
Two types of items are item-by-item (IBI) where items are presented individually with their own response options, or matrix items, where questions are presented in a grid with response options appearing once at the top of the grid and statements appearing in the left margin of the grid. Presently, no studies have explored whether people respond in the same way for IBI items presented in an online survey versus matrix items in a paper survey, and vice versa.
In this study, two groups of respondents were randomly assigned to different versions of a personality survey. Specifically, participants were randomly assigned to take either the paper survey with IBI questions (condition 1) or the paper survey with matrix questions (condition 2). A week later, those who responded to IBI questions completed the online survey with matrix questions (condition 3), and those who responded to matrix questions completed the online survey with IBI questions (condition 4).
An analysis of responses showed slight differences between survey conditions. For IBI questions on paper, the IBI respondents were more likely to give extreme responses, like agree strongly or disagree strongly, at lower levels of a personality trait than matrix questions asked online. However, there was no significant evidence that people respond differently to questions asked in an IBI format or matrix format across surveys administered on paper or online.
- Academic Unit
- Psychological and Quantitative Foundations
- Record Identifier
- 9984437258302771
Metrics
12 Record Views