Multiple-choice (MCQs) and short-answer questions (SAQs) are the most common test formats for assessing English reading proficiency. While the former provides test-takers with prescribed options, the latter requires short written responses. Test developers favor MCQs over SAQs for the following reasons: less time required for rating, high rater agreement, and wide content coverage. This mixed methods dissertation investigated the impacts of test format on reading performance, metacognitive awareness, test-completion processes, and task perceptions.
Participants were eighty English as a second language (ESL) learners from a Midwestern community college. They were first divided into two groups of approximately equivalent reading proficiencies and then completed MCQ and SAQ English reading tests in different orders. After completing each format, participants filled out a survey about demographic information, strategy use, and perceptions of test formats. They also completed a 5-point Likert-scale survey to assess their degree of metacognitive awareness. At the end, sixteen participants were randomly chosen to engage in retrospective interviews focusing on their strategy use and task perceptions.
This study employed a mixed methods approach in which quantitative and qualitative strands converged to draw an overall meta-inference. For the quantitative strand, descriptive statistics, paired sample t-tests, item analyses, two-way ANOVAs, and correlation analyses were conducted to investigate 1) the differences between MCQ and SAQ test performance and 2) the relationship between test performance and metacognitive awareness. For the qualitative strand, test-takers’ MCQ and SAQ test completion processes and task perceptions were explored using coded interview and survey responses related to strategy use and perceptions of test formats.
Results showed that participants performed differently on MCQ and SAQ reading tests, even though both tests were highly correlated. The paired sample t-tests revealed that participants’ English reading and writing proficiencies might account for the MCQ and SAQ performance disparity. Moreover, there was no positive relationship between reading test performance and the degree of metacognitive awareness generated by the frequency of strategy use. Correlation analyses suggested whether a higher or lower English reading proficiency of the participants was more important than strategy use. Although the frequency of strategy use did not benefit test performance, strategies implemented for MCQ and SAQ tests were found to generate interactive processes allowing participants to gain deeper understanding of the source texts. Furthermore, participants’ perceptions toward MCQs, SAQs, and a combination of both revealed positive and negative influences among test format, reading comprehension, and language learning. Therefore, participants’ preferences of test format should be considered when measuring their English reading proficiency. This study has pedagogical implications on the use of various test formats in L2 reading classrooms.