Files and links (1)
Dissertation_I-Chun Hsiao2.84 MB
Embargoed Access, Embargo ends: 06/29/2028
Dissertation
Fairness, validity, and impact of a standardized English proficiency test in EMI contexts and beyond: a mixed-methods study of test takers’ perceptions
University of Iowa
Doctor of Philosophy (PhD), University of Iowa
Spring 2026
DOI: 10.25820/etd.008406
Abstract
While English proficiency tests such as TOEFL, IELTS, and the Duolingo English Test have been extensively examined in assessment research, locally developed large-scale assessments remain underexplored, particularly in non-English-dominant higher education contexts. This study addresses this gap by investigating test takers’ perceptions of the BEST Test of English Proficiency (BESTEP), Taiwan’s first academically oriented English proficiency test introduced under the Bilingual 2030 policy. Specifically, the study examines the BESTEP’s fairness, validity, and its broader impact on learning experiences and future opportunities.
Building on Green’s (2014, 2020) Framework for Effective Assessment Systems, this study proposes a Quality of Standardized Language Tests model that conceptualizes test quality as an interconnected system comprising fairness, validity, impact, reliability, and practicality. Fairness is positioned at the same conceptual level as validity, drawing on Wallace’s (2018) Language Assessment Fairness Model, while validity is framed in accordance with the Standards for Educational and Psychological Testing (AERA et al., 2014). Test impact is conceptualized through Shih’s (2007) Washback Model and related work (Bachman & Palmer, 1996; ILTA, 2024), involving both intended and unintended consequences at individual and systemic levels.
An explanatory sequential mixed-methods design was employed. Quantitative data were collected through an online survey (N = 424) adapted from Wallace and Qin (2021), Dong (2022), and Nguyen (2023). Content validity was established through expert review (scale-CVI = .93), and the final 48-item instrument demonstrated strong internal consistency (Cronbach’s α = .88–.93) and construct validity (composite reliability > .75; average variance extracted > .50). Qualitative data were gathered through semi-structured interviews with 15 test takers representing diverse institutional and disciplinary backgrounds. Quantitative analyses included descriptive statistics, confirmatory factor analysis, and nonparametric group comparisons, while qualitative data were thematically analyzed and member-checked to enhance credibility.
Quantitative results indicated generally positive perceptions of fairness, relevance, and impact, though distributive fairness and perceived test utility received comparatively lower ratings. No significant differences emerged by gender, test-taking frequency, participation in English as a medium of instruction (EMI) courses, or year of study; however, institutional type and Ministry of Education (MOE) funding category were associated with significant differences in perceptions. Qualitative findings further revealed that fairness judgments extended beyond test content and scoring to involve administrative policies, access conditions, and interactional experiences during test administration. While BESTEP was viewed as broadly aligned with general language competence, concerns were raised regarding its alignment with EMI demands, its discriminatory power, and its difficulty level. In terms of impact, the test exerted limited influence on learners’ motivation, confidence, and study behaviors, and its perceived benefits were largely short-term and immediate. Participants also questioned its recognition for further study and employment, particularly in comparison with internationally established assessments.
By centering test takers’ perspectives, this study advances a more context-sensitive and justice-oriented understanding of standardized language assessment. It demonstrates that test quality must be evaluated not only through psychometric rigor but also through stakeholder experiences and institutional contexts in which tests are implemented. The findings provide methodological guidance for mixed-methods research and for the adaptation and validation of perception-based survey instruments. They also provide practical and policy-relevant insights for locally developed large-scale assessments in Taiwan and other non-English-dominant higher education systems.
Details
- Title: Subtitle
- Fairness, validity, and impact of a standardized English proficiency test in EMI contexts and beyond: a mixed-methods study of test takers’ perceptions
- Creators
- I-Chun Hsiao
- Contributors
- Lia Plakans (Advisor)Catherine Welch (Committee Member)Pamela Wesely (Committee Member)Xun Yan (Committee Member)
- Resource Type
- Dissertation
- Degree Awarded
- Doctor of Philosophy (PhD), University of Iowa
- Degree in
- Teaching and Learning (Literacy, Culture, and Language Education)
- Date degree season
- Spring 2026
- DOI
- 10.25820/etd.008406
- Publisher
- University of Iowa
- Number of pages
- xv, 238 pages
- Copyright
- Copyright 2026 I-Chun Hsiao
- Language
- English
- Date submitted
- 04/13/2026
- Description illustrations
- illustrations, tables
- Description bibliographic
- Includes bibliographical references (pages 195-215).
- Public Abstract (ETD)
This dissertation investigates test takers’ perceptions of the fairness, validity, and impact of a locally developed large-scale English proficiency assessment in Taiwan’s non-English-dominant higher education context. Using a mixed-methods approach, the study first surveyed 424 students and then interviewed 15 students from different universities and academic disciplines. It also documented the systematic adaptation and validation of perception-based survey instruments to ensure conceptual clarity and reliability.
Findings revealed that evaluation of test fairness extends beyond its content and scoring to include administrative policies, access conditions, quota allocation, and interactional factors such as proctor professionalism. Test takers questioned whether the test fully reflects the English skills required for their university learning, and they reported limited test impact on motivation or long-term learning. Moreover, the test was perceived as having relatively low difficulty, limited ability to differentiate performance, and limited recognition for further study, employment, or comparison with other internationally recognized English proficiency tests.
By centering test takers’ voices, this study indicates that test quality is more than technical accuracy; it is socially situated and influenced by institutional policies and practices. This dissertation advocates a holistic, context-sensitive approach to evaluating standardized language tests, considering social, ethical, and policy dimensions beyond test scores and technical accuracy.
- Academic Unit
- Teaching and Learning
- Record Identifier
- 9985176975902771
Metrics
1 Record Views