Risk of Bias in Experiments, Quasi-Experiments and Natural Experiments Across Disciplines: Discussion Paper and Assessment Framework

Hugh Sharma Waddington; David B. Wilson; Terri Pigott; Ariel M. Aloe; Peter Tugwell; Vivian Welch; Gavin Stewart

doi:10.1177/18911803261435894

Back

Risk of Bias in Experiments, Quasi-Experiments and Natural Experiments Across Disciplines: Discussion Paper and Assessment Framework

Journal article

Open access

Peer reviewed

Risk of Bias in Experiments, Quasi-Experiments and Natural Experiments Across Disciplines: Discussion Paper and Assessment Framework

Hugh Sharma Waddington, David B. Wilson, Terri Pigott, Ariel M. Aloe, Peter Tugwell, Vivian Welch and Gavin Stewart

Campbell systematic review, Vol.22(2), pp.1-17

06/01/2026

DOI: 10.1177/18911803261435894

Files and links (1)

url

https://journals.sagepub.com/doi/10.1177/18911803261435894View

Published (Version of record) Open Access

Abstract

The evidence we provide to support decision-making should be rigorously appraised so that the findings are shown to be valuable. We discuss the risk of bias in impact evaluations on social and natural science topics - that is, studies using a variety of experimental, quasi-experimental and natural experimental approaches to quantify the causal effect of an intervention, program or policy on an outcome of interest. Existing tools to facilitate evaluation of the risk of bias are usually conceptualized to assess either randomized controlled trials (RCTs) or non-randomized studies of interventions, often called quasi-experimental designs (QEDs) or natural experimental evaluations, but not both. The tools do not adequately reflect how common sources of bias might be addressed in impact evaluations in the social and natural sciences, or the bias sources particular to certain types of design, such as participant reactivity to researcher observation in a trial, or when modelling incorporates known selection mechanisms other than randomization, such as subversion of the assignment rule in a discontinuity design. We present a heuristic to assist reviewers in assessing the confidence in causal inferences. Our approach emphasizes four common sources of bias across RCTs, QEDs and natural experiments - the equivalence of groups, the fidelity of study conditions, the adequacy of measurement, and the reporting of analyses - and we provide signaling questions to evaluate these sources of bias for particular types of study. The approach should be adapted to suit interventions and review topics of interest.

Social Sciences

Social Sciences - Other Topics

Social Sciences, Interdisciplinary

Details

Title: Subtitle: Risk of Bias in Experiments, Quasi-Experiments and Natural Experiments Across Disciplines: Discussion Paper and Assessment Framework
Creators: Hugh Sharma Waddington - London Sch Hyg & Trop Med, Dept Populat Hlth, Planetary Hlth Grp, Keppel St, London WC1E 7HT, England
David B. Wilson - George Mason University
Terri Pigott - Georgia State University
Ariel M. Aloe - University of Iowa
Peter Tugwell - University of Ottawa
Vivian Welch - Campbell Collaborat, Philadelphia, PA USA
Gavin Stewart - Newcastle University
Resource Type: Journal article
Publication Details: Campbell systematic review, Vol.22(2), pp.1-17
DOI: 10.1177/18911803261435894
eISSN: 1891-1803
Publisher: Sage
Number of pages: 17
Grant note: Pathfinder Initiative (project no: 227165/Z/23/Z) / Wellcome Trust
Language: English
Date published: 06/01/2026
Academic Unit: Education Administration; Psychological and Quantitative Foundations
Record Identifier: 9985161450702771

Metrics

1 Record Views