Research Designs for Replications
In design-based approaches, the researcher uses research designs to systematically test and address assumptions under the Causal Replication Framework (CRF). If replication failure is observed—and all other assumptions are met—then the researcher may infer that the tested assumption was violated and resulted in treatment effect variation.
There are two well-known approaches to replication: direct and conceptual replications. The CRF provides a formal way to understand each of these approaches. Direct replications seek to examine whether two or more studies with the same well-defined causal estimand yield the same effect.
The most stringent forms of direct replication seek to address all replication and individual study design assumptions. That is, these approaches attempt to hold all study characteristics fixed, while drawing new random samples for each replication study. When all assumptions are met, comparison of study results may be considered a test of statistical replication. However, on their own, statistical replications are rarely of interest because it is often impossible to reproduce the exact same conditions over multiple studies, even for the simplest interventions.
Below we discuss examples of research designs for direct and conceptual that are feasible and desirable in applied settings.
Direct Replications
In research designs for direct replication, the researcher examines whether two or more studies with the same causal estimand produce the same effect after introducing systematic variations in the research design, estimation approaches, or reporting procedures. If results do not replicate, the researcher interprets the cause of the replication failure as bias in the research design or estimation procedure or incorrect reporting.
Our team has done a lot of thinking about one type of direct replication — within-study comparisons (WSC) for evaluating non-experimental method performance. This approach examines whether a quasi-experimental design is able to replicate the effect from an experimental benchmark with the same target population.
Conceptual Replications
In research designs for conceptual replications, the researcher examines whether two or more studies with potentially different causal estimands produce the same effect. In prospective approaches, the researcher may introduce systematic variations in treatment conditions, in outcomes used for assessing effects, in target populations and in settings for conducting the study. In systematic conceptual replication studies, when replication failure is observed, the researcher is able to identify the source of the effect variation.
Further Reading
Wong, Vivian C., Peter M. Steiner, and Kylie L. Anglin. (2020). Design-Based Approaches to Causal Replication Studies. (EdWorkingPaper: 20-311). Retrieved from Annenberg Institute at Brown University: https://doi.org/10.26300/xsqw-c323
Design replication studies (within-study comparisons) as direct replications.
