Randomized controlled trials to evaluate impact : their challenges and policy implications for medicine, education, and international development

Policy makers in education and international development have lately gravitated toward the randomized controlled trial (RCT)—an evaluation design that randomly assigns a sample of people or households into an intervention group and a control group in order to measure the differential effect of the intervention—as a means to determine program impact. As part of federal regulations, the U.S. Department of Education and the U.S. Agency for International development explicitly declared a preference for the RCT. When advocating for adopting the RCT model as the preferred evaluation tool, policy makers point to the success story of medical trials and how they revolutionized medicine from Medieval charlatanry to a modern life-saving discipline. By presenting a more nuanced account of the role of the RCT in medical history, however, this study finds that landmark RCTs were accompanied with challenges, Evidence-Based Medicine had rightful critics, and opportunistic biases in drug trials apply equally to education policy and international development. This study also examines the recent privileged role of the RCT in education and international development, concluding that its initial promise was not entirely born out when put into practice, as the national Reading First Initiative exemplifies. From a comparative perspective, the RCT movements also encountered major RCT critics, whose voices were not initially heard. These voices, however, seem to have contributed to a swing of the pendulum away from RCT primacy back towards greater methodological pluralism. A major conclusion of this study is that policy makers should exercise great caution when using RCTs as a policy evaluation tool. This conclusion is arrived at via considering RCT biases, challenges, and limited generalizability; understanding its interpretive-qualitative components; and broadening the overall methodological repertoire to better enable evaluations of macro-policy interventions.