CAMBRIDGE – Many organizations, from government agencies to philanthropic institutions and aid organizations, now require that programs and policies be “evidence-based.” It makes sense to demand that policies be based on evidence and that such evidence be as good as possible, within reasonable time and budgetary limits. But the way this approach is being implemented may be doing a lot of harm, impairing our ability to learn and improve on what we do.
The current so-called “gold standard” of what constitutes good evidence is the randomized control trial, or RCT, an idea that started in medicine two centuries ago, moved to agriculture, and became the rage in economics during the past two decades. Its popularity is based on the fact that it addresses key problems in statistical inference.
For example, rich people wear fancy clothes. Would distributing fancy clothes to poor people make them rich? This is a case where correlation (between clothes and wealth) does not imply causation.
Harvard graduates get great jobs. Is Harvard good at teaching – or just at selecting smart people who would have done well in life anyway? This is the problem of selection bias.