Comments 24

The Problem With Evidence-Based Policies

Feb 25, 2016 Ricardo Hausmann

Many organizations, from government agencies to philanthropic institutions and aid organizations, now demand that programs and policies be “evidence-based.” But the way this idea is being implemented may be doing a lot of harm, impairing our ability to learn and improve on what we do.

CAMBRIDGE – Many organizations, from government agencies to philanthropic institutions and aid organizations, now require that programs and policies be “evidence-based.” It makes sense to demand that policies be based on evidence and that such evidence be as good as possible, within reasonable time and budgetary limits. But the way this approach is being implemented may be doing a lot of harm, impairing our ability to learn and improve on what we do.

The current so-called “gold standard” of what constitutes good evidence is the randomized control trial, or RCT, an idea that started in medicine two centuries ago, moved to agriculture, and became the rage in economics during the past two decades. Its popularity is based on the fact that it addresses key problems in statistical inference.

For example, rich people wear fancy clothes. Would distributing fancy clothes to poor people make them rich? This is a case where correlation (between clothes and wealth) does not imply causation.

To continue reading, please log in or enter your email address.

To access our archive, please log in or register now and read two articles from our archive every month for free. For unlimited access to our archive, as well as to the unrivaled analysis of PS On Point, subscribe now.

Featured

The Despot and the Diplomat

Jun 20, 2018 Christopher R. Hill
Why Host the World Cup?

Jun 20, 2018 Andrew Zimbalist
Nationalism Will Go Bankrupt

Jun 20, 2018 Anatole Kaletsky
Why Markets and Political Scientists Disagree on the G7

Jun 19, 2018 Mohamed A. El-Erian
Can the Euro Be Saved?

Jun 13, 2018 Joseph E. Stiglitz

0 Comments on this paragraph, 30 in all 30 Comments on this article

Emily Hayter Apr 15, 2016

I agree with the issues pointed out about the danger of RCTs as a 'gold standard'. But not all implementing agencies take the approach described. At INASP we are leading one of 5 DfID-funded consortia working around around the world on evidence informed policymaking. Our Consortium, called the VakaYiko Consortium, consists of 5 organisations working in 4 core countries (Uganda, Zimbabwe, South Africa and Ghana) to build civil servants' capacity for using evidence. Across these countries we are working with 3 ministries/depts, 3 Parliaments and 1 civil service training college as well as local research and civil society organisations. Read more here: http://www.inasp.info/en/work/vakayiko/

We look at four types of evidence which are needed for policy: administrative data/statistics, citizen knowledge (eg. gained through stakeholder consultations), research evidence, and practice informed evidence (eg. professional expertise/experience). Our approach considers capacity at multiple levels: civil servants' individual capacity to find, evaluate and communicate quality evidence to decision makers, organisational processes for determining and responding to evidence needs, and a wider enabling environment of media and civil society including networks between researchers and policymakers.
Bakhodur Eshonov Mar 6, 2016

It might be wrong considering development policies just as a set of ad hoc 'social interventions' by policy makers elected for 3-5 years. Instead, successful development requires all the policies and interventions to be well integrated into a coherent long-term strategy (vision). The latter requires holistic approach integrating both economic, social, spatial, and institutional (governance) transformations in a given society (nation). The latter means that any interventions could be assessed only in the context of the success of the longer term strategy, which in most of cases takes decades, sometimes generations.
Bakhodur Eshonov Mar 6, 2016

A bit old publication (2009), by Jan Vandermoortele (UNDP) who also was questionning this new 'fashion" - "...Evidence-based‘ policymaking has recently emerged as development jargon. It merits a pause to reflect on its meaning. Does it mean that in the past, policies were made by
people who sat in a dark room or were blindfolded; or who were driven by emotions or beliefs? Does it mean that policies are now made by purely rational beings; people who are no longer influenced by experience, context or politics?" (http://www.devstud.org.uk/downloads/4b9ea0272d30b_Vandemoortele_PAPER.pdf).
YangKai Liu Mar 1, 2016

RCT is never so crude as the author suggested here, it is far more efficient than the author described. It will be a pity to write down or even write-off one method before understanding its full potential and efficiency.
First, leaving half of your sample for control group is a terrible waste of resource, depending on sample size, it is common to conduct a successful RCT with smaller control group.
Second, one variable per RCT is also a terrible waste. With proper, it is not uncommon to have one RCT to test four to five variables, each with two to five distinct values. Following his example, one test can include several different makes of tablets paring with several different software programs.
As a final remark on RCT, I am curious about the unit used in the example. Since the students are the subject of the study, why don't we use student as the unit instead of school? In this way, one may not even need 300 schools to draw a meaningful conclusion from the RCT.
Paul Daley Feb 29, 2016

Interesting. Randomized control trials were developed for use in cases where your ignorance was profound and the consequences of being wrong were severe. The costs of the test relative to the potential benefits was also a factor. So, Hausmann's point is correct; they can't be used in all cases, though there are special circumstances in which they may be appropriate
Jose araujo Feb 29, 2016

@Steve

Infrastructure, multi year capacity investments are always risky. Actually one of the roles of government is promote investments that markets wouldn't.

Regarding the government efficiency on social goods, the reality is that without prejudices and inefficiency assumptions, government is by default more efficient then markets, due to the characteristics of the good and of the government action: Compulsory capacity of rates, reduced debt costs, lack of profit among others explain why Education, Health, Justice, Order are goods better produced by governments. Actually evidence also proves it.
Phillip Tussing Feb 28, 2016

Hausmann certainly knows that what he is proposing is "action research", or "participatory action research", which is a well-known and well-defined field of development economics. I agree that it often, or even in the majority of cases, would be preferable to RCTs to develop strategies to improve specific situations, especially in instances when there are not a small number of options to consider. t is an important arrow to the development economist's bow. Why doesn't he just say this? No need to re-invent this wheel, a lot of good research has gone into it. For example: http://www.qualitative-research.net/index.php/fqs/article/view/1801/3334
M Chrisney Feb 28, 2016

I think there are, at least, 4 questions policymakers should use to assess evidenced-based policies.
https://twitter.com/MChrisney/status/703966684604932097

1. Is the setting for the experiment similar to the setting for implementing the policy? This is a problem of transferability.

2. Could scaling up a pilot program have unintended negative effects? This is the problem of scaling up.

3. Can we replicate the conditions needed for success? Any differences can undermine the apparent cause-effect linkage of the policy/program.

4. How widespread are the benefits? This can affect the political economy of implementing a policy based on RCT evidence.
barry stanley Feb 28, 2016

Whether or not Ricardo's solutions would stand the test of time, the issue of evidence-based policies certainly need reviewing.
It seems to me that in medicine all ideas that are in anyway in conflict with established evidence-based protocols are rejected, with little or no consideration.
This can occur even in the presence of overwhelming associations. The outstanding example is the associations between prenatal alcohol exposure and DSM psychiatric diagnoses: associations ignored by Psychiatry, often on the grounds that there is no supportive evidence based research. This may be true, but only because of the lack of funding and the reluctance of researchers to pursue such a politically charged subject.
Barry Stanley
eusebio manuel Feb 28, 2016

When you live in a state of law citizens have to act with inteligence hanf help end poverty
T.N. Srinivasan Feb 28, 2016

Ricardo, I am sure, knows that RCT is a method for making an internally valid inference from the sample with which the RCT experiment was done. To extend its findings beyond the sample and the experiment is not statistically valid ---for validation of findings to other contexts would in general require repeating the experiment in each context or embedding all contexts in a super model and use the relevant statistical inference techniques. This is well known and uonderstood, Ricardo should read Section 7e.2 of C.R. Rao.s classic text Linear Statistical Inference.
tn
Daniel Kloke Feb 27, 2016

Clearly, the author does not understand the mechanics of learning, which is a process of neural path development and training, in which tablet devices have limited application. In the example given, tablets might not be able to capture all, or even any, of the critical components of a successful teaching strategy.

My heart goes out to students subjected to such haphazard and irrational experimentation. And the author (and several commenters) might take the time to acquaint themselves with the basics of neurobiology and brain function.
T B Feb 26, 2016

The strategy suggested here would result in following random noise in the data. The result of this approach would almost always (through random fluctuations in the data) show that tablets improved learning. This would not be a statistically valid result. Perhaps Mr. Hausmann's use of this kind of incorrect reasoning as the minister of planning for Venezuela hastened that country's economic disaster.
- Steve Hurst Feb 26, 2016
  
  @TB
  At one stage I spent quite a bit of time with mathematicians and notable institutions trying to define the certainty associated with limited numbers of events, because in the real world you have limited event data and some events cost considerable money to repeatedly replicate. It rapidly became data porridge. The same problem frequently occurs with economics as far as I can see, limited data sets. It doesnt bother me if people want to take time trying to validate what they are doing but ultimately there is no statistical certainty in many cases. The idea or comfort of verification is often false.
Henry Lucas Feb 26, 2016

A excellent and succinct statement of the issues. The only point I would add is that in clinical trial it is double-blinded RCTs that are the real gold standard . Because blinding is almost never practical in other areas that condition is omitted by those arguing for the wide-scale application of this approach. My experience would suggest that placebo and nocebo effects can be very powerful in social interventions. Both those managing an intervention (most of us prefer positive findings) and those receiving it (who often assume that success may lead to additional funding) have incentives to (consciously or unconsciously) go out of their way to make the intervention succeed.
Tariq Maketab Feb 26, 2016

Steve, sure you meant "public"?
- Steve Hurst Feb 26, 2016
  
  @Tariq
  
  ¯\_(ツ)_/¯
  
  You got me there man, sticky keyboard
  
  I've got nothing against public spending, just models to take the risk away dont really matter because there is no risk
Michael Public Feb 26, 2016

Evidence based thinking is highly effective when dealing with inorganic things. Like measuring the lead content in the air. Here there is no substitute for evidence based approached. When dealing with human behavior and organic subjects the subjects are vastly different, constantly changing and deeply complex and this way of thinking is often counterproductive. Of course, this viewpoint goes against the grain of the general consensus, so expect massive resistance from almost every blowhard when try to explain why homeopathy actually does work or why planting Monsanto's high-yield seeds is bad in the long run.
Bernard Fudim Feb 25, 2016

The most profound understanding that Ricardo has brought to my mind, is that the randomized nature of human thought and societal relations, can lend itself poorly to a Randomized Controlled Test, because you have the interaction of two or more random parameters that have not carefully defined, and there is no allowance for a statistical regression. Of course in the instance of education, the variables are affordability and quality of internet connection, the distractive nature of the internet, any increase in complacency of the teacher after the internet is introduced. Even within an economic RCT the variables are likely, isolation, religious fervor, cultural history, etc.
Renee Ho Feb 25, 2016

I think we need a website to crowdsource the stories in which this "auditor-becomes-the-R&D-department" happens. Here's one from http://whenpeoplecomefirst.com/evidence/evidence-based-global-public-health

The anthropologist Vincanne Adams provides a sad example of what is all too common in the sector. In 2005, a project team set out to train health workers in techniques of safe motherhood in rural Tibet. At the last minute, their funders told them that their methodology was not rigorous enough—it was “not scientific.”

Adams explains that, “Despite the fact that more women died in childbirth in rural Tibet than in just about any other country, and despite the fact that training would be conducted in a large rural prefecture and was endorsed by the [Tibetan Autonomous Region] Health Bureau, we were told that in their catchment area ‘not enough women die to get a robust power calculation.’”

As a result of this perverse logic—and to maintain their funding—, the project team changed course and ran an RCT in the capital, Lhasa, to compare traditional Tibetan medicine with a Western drug that prevents hemorrhage in delivering women. In spite of the fact that this study had little relevance to the Tibetan Health Bureau, the research project was deemed a success. It was considered a success not because of its outcomes on health but the study team was able to do an RCT. The objective of the project was defined by the method rather than the other way around.
Jose araujo Feb 25, 2016

Probably the most important instrument in social sciences are monographs. Studies that document and allow us to understand the behavior of humans. Then and only then we can formulate hypothesis that we test following the appropriate methodologies (RCT and many others)

Problems are that models and experiments and data are used without hypothesis, and many times are just data points that prove prejudices.

Its time to separate economic science from economic politics, and that economists behave like scientists in the first place.
- Jose araujo Feb 25, 2016
  
  Too much models and very few monographs are produced by the economic science.
Cris Perdue Feb 25, 2016

Thank you Professor Hausmann, beautifully stated. I'd like to just a few words of my own in support.

Indeed as you say, in cases where a well-defined intervention can be compared with a baseline case, and other factors influencing the outcomes can be reasonably-well identified, randomized control trials can be great.

But many programs and policies do not have these characteristics, and in these cases the cost of trials is high and the results are far too likely to be misleading.

As you probably well know, this kind of distinction is sometimes referred to in statistics as confirmatory data analysis (e.g. an RCT), as opposed to exploratory data analysis, which is much more like the traditional careful observation that good men and women of science and technology have always done.
Barry Schachter Feb 25, 2016

Lessons traveling poorly isn't unique to RCTs. That is a general problem with inference-based knowledge. Perhaps the Guatemala/Norway example is a straw man.
Akib Khan Feb 25, 2016

The questions Ricardo poses re the ‘external validity’ of RCTs are valid ones, one must admit. But it's also true that, over the last two decades, the same issues have gained increasingly more traction within the so-called Randomistas and the recent studies have put a lot more emphasis on the generalizability of results compared to their predecessors.

As Ricardo has articulated, the 'context' may have a significant bearing on the results of an RCT. But rather ironically, it may play a critical role in his example as well. His convenient assumption re the context here is that the teachers face the right incentives to experiment with the tablets to facilitate their students' learning - which, given the reality of many developing countries, is not an innocuous one. There are potentially more than one way of introducing tablets in schools, so how would one determine the relative effectiveness of these methods? By using an RCT, may be?! Ricardo's model doesn't provide any answer to that.

Of course, RCTs don't offer a panacea to all the important questions in the domain of development policy and programming. But the increasing emphasis on external validity, thanks to informed criticisms like this one, is providing the researchers with the right incentives to be more careful about both their design and claims. A good example could be the recent strand of papers on the impact of microfinance which shows how RCTs might provide useful evidence on development programs across different contexts.
- Akib Khan Feb 25, 2016
  
  'There are potentially more than one way of introducing tablets in schools' - just for clarification, I meant that there are other ways of introducing the use of tablets apart from that specified by Ricardo.
Benjamin MG Feb 25, 2016

Sure you didn't mean "trawling" the design space?
Steve Hurst Feb 25, 2016

Pubic spending is risk averse, private business is not. There is no point in trying to mix the way the two work.

The problem with public spending is the money does not need to be earned, it is gifted by the taxpayer. Nothing is going to change that.

Becasue there is no customer driven transaction the public spender seeks to verify they are spending the money in a justifiable way. Nothing is going to change that. In short you are wasting your time.

Be grateful there is a private sector that is taking risk, for otherwise there would be no public sector. The most striking difference between public and private mentality is highlighted by Steve Jobs comment about committees - "It's really hard to design products by focus groups. A lot of times, people don't know what they want until you show it to them." That outlook is totally alien to the public sector
- Matthew McCarthy Feb 26, 2016
  
  @Jose Araujo - Steve didn't say that public spending was without risk, just risk averse. Different concept.
  As to the relative efficiency of government and the private sector, you should be aware of examples and counterexamples for either side of that argument and not claim a "default" assumption.
- Jose araujo Feb 26, 2016
  
  Why would you ever think that? i.e. That public spending involves no risk?
  
  Is there any fundamental change in the investment models you apply in private vs public, that neglects risk?
  
  You can greatly reduce risk by having the investment public. Infrastructure is the case, security, health, education, etc, but that doesn't mean that risk isn't accounted for.
  
  By default government is much more efficient producing and managing this "social" goods then the public. i.e. markets.

http://prosyn.org/OOvVfVG;

Log in/Register

The Problem With Evidence-Based Policies

To continue reading, please log in or enter your email address.

Featured

The Despot and the Diplomat

Why Host the World Cup?

Nationalism Will Go Bankrupt

Why Markets and Political Scientists Disagree on the G7

Can the Euro Be Saved?

New Comment

Email this piece to a friend

Contact us

Toward a More Democratic Europe?

Rediscovering the Promise of Nuclear Power

The Western Crack-Up

The Political Decline of Religion in the Middle East

The Epidemic We Must Stop Ignoring

Nationalism Will Go Bankrupt

Why Host the World Cup?

The Despot and the Diplomat

Cookies and Privacy