Thursday, October 23, 2014
20

Open-Access Economics

CANBERRA – The brouhaha over Carmen Reinhart and Kenneth Rogoff’s article “Growth in a Time of Debt” may be the most conspicuous and incendiary scholarly controversy since 1974, when two earlier economists, Robert Fogel and Stanley Engerman, published a notorious book, Time on the Cross, defending the efficiency of American plantation slavery.

As with Time on the Cross, the Reinhart/Rogoff controversy, while ostensibly stemming from the authors’ statistical procedures, is actually rooted in the purposes to which others put their study.

Some of the results reported by Fogel and Engerman were used – not by the authors themselves, it should be noted – to challenge affirmative action and question the civil-rights movement. Similarly, some of the results reported by Reinhart and Rogoff have been used by politicians and others to justify fiscal austerity.

When the problems with the Reinhart/Rogoff analysis came to light, the critics were aghast. The authors had inadvertently omitted data, used a questionable weighting scheme, and employed an erroneous observation on GDP growth.

This raised uncomfortable questions not only about the efficacy of austerity, but also about the reliability of economic analysis. How could a flawed study have appeared first in the prestigious working-paper series of the National Bureau of Economic Research (NBER) and then in a journal of the American Economic Association? And, if this was possible, why should policymakers and a discerning public vest any credibility in economic research?

It was possible because economists are not obliged to make their data and programs publicly available when publishing scientific research. It is said that NBER working papers are even more prestigious than publication in refereed journals. Yet the Bureau does not require scholars to post their data and programs to its Web site as a condition for working-paper publication.

Independent scholars seeking to replicate these studies’ findings must first replicate the data and then replicate the programs. And, as empirical economics has progressed, the difficulty of doing so has grown. Reinhart and Rogoff may have used a relatively small set of mostly publicly available data, but the profession as a whole is using ever-larger tailor-made data sets.

Big data promises big progress. But large data sets also make replication impossible without the author’s cooperation. And the incentive for authors to cooperate is, at best, mixed. It is therefore the responsibility of editorial boards and the directors of organizations like the NBER to make open access obligatory.

Moreover, in a discipline that regards ingenuity as the ultimate virtue, those who engage in the grunt work of data cleaning and replication receive few rewards. Nobel prizes are not awarded for constructing new historical estimates of GDP that allow policy analysis to be extended back in time.

Then there is the fact that correlation is not causation. In the case of Reinhart and Rogoff, the observation that highly indebted countries grow more slowly, even if true, does not tell us anything about whether high debt causes slow growth or vice versa.

These are difficult questions, but they have simple solutions. What is needed is not more sophisticated statistical methods, but serious historical analysis of the political and economic particulars of specific historical cases in which countries were burdened with heavy debts. A proper historical analysis would help to identify cases in which debt was incurred for reasons other than the state of the economy, so that causality arguably runs from debt to growth, rather than the other way around.

Economic historians have shown how this can be done. My Berkeley colleagues David and Christina Romer, for example, faced an analogous problem when seeking to determine whether monetary-policy shocks affect economic growth. They used careful historical analysis to identify and focus on cases in which the policy stance changed for reasons not having to do with the current state of the economy. Doing so allowed them to isolate the impact of monetary shocks on growth.

Statistics are helpful. But in economics, as in other lines of social inquiry, they are no substitute for proper historical analysis.

In impugning the authors’ motives and criticizing the uses to which others have put their research, critics of Reinhart and Rogoff have taken their eye off the ball. The real problem is scholarly procedures and priorities, not motives. If the problem of procedures and priorities is addressed, the fact that politicians are tempted to misuse scholarly analysis for their own ends will take care of itself.

In other words, what is true of the economy is equally true of economic analysis. A crisis is a terrible thing to waste.

Hide Comments Hide Comments Read Comments (20)

Please login or register to post a comment

  1. CommentedRobert Tetlow

    Professor Eichengreen's remarks are mostly wrong - headed, in my opinion. R&R made their data available; that's why their error (and weighting choice) was uncovered. It is ridiculous to suggest that a working paper series should have some journal - like screening mechanism simply because it is "prestigious." And, in any case, it is not the job of any one paper (or book) to establish the truth, but rather the whole strand of literature. Anyone who bases a major policy decision on whatever has appeared in the latest working paper is an idiot. To my mind, the system in this case worked pretty much the way it is supposed to.

  2. CommentedEdward Ponderer

    Economists and there measures are great for evaluating a huge amount of data statistically and applying some general course corrections. Once upon a time...

    It was like in a 2D PACMAN game -- where are the ghosts, where is the food -- okay, go forward and turn right at the next corner and eat -- then we'll reassess. Milton Friedman and his like were masters of the game.

    Today, we are like people escaping a flame and smoke engulfed building, and hoping to make it to a new one with a brighter future.

    Opening our eyes for a split second to examine the diffused light in the smoke and judge whether to turn right or left, then shutting them for a bit because of the sting -- were we to depend only on that -- would surely doom us.

    Rather our legs inform us that there is a chair in front of us, walk along its perimeter and don't trip! Our hands find a doorknob for us, and later the banister of the stairwell. The left side of our face feels the more intense heat and warns us to look towards the right for means of escape. Our lungs warn us that its time to get down and crawl for a bit.

    We get vast continuous feedback leading to a continuous, complex varied movement which securely takes us out of the chocking pale red darkness out into the clear blue sky and life.

    So it is where our oh so global world finds itself today. Like our whole bodies, the world must become involved -- through a sense of mutual responsibility, through round tables, through listening with an open ear, and speaking with an honestly concerned mouth.

    The methods to become this type of society through integral education and media environment stand before us. There is a window of opportunity beginning from when the smoke alarms go off, until we pass out from smoke inhalation. There is still time, lets start using it wisely.

  3. CommentedZsolt Hermann

    The problem is different.
    The fault is not with the economists, and it is not with their research method.
    The problem is the whole framework, paradigm they operate in.
    The whole socio-economic system we think we exist in, the whole set of laws, tools we use for measurements and solutions is artificial.
    Humanity a while ago decided we exist outside, above the natural system we evolved from, and we can set up subsystems, new laws and frameworks to our liking, as if we existed in a bubble, where we can do whatever we want.
    What is happening today in the form of the deepening crisis is that this bubble started bursting, as humanity expanded, evolved enough that our previous "borrowed time", seemingly loose, free system became so integrated within the natural system, that today we cannot escape the laws that have been operating around us all along, which laws now started directly affecting us again.
    Today's politicians, experts base their decisions, solutions on coordinates that simple do not exist, have no validity in the natural system we exist in, since our whole lifestyle is unnatural and thus unsustainable.
    The only way forward is through studying and accepting the natural laws of general balance and homeostasis we have been ignoring for a long time now.
    Humanity needs to adapt back to the natural system as any other living organisms in order to continue developing in the evolutionary process.
    We have all the necessary data, all the talent, ingenuity and adaptability to make it happen, and thus achieve a much better, and sustainable quality of life we ever had before.

  4. CommentedStamatis Kavvadias

    Not only that, you are convinced in advance what the historical data should prove!!!!!!!!!!

    Thanks again for the scientific etiquette.

  5. CommentedStamatis Kavvadias

    So, prof. Eichengreen, you think economists should not make their science precise (for all the obfuscation by your claim for availability of data *and programs* with publication), but they should rely on an even less precise science: history! This is amazing!

    Prof. Eichengreen, the only *owners* of the "correct" interpretation of history that I know, are communist parties!

    Thank you very much for the insight on the path forward for economics. It seems to me, you are only seeking ways for economists to get published more and easier.

      CommentedStamatis Kavvadias

      Not only that, you are convinced in advance what the historical data should prove!!!!!!!!!!

      Thanks again for the scientific etiquette.

      CommentedStamatis Kavvadias

      Not only that, you are convinced in advance what the historical data should prove!!!!!!!!!!

      Thanks again for the scientific etiquette.

  6. CommentedRob Baston

    Precisely. Results need to be reproducible. R&R would have made a better contribution if they had published their data along with their book, originally.

    By the way, rating agency use of historical sub-prime mortgage data was behind the high ratings they assigned. Correlation is no proof of causality, and history no guide to the future.

    And simply pointing up possible causality is no use if you cannot provide a reasonable intellectual model for that causality.

  7. CommentedFrank O'Callaghan

    Professor Eichengreen is quite clear when he says:

    "It was possible because economists are not obliged to make their data and programs publicly available when publishing scientific research."

    It is why economics is not a Science. By and large the area has been captured by ideology. Many practitioners are -unknowingly- in the grip of ideologues of whom they are unaware. Peer review in economics it more informative of the uniformity of the peers than anything it says about the material reviewed.

  8. CommentedBill Ellis

    Michael says..."Other multiple researcher projects found similar thresholds."

    They found correlation's. But the causation runs the other way.
    So there is no "threshold" to be found...

  9. CommentedWilliam Osterberg

    Having published empirical work in economics I can easily imagine how this might have happened. Problems in being able to replicate results in published papers are nothing new. What is needed is the recognition by professors in graduate schools, and at the journals, that efforts to replicate are worthy of recognition.
    Mandatory provision of data would be a good requirement for publication in journals and some journals might do this. However, often empirical work uses data that cannot be made public, such as official fx intervention.
    In addition, although it might not be the case with the work in question, it is not easy to find referees for journal articles that will put in much time or effort. This is in part because often referees do not get paid and their work is not counted much professionally.
    Finally, Dr. Eichengreen has pointed to a serious flaw in graduate education in economics; there is often little or no teaching of history. The same could be said about comparative economics.

      CommentedStamatis Kavvadias

      I disagree in almost everything you say. Programs do not need to be made available. Peer researches replicating experiments in their own setup is a better indication of the relative error that research results have, and should not be hidden. As for data sets, for those parts of them that can be made public, if there is a serious community of scientists anywhere among economists, they can create a publicly accessible database of observations. This problem is not for publications to solve.

      The problem is that most economists are charlatans, and propose things that have *nothing* to do with science. If you want an example, economics need to follow the path historically taken by computer science, from qualitative studies to quantitative studies.

      The rest are excuses. The next thing you will tell us is to trust historians in political decisions. That is enough!

  10. CommentedJoey Cheung

    Nice to have an established econometrician against R-R, not for (like James Hamilton.)

  11. Portrait of Michael Heller

    CommentedMichael Heller

    Barry: Oops to myself and apologies to you. The book appeared before the paper. Even so, I stick to the argument. And I meant whinging (coding error).

  12. Portrait of Michael Heller

    CommentedMichael Heller

    C'mon Barry. You're making a hysterical Mount Everest out of a mere mole hill. All that matters now is the simple warning, not unique to R&R, that large debt is likely to lower growth. Other multiple researcher projects found similar thresholds. Also, you're winging about a working paper that appeared before subsequent peer reviewed papers, before a stupendous scholarly book, and before the recent re-calculations of R&R's unique data set, which appeared this month and that suggest even lower growth per 90% debt-to-GDP threshold. In addition, I think you're wrong about structural reform (oops, the thing you call austerity). References in my blog last week.

      CommentedRalph Musgrave

      Contrary to your claims that we should not point to Rogoff’s devious motives, what explanation is there for refusing to publish data other than a “devious motive”? And what adds substance to that charge is that Rogoff has published numerous articles which are propaganda rather than cold rational argument: articles aimed at scaring the public about the supposedly disastrous consequences of debts over 100% or so of GDP.

      As to the idea that debts over the 100% level are a problem, the UK’s debt ratio was over 200% after the Napoleonic wars and again after WWII. In both cases, growth appears to have been unimpaired. So how did Rogoff deal with that awkward fact? Well as we all know, he just omitted it from his data - or to be strictly accurate he attached the same weight to the UK’s healthy 20 years of post WWII growth as he did to ONE YEAR of New Zealand’s growth (a country a tenth the size). I.e. he all but ignored the latter 20 year UK experience.

      In short, Rogoff and Reinhart just fiddled the evidence.

      CommentedRalph Musgrave

      Contrary to your claims that we should not point to Rogoff’s devious motives, what explanation is there for refusing to publish data other than a “devious motive”? And what adds substance to that charge is that Rogoff has published numerous articles which are propaganda rather than cold rational argument: articles aimed at scaring the public about the supposedly disastrous consequences of debts over 100% or so of GDP.

      As to the idea that debts over the 100% level are a problem, the UK’s debt ratio was over 200% after the Napoleonic wars and again after WWII. In both cases, growth appears to have been unimpaired. So how did Rogoff deal with that awkward fact? Well as we all know, he just omitted it from his data - or to be strictly accurate he attached the same weight to the UK’s healthy 20 years of post WWII growth as he did to ONE YEAR of New Zealand’s growth (a country a tenth the size). I.e. he all but ignored the latter 20 year UK experience.

      In short, Rogoff and Reinhart just fiddled the evidence.

Featured