How to Read an Empirical Paper

Zahi gave us a paper to read for his seminar today, and asked us to read it very carefully and come prepared to talk about it.  The paper was very well written, displayed an impressive use of data, and exploited an interesting variation in real estate regulation.  A few of us in the class had objections to the use of data, or alternative stories that might drive the results.  Zahi discussed this paper at NBER a few months ago, and his critique turned out to be far more substantial, and more fundamental, than ours.

The paper used a difference-in-differences identification technique.  Diff-in-diff requires a treatment and a control group, whose varying behavior before treatment must be predictable functions of the same set of variables.  This means, while the treatment and control groups don’t have to be identical, the researcher should fully understand what causes the difference.  Strictly speaking, the treatment should also be applied randomly so there is no “selection bias,” or ability of the data points to choose whether they are in the treatment or control group.

In the paper we discussed, Zahi argued, the treatment and control groups were not the same, and were not functions of the same variables.  They were subject to completely different market forces, so the paper’s conclusions reduce to a simple and rather meaningless observation.  In addition, the treatment was not assigned randomly.  He argued that self-selection is a big problem for this paper.

How to Read an Empirical Paper

  • Find the author’s identification technique.
  • Go back to basics and ask what a textbook application of the technique would require.
  • Think carefully about whether the paper meets the textbook requirements. Consider
    • the author’s selection and use of data,
    • the stated and implicit assumptions required to make the findings valid,
    • the paper’s internal vs. external validity.
  • Consider whether deviations from the ideal can be (and are) successfully defended.
  • Ask where this paper fits into the literature, and what it’s impact is likely to be.