Matthew Olckers Economist by day; graffiti nerd by night.

Oomph versus sizeless science

I have been reading Ziliak and McCloskey’s The Cult of Statistical Significance. I am extremely relieved I stumbled upon their argument early in my research career. If I hadn’t, I would have continued on the path to sizeless science.

Their argument is that many fields, and especially economics, focus only on statistical signficance—the sizeless science—and ignore the magntiude—the oomph or economic significance. Statistical significance merely shows the probability of observing some measure in the data given that our hypothesis is true (where the hypothesis is often that the measure equals zero). Ziliak and McCloskey stress that statistical significance is nethier necessary nor sufficient for economic signficance.

To show that sizeless science is widespread, they apply questions to all empirical studies published in the American Economic Review in the 1980s and 1990s. My stomach turned with the thought that I could just as easily make the same mistake. I repeat the questions below as a reminder to myself to apply these standards to my own work. With some luck, my work will be all about the oomph.

Does the article:

  1. Use a small number of observations, such that statistically significant differences are not found merely by choosing a very large sample?
  2. Report descriptive statistics for regression variables?
  3. Report coefficients in elasticities, or in some other useful form that addresses the questions of “how large is large”?
  4. Test the null hypothesis that the authors said were the ones of interest?
  5. Carefully interpret the theoritical meaning of the coefficients? For example, does it pay attention to the details of the units of measurement and to the limitations of the data?
  6. Eschew reporting all standard errors, t-, p-, and F-statistics, when such information is irrelevant?
  7. At its first use, consider statisitcal significance to be the one among other criteria of importance?
  8. Consider the power of the test?
  9. Examine the power function?
  10. Eschew “asterisk econometrics”, the ranking of coefficients according to the absolute value of the test statistic?
  11. Eschew “sign econometrics,” remarking on the sign but not the size of the coefficient?
  12. Discuss the size of the coefficients?
  13. Discuss the scientific conversation within which a coefficient would be judged large or small?
  14. Avoid choosing variables for inclusion soleley on the basis of statistical significance?
  15. Use other criteria of importance besides statistical significance after the crescendo?
  16. Consider more than statistical significance decisive in an an empirical argument?
  17. Do a simulation to determine whether the coefficients are reasonable?
  18. In the conclusions, distinguish between statistical and economic significance?
  19. Avoid using the word significance in ambiguous ways.