Friday, June 22, 2007
The Death of Significance?
by Tom Bozzo
(Cross-posted at Total Drek.)
(*) This sometimes leads to wacky advice being given to everyday applied researchers from econo- or sociometricians, of the "if a result from an inconsistent esitmator goes away with a consistent (but inefficient) procedure, be suspicious [or vice-versa]." Armstrong's bottom-line recommendations address the reasonable suspicions that might arise.
At Decision Science News (another h/t to Brad DeLong), Dan Goldstein prints a comment from J. Scott Armstrong who has "concluded that tests of statistical significance should never be used." [Emphasis mine.] He is not conducting statistical performance art, and I substantially agree with the conclusion. A couple random remarks:
- There are results which lead to a conclusion that social science researchers tend to tweak their statistical models to cross significance thresholds so they can produce positive results with (presumably) greater probability of publication. But,
- To do so invalidates the published inferences. Because,
- The "classical" statistics reported by most software packages are invalid under any pretesting (i.e., deciding on a model specification based on results from preliminary estimation). And,
- The prospects for computing or simulating correct statistics are as good as the quality of the researcher's choice trail. But,
- A lot of social science "theories" don't determine the full set of explanatory variables, making the lure of statistical model diagnostics attractive. Though,
- There are families of models (e.g., the 'flexible functional form' cost models in economics, which I work with) where individual coefficients have no theoretical interpretation, in which case the researcher has no direct basis for evaluating the consequences of a restriction. More broadly,
- Properties of social science data often mean we need to use consistent but inefficient estimators; sometimes "better" significance from inappropriate estimation methods has little if any meaning. (*) Last,
- Some researchers (not least many who publish empirical results in top economics journals) tend to focus excessively on statistical significance to the detriment of more interesting discussions of the non-statistical significance of their results. (Views differ.)
Authors... instead... should report on effect sizes, confidence intervals, replications/extensions, and meta-analyses.For those of you with institutional access, links to the International Journal of Forecasting article are at the Decision Science News link.
(Cross-posted at Total Drek.)
(*) This sometimes leads to wacky advice being given to everyday applied researchers from econo- or sociometricians, of the "if a result from an inconsistent esitmator goes away with a consistent (but inefficient) procedure, be suspicious [or vice-versa]." Armstrong's bottom-line recommendations address the reasonable suspicions that might arise.
Labels: Econometrics, Economics, Social Science, Statistics
Comments:
<< Home
Apologies if this is redundant:
http://www.stat.columbia.edu/~cook/movabletype/archives/2007/06/is_significance.html
In particular:
"the difference between "significant" and "not significant" is not itself statistically significant." at
http://www.stat.columbia.edu/~gelman/research/published/signif4.pdf
http://www.stat.columbia.edu/~cook/movabletype/archives/2007/06/is_significance.html
In particular:
"the difference between "significant" and "not significant" is not itself statistically significant." at
http://www.stat.columbia.edu/~gelman/research/published/signif4.pdf
I have a horrible feeling, judging strictly by that "in particular," that statistics has just discovered Convexity.
Can acceleration be far behind?
Post a Comment
Can acceleration be far behind?
<< Home
