Revised evidence for statistical standards
Résumé
Johnson proposes replacing the usual P = 0.05 standard for significance with the more stringent P = 0.005. This might be good advice in practice, but we remain troubled by Johnson's logic because it seems to dodge the essential nature of any such rule, which is that it expresses a tradeoff between the risks of publishing misleading results and of important results being left unpublished. Ultimately such decisions should depend on costs, benefits, and probabilities of all outcomes.