In this sequel, I want to tell a story which perhaps indicates that the philosophical positions of scientists and statisticians have less influence on what really goes on than much more practical things like publication policies of journals. This story concerns a paper I saw nearly 50 years ago. I hope that this sort of thing couldn’t happen now, but I wish I felt more confident about that.
The paper was in a journal of animal behaviour, I really don’t remember which one but I think it was a “reputable” journal. I haven’t tried to locate it, so everything I say is from memory and thus unreliable. But back then, you needed to have found something statistically significant at the 5% level at least (that is, p ≤ 0.05) to have any hope of getting your work published.
Anyway, the scientist had observed baby birds for the first fifteen days of their life, and had measured some attribute (I don’t recall what) each day. To analyse the results, the value of the attribute on the i-th day was compared with the value on the j-th day, for all pairs {i,j}. One of these pairs of values was found to be significantly different, and on the strength of this significant result, the paper was accepted for publication.
Now I think it hardly needs saying, but apparently it escaped the authors, that the real discovery they had made is that only one pair of results are significantly different, when I would have expected five!
