Dear Dr Harnad,
You were kind enough to visit my blog yesterday and comment on my musings on citations, giving details of some of your papers on validation of metrics. I tried to read your paper on “Validating research performance metrics against peer review rankings”.
Unfortunately I choked on the first sentence of the second paragraph, where you claim that metrics provide objective, and peer review subjective, evaluation of research. (Your italics.)
Of course peer review is subjective (like everything we do, from marking students’ coursework to deciding their class of degree). However, to say that metrics give objective evaluation of research is like saying that phrenology (measuring bumps on the skull) provides objective evaluation of tendencies to criminality or lunacy.
In fact it is worse. Nobody can disagree with the actual measurement of a skull bump. But even your input data is subjective. A paper cannot be cited until it is published, and this depends on the subjective judgment of referees. Maybe it hasn’t happened to you, but most researchers can tell stories of good papers that languished because of bad referees’ reports. I had one which was never published, and the result was rediscovered more than thirty years after I had abandoned hope of getting it into print.
Not only that, but your criteria are not as objective as you make out, either. I urge you to read a paper by Adler, Ewing and Taylor (and several comments) on “Citation statistics” in the journal Statistical Science 24 (2009), 1-28. Professional statisticians have serious concerns which need to be addressed. I feel that you would agree with much that is said there.
One of my pet hates in our current Newspeak is the word “transparent”. Some people think it means “fair”. Others think that if they use it often enough they will persuade the rest of us that it does mean “fair”. Perhaps you should have said “transparent” rather than “objective” in the opening sentence of your paper. This touches on the real problem with metrics, of course: they become targets, as we have seen with school league tables.
Nobody should doubt that the bean-counters want to introduce metrics. We must address the issues now.