Today I had the unpleasant job of assembling some data on citation of my colleagues’ recent publications for the College bureaucracy. Typically, the job was not very precisely specified.
The first alarm bell was “This can easily be extracted from PubLists”. Now PubLists is a very new system the College has installed for monitoring our pubications data. There are several reasons why it is not very good for mathematics (these surely apply to other subjects too).
- The main problem is that it is an automated system which accepts feeds from three sources: Web of Science, PubMed (a database of medical publications), and the arXiv, a world-wide preprint server which specialises in maths and physics. Now anyone can put their papers on the arXiv; this most definitely does not constitute publication, and no research assessment will accept these as being the equivalent of published papers.
- Researchers get the chance to inspect the lists the computer has compiled of their papers and make corrections. But it is early days yet, and we are not all in the habit of doing so regularly. If papers are not there (not an uncommon occurrence), we can enter them manually; but this is a lot of trouble, and most people don’t bother.
- Finally, it accepts papers as soon as it finds them, before there is time for any citations, even in the most frenetic parts of the academy.
There is a much more reliable source of information about mathematics publications, namely MathSciNet. The editors of this wonderful resource have done an extremely thorough job of giving every publishing mathematician an author ID; the risk of misattribution is essentially zero. (None of the sources used by PubLists has such a safeguard.) Also, it scans proper publications, not the arXiv, and the papers it reports certainly meet any reasonable criterion for being published. However, despite my representations to the PubLists administrators, this source of information has not been used.
The next problem is “Which papers?” Anybody who has thought, even briefly, about the use of citation data realises that there are big differences between fields. Even within the limited sample of pure mathematicians at Queen Mary, it is clear from the data that in some areas, many more citations are routinely given than in others; citation numbers in mathematical physics are not comparable with those in algebraic topology. Also, it is widely realised that citations in pure maths occur at a slower rate than in many other fields. The standard 1- to 2-year window used by the citation index is quite inappropriate. (Among Queen Mary pure mathematics papers, only one or two published within the last three years have had any citations at all!) I have ranted on this point here before and don’t intend to rehearse the arguments.
A further problem is coverage. Among the people on the list, a couple are statisticians, one is a linguist as well as a logician, and one is a cosmologist as well as a pure mathematician; much of their work is published (and, more importantly, cited!) in journals which are not covered by MathSciNet. A further problem is that one of my colleagues produces packages for algebraic and combinatorial computing which are used all over the world. But of course the packages and documentation are not typically covered by MathSciNet, so much of his best work is completely invisible to the citation system.
Despite this, I decided to proceed, taking my cut-off as the date of the previous Research Assessment in 2001. There was no sensible way to deal with the missing citations in other fields. I could at least produce a list, and possibly get the bureaucrats off our backs for a short time.
The really depressing part of the exercise was the results I obtained. I have a fair idea of what my colleagues’ best work is. The fact that I was on the RAE panel in 1996 and 2001 should be evidence that I know how research is judged by experts. It became clear that the papers which were being produced by the citation lists were rather poorly correlated with any reasonable list of our best papers since 2001.
My final feeling was this. I am very sorry to have to report this; I never thought that I would come to this conclusion about the profession I have spent my life in. I am less than three years away from retirement, and by the time of the next assessment (the absurdly-titled “Research Excellence Framework”) I will be gone. Somebody else will have to do this nonsense. Worse, others will have to suffer from the fact that such nonsense is being done.