Over at the TLS (enjoy Mr. Murdoch’s free content while you can), Stefan Collini lays into the successor to the RAE, the REF. His ire is mainly reserved for the new assessment category of ‘Impact’ which, under the terms of the consultation document, would amount to 25% of each unit of assessment’s score.
The article has elicited a predictable chorus of approval from arts and humanities academics. I can’t help agreeing, though, with my former Liverpool colleague Phil Davis that there is nothing particularly wrong with assessing ‘impact.’ The problem is more that this is currently skewed towards assessing the potential for economic exploitation rather than considering social value. Even so, two of the proposed ‘impact’ categories are ‘cultural enrichment’ and ‘social welfare’.
That said, Collini still misrepresents, or misunderstands, what the consultation document says. 60% of the REF score will still be based around an assessment of research ‘outputs’ (books and articles to you and me.) So Collini’s hypothetical colleague producing excellent work on Victorian poetry is unlikely to be thrown on the academic scrapheap just yet because their university will need his/her international quality research to stick on its return of outputs (which will count for most of its score anyway.)
Moreover, not everyone will have to produce evidence of the ‘impact’ of their research. This part of the REF will be assessed by case studies and, as I understand it, each unit of assessment will only have to produce one case study per 5-10 FTE members of staff. Neither will academics suddenly have to frantically search for money-making research ideas (TV/radio series, stocking-filler books, merchandising) because though ‘impact’ ought to have become ‘evident’ sometime between 2008 and 2012, the research that produced that impact could have been generated a while before the census dates.
I would suggest that the REF consultation document, though it could with revisions, not least in terms of broadening what ‘impact’ means, actually represents a fairer and more flexible way of assessing academics ‘research activity’. The downvaluing of the importance of bibliometric data ought to reassure many who feared that the assesssment of outputs was going to turn into a crude exercise in citation counting. Other elements of the REF terms of assessment strike me as better than those employed under the RAE. For example, unlike ‘esteem’, ‘impact’ is much less open to the manipulation of networks of academic patronage (Prof. Y invites Prof. X to give a paper at his/her prestigious international conference and the favour is duly reciprocated.) The readiness of REF panels to evaluate so-called ‘grey literature’ also broadens the category of what we can count as an output, giving value to academic contributions to non-academic publications. Again, this ought to reassure those worried that REF would given even more clout to a small number of internationally-respected, peer-reviewed academic journals.
A bigger question is whether we ought to have research assessment exercises at all, given that they predictably reward institutions with in-built structural advantages in terms of staffing and resources. (Chris Dillow put up a very good post on the value of ‘elite’ universities over here.)