Oi! REF!

Over at the TLS (enjoy Mr. Murdoch’s free content while you can), Stefan Collini lays into the successor to the RAE, the REF. His ire is mainly reserved for the new assessment category of ‘Impact’ which, under the terms of the consultation document, would amount to 25% of each unit of assessment’s score.

The article has elicited a predictable chorus of approval from arts and humanities academics. I can’t help agreeing, though, with my former Liverpool colleague Phil Davis that there is nothing particularly wrong with assessing ‘impact.’  The problem is more that this is currently skewed towards assessing the potential for economic exploitation rather than considering social value. Even so, two of the proposed ‘impact’ categories are ‘cultural enrichment’ and ‘social welfare’.

That said, Collini still misrepresents, or misunderstands, what the consultation document says. 60% of the REF score will still be based around an assessment of research ‘outputs’ (books and articles to you and me.) So Collini’s hypothetical colleague producing excellent work on Victorian poetry is unlikely to be thrown on the academic scrapheap just yet because their university will need his/her international quality research to stick on its return of outputs (which will count for most of its score anyway.)

Moreover, not everyone will have to produce evidence of the ‘impact’ of their research. This part of the REF will be assessed by case studies and, as I understand it, each unit of assessment will only have to produce one case study per 5-10 FTE members of staff. Neither will academics suddenly have to frantically search for money-making research ideas (TV/radio series, stocking-filler books, merchandising) because though ‘impact’ ought to have become ‘evident’ sometime between 2008 and 2012, the research that produced that impact could have been generated a while before the census dates.

I would suggest that the REF consultation document, though it could with revisions, not least in terms of broadening what ‘impact’ means, actually represents a fairer and more flexible way of assessing academics ‘research activity’. The downvaluing of the importance of bibliometric data ought to reassure many who feared that the assesssment of outputs was going to turn into a crude exercise in citation counting. Other elements of the REF terms of assessment strike me as better than those employed under the RAE. For example, unlike ‘esteem’, ‘impact’ is much less open to the manipulation of networks of academic patronage (Prof. Y invites Prof. X to give a paper at his/her prestigious international conference and the favour is duly reciprocated.) The readiness of REF panels to evaluate so-called ‘grey literature’ also broadens the category of what we can count as an output, giving value to academic contributions to non-academic publications. Again, this ought to reassure those worried that REF would given even more clout to a small number of internationally-respected, peer-reviewed academic journals.

A bigger question is whether we ought to have research assessment exercises at all, given that they predictably reward institutions with in-built structural advantages in terms of staffing and resources. (Chris Dillow put up a very good post on the value of ‘elite’ universities over here.)

Published in: on November 26, 2009 at 1:07 pm  Comments (3)  
Tags: , , ,

The URI to TrackBack this entry is: https://edwardvallance.wordpress.com/2009/11/26/oi-ref/trackback/

RSS feed for comments on this post.

3 CommentsLeave a comment

  1. Interesting piece Ted. But in the same breath as you provide welcome corrections to Collini (and other, still more alarmist and head-in-sand voices from the humanities) a series of pretty fundamental problems remain.
    The first is basic and concerns the lack of a definition of ‘impact’. They don’t even say whether the impact should be positive. I could write an article recommending the castration of sexual deviants, and if my ideas were taken up, the article could reap an excellent impact score, irrespective of the ethics.
    A second one is also pretty basic. While you see impact as a healthy antidote to ‘networks of academic patronage’, don’t they put in place other networks of non-academic patronage? For instance, at the moment I can choose to study ‘obscure’ subjects. I do this not because I’m an obscurantist but precisely because I think they should be less obscure, and I hope the existing dissemination arrangements (which are being treated as if they counted for zero, despite the fact that more and more people are reading scholarly journals these days), will help convey my findings to the world at large. But if I start to do research under an impact-driven agenda, I may just give up finding out interesting new stuff, and just satisfy myself with recycling the familiar, at the behest of HEFCE, or of TV producers.

  2. As to your castrated sexual deviants, Alex, I think someone would be asking hard questions about why/how this had slipped past your own university’s ethical standards approval process.

    Or are you thinking of becoming the new Melanie Phillips?

    But I take the point that ‘impact’ is far from perfect. On the other hand, most of the reward and perhaps more than 60% in the end will still go to disseminating research ‘inside’ academia. (Although again, I take the point that it’s not just academics who read academic journals.)

  3. Ethics committees? They only stop people doing stuff, according to bureaucratically prescribed norms. They never interfere with people saying stuff (and if they did there could be free speech issues)…
    By the way, an interesting discussion here:

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: