It's time we started measuring third level research
Leftfield:Over the past couple of years there has been more scrutiny of what exactly academics do. Part of the problem in the assessment of what we academics do is that we seem to do lots of things. We teach, supervise, design courses, examine, sit on committees, hunt grant money, reach out to the community and so on. But in reality we do one thing: communicate knowledge, be it to the academy (research), students (teaching) or to business and society.
And here is where the problem is. How can we measure these activities so as to allow managers to allocate resources and to reward those who perform best?
We can measure business impact in a crude fashion by patents gained, monies raised and spinoffs created. We can measure teaching efficiency in a crude fashion by the hours an academic spends in class. The institute of technology (IoT) sector has a class-contact norm of typically 16 hours per week. That is not the case in universities, leading to the assumption that, as university academics teach less, they do less.
Nothing, alas, could be further from the truth. The key difference between the average IoT lecturer and the average university lecturer is the expectation and requirement of research activity.
Research is akin to venture capital. We need to do a lot to get a little output. Success is rare and failure the norm.
Acceptance rates (success) in decent journals or in gaining leading research grants are often less than 10 per cent. Research grants can take months to complete with no guarantee of success.
And doing a paper takes time. It takes typically in excess of 100 hours of work to get a paper to a stage where one is happy to put it out for even working-paper review. And then it takes a long time to get it published. The paper is usually under review at symposiums and conferences and further hundreds of hours are spent refining and tweaking.
Typically a paper gets published after a couple of anonymous referees have commented on it. Doing this referee job takes time – typically up to five hours reading and writing per paper. If one does 12 reviews a year, that’s 60 hours, or the best part of two working weeks. Imagine the work of an editor of a journal receiving 300 or more papers a year.
Teaching also takes time beyond the classroom. For every hour spent in the class you need three hours to prepare, review, reflect and redo the work. Even if one has a stack of slides and a good strategy, it’s imperative to do a mock runthrough (that takes about the same time as teaching), noting as one does what is outdated, what is wrong, what doesn’t flow and so on. So a 16-hour class contact in the IoT doesn’t leave a lot of time in teaching term to do any research. There are always evenings and weekends, but any IoT staff who wish to research in a significant sense use the summer breaks to do it – the Christmas and Easter are usually taken up with marking essays.
By all means let’s look at metrics of activity. But let’s recall that these metrics typically measure output, not input, and therefore can’t be used as such for the measurement of efficiency or, still less, effectiveness. That’s not to say we should not measure research output. We should.
We have experience in the UK of several generations of research measurement. Many of us have been involved in these as assessors or as units of measurement. We can and should design a model that builds on these and improves on them, that measures research output, across units and the sector. This will show that there are many who simply do not engage in research (as measured). Every academic knows of people who simply turn up, teach and disappear. This puts an unfair burden on those who do research, and it is they who should be shouting loudest for such a metric.
Until universities, the Department of Education and the Higher Education Authority determine where the balance should lie between research , teaching and other roles we cannot begin to reward or discipline academics for “success” and “failure”.
As it is, we are not measuring research in any manner in Ireland.
Perhaps we should start.
* Brian Lucey is professor of finance at Trinity College Dublin