Research performance of universities has soared over recent decades
Opinion: ‘Innovation index’ measurement has some methodological flaws
A high quality innovation and research programme will be vital to Ireland’s recovery.
In recent weeks the quality of research performed in Irish universities and in particular the attitude of businesses to such research has been called into question. The trigger for this was a report on August 12th in the Times Higher Education (THE) magazine announcing the results of its World Academic Summit Innovation Index.
The index ranks 30 nations on the ability of their universities to “attract funding from companies in order to carry out work in innovation and research on their behalf”.
South Korea came out in pole position and Singapore in second place. Ireland’s ranking at 30, however, attracted much negative coverage here, with extreme headlines such as “Irish universities are the worst in the world at attracting business funding”. How seriously should we take these results and conclusions?
For many of the core activities of universities it is difficult to measure quality directly, so rankings typically use various indirect or proxy quantitative metrics. One is the “innovation score” which claims to capture a university’s “ability to engage in knowledge transfer” by dividing the research income an institution earns from industry by the total number of academic staff it employs.
This year the THE report extended the use of this innovation score to provide a national “innovation index”. But on analysis there appears to be some serious flaws in the approach.
Firstly, income from industry alone is not a valid measure of the innovation contribution of a university. This approach ignores completely other key elements of technology transfer such as patents, licence deals and spin-outs.
In addition, some higher education institutions are inherently more focused on science and technology than others. So if the metric used is industry income per academic staff it is clearly inappropriate to treat comprehensive universities with large faculties of humanities, for example, in the same manner as more technology-focused institutions. A similar scale of industry investment will yield vastly different scores for two such institutions.
Specific cases in the innovation index also raise questions about the effects of including certain institutions and excluding others. India, which is ranked at number 10, has more than 550 universities yet none were included in the national calculation for India. Instead data from only three Indian institutes of technology were used. This is not a valid basis for characterising a national system.
South Korea’s top rating came through processing data from just six universities, including SKKU, which is the corporate university of Samsung Corporation and Pohang University of Science and Technology, which receives massive funding from its founding industry partner Pohang Iron and Steel Co, the world’s largest steelmaker. Such extreme examples of industry engagement clearly distort the analysis, especially when a crude single metric is used.
From these and other system flaws one is forced to conclude that the THE innovation index does not provide nuanced information about innovation and cannot form a valid basis for international comparisons of the degree of industry-academic research collaboration.
How has Ireland fared in other more robust comparisons?