Source: ShanghaiRanking Consultancy, 2010.

Every year a world university ranking is released by a number of educational and commercial associations and bodies. And every year countless number of students, university officials and even politicians wait impatiently for them.

As one of the latest Nature Special Reports (2010) on educational topics reveal, these rankings have the power to put in action a whole mechanism. A position in the higher echelon of the rankings can influence policy makers higher education investments, determine which institutions attract the best researchers and students and trigger a race between universities to strengthen their position for the year to come.

Among them the most influential are the Academic Ranking of World Universities known also as Shanghai ranking and the World University Ranking released by Britain’s Times Higher Education Magazine in collaboration with QS ( a London base higher education media company).

While the Academic Ranking of World Universities was created in 2003 by the Center for World Class Universities of Shanghai Jiao Tong University in an attempt to benchmark Chinese universities against the top universities in the world, the World University Ranking released by the Times Higher Education Magazine is more the result of a commercial publishing exercise started in 2004.

Both however have attracted increased attention in the last few years, being as much appreciated by officials of universities that did well in the rankings as they were highly contested, probably by many more.

The criticism, which mainly comes from academics and universities, is directed towards the methodology and data gathering process used to compile these ranking systems. The main issues outlined are the narrow range of performance measures and criteria used which leaves out more than it includes.

The Academic Ranking of World Universities, compiled by Shanghai Jiao Tong University in China and generally known as the Shanghai index, focuses exclusively on research output and citation impact. The six performance measures used to calculate the index are:

• # Alumni winning Noble Prizes and Fields Medals

• # Staff winning Noble Prizes and Fields Medals

• # Highly cited researchers selected by Thomson Scientific

• # Articles published in journals of Nature and Science

• # Articles indexed in Science Citation Index – Expanded and Social Sciences Citation Index

• # Per capita performance with respect to the size of the institution

On the other side of the bar, the World University rankings compiled by the Times Higher Education Magazine are generally based on composite scores that aggregate weighted indicators, being based mostly on reputational surveys. The indicators used in the assessment process are:

• # Peer review score

• # Employer review score

• # Staff / Student score

• # Citation staff score

• # International staff score

• # International students score

Unfortunately, none of these rankings can clearly picture what is the real value and performance of world’s higher education institutions. While the Shanghai Index concentrate too much on academic research, this being the field which can be most easily monitored, due to its internationalized nature, the other one compiled by the British THE Magazine is focused too much on peer review and reputational surveys.

This lack of homogeneity creates big discrepancy between the two ranking systems which can be easily observed if looking at the rankings. Only few university keep the same position in the two ranking system, the most known one being Harvard University. To better picture these issue we have picked three examples from both rankings, for the year 2009.

The need for improved and more nuanced assessment which to include a wider range of criteria and to assess more intangible qualities, like educational excellence, which will also allow a higher degree of homogenization between rankings, making them more accurate and relevant seems evident.

Fortunately, a new generation of ranking systems that addresses part of these issues started or are about to emerge. The new systems, as acknowledged in the Nature (2010) article “ The nature game” will be more multidimensional , comparing universities less on aggregate numbers and more on aspects such as research, teaching and industrial engagement.

Firstly the World University Rankings compiled by the Times Higher Education Magazine, will have, starting with 2010, a new look and will be based on a totally overhauled performance assessment system. The rankings will be drawn from the Global Institutional Profile Project, launched by the most renowned data analyst and provider “Thomson Reuters” . It will include details such as research funding, number of researchers and PhDs awarded as well as measures of educational performance. The system will continue to use reputational surveys but the number of reviewers will be increased from 4,000 to about 25, 000 for better accuracy. An audit procedure to guard against misinformation provided by universities was also set in place.

European Commission has also launched a new project in an effort to create a global database of universities, called the U-Multirank (Global Ranking of Universities), which will be run by the Consortium for Higher Education and Research Performance Assessment. This new ranking system will allow the comparison of universities with similar activities, missions and research interests.

The Organization for Economic Co-operation and Development (OECD) is also trying to come into help, currently coordinating a pilot project for developing new metrics for assessing teaching and learning outcomes. The project is named the Assessment of Higher Education Learning Outcomes (AHELO) and is intended to try to measure complex aspects such as ability of students to think critically or number of new ideas generated by students.

Finally, the European University Association, which represents more than 800 universities is currently in the process of developing an annual review of all international rankings by assessing their methodologies and scrutinizing why institutions rise or fall in the rankings.

Hopefully all these new initiatives will create a better, more accurate and homogenized performance assessment  and benchmarking platform for universities and other educational bodies in the years to come.

For more information on world academic rankings you can fallow the smartkpis.com blog posts:

• smartkpis.com (2010), The 2010 Academic Ranking of World Universities in Economics / Business, available at http://www.smartkpis.com/blog/2010/08/19/the-2010-academic-ranking-of-world-universities-for-economics-business-subjects/, (accessed 20 August 2010)
• smartkpis.com (2010), The 2010 Academic Ranking of World Universities, available at http://www.smartkpis.com/blog/2010/08/17/the-2010-academic-ranking-of-world-universities-arwu/, (accessed 20 August 2010)

References

• Nature, Special Report (2010), University rankings smarten up, Volume 464, published on 4 March 2010, available at http://www.nature.com/news/2010/100303/pdf/464016a.pdf , (accessed 20 August 2010)
• Nature (2010), The ratings game, Volume 464, Issue 7285, published in 4 March 2010, available at http://www.nature.com/nature/journal/v464/n7285/pdf/464007b.pdf , (accessed 20 August 2010)
• Center for World-Class Universities of Shanghai Jiao Tong University (2010), ARWU 2019, available at: http://www.arwu.org/ARWU2009.jsp,  (accessed 20 August 2010)
• Times Higher Education (2010), World University Rankings 2009, available at http://www.timeshighereducation.co.uk/WUR.html , (accessed 20 August 2010)