The United Kingdom Research Assessment Exercise 2001
There can be few academics in any discipline in UK higher education who are not anticipating the end of 2001. Their anticipation will arise from the fact that the outcome of the current RAE (Research Assessment Exercise) will be published during December. However, the concern in academic nursing circles will be considerably higher than in other disciplines because this is the RAE in which nursing has to do well. As was recently stated in a report from the University of East Anglia (Lindsay et al. 2001):
A third successive appearance at the bottom of the RAE ladder may have repercussions well beyond the career pathways of individual researchers, or the reputation of individual university departments of nursing and midwifery. Nursing at the beginning of the 21st century is, as it seemingly has been for years, at a time of upheaval in its education programmes, its place in health care, its relationships with other health professions and its place in the hearts and minds of the public. If nursing cannot demonstrate an ability to provide the evidence-base for its own practice, it will be difficult to assert its autonomy and status within a modern health service.
In previous RAEs (1992, 1996) nursing came bottom with the lowest median score, on a scoring system which ranges from 1 (low) to 5 (high), across the board compared with all other disciplines, or units of assessment (UoAs) as they are termed officially (Watson 1997). In 1996 only King’s College London received a top score of 5, and two (Manchester and Surrey) received a score of 4 with the remaining departments scoring 3 or less.
The methodology of each RAE has changed so that there is no method of ‘playing the game’ in the same way each time and the scoring system has also been changed. This was probably most marked between 1992 and 1996. Unlike other subjects, nursing has also not collaborated well across the discipline. The relatively recent historical establishment of some nursing departments before the vast majority was formed in the past decade, has led to an apparent reluctance to share knowledge about the RAE. One could argue that this is a classic example of nursing acting politically at the individual department level but failing collectively for the discipline as a whole.
The RAE is normally a four yearly return of research activity by UK universities, which is sponsored by the respective higher education funding bodies in the four countries (England, Scotland, Wales and Northern Ireland) that make up the UK. The latest RAE 2000 was postponed until 2001 in order to ensure that millennium celebrations and millennium bugs were well out of the way. The RAE recognizes almost 70 UoAs of which nursing is one. The RAE currently consists of a return of research activity in terms of publications, research income spent, postgraduate student activity: number supervised and number graduated, and is accompanied by a textual submission the length of which depends upon the number of people being returned. Those ‘returned’ by departments are academics who are considered to be research active. Departments are not obliged to return all their members therefore an element of selection takes place. This is not just a problem for members of nursing departments, as is illustrated by a report in The Times Higher Educational Supplement (THES) (Goddard 2001) that showed that young researchers and women are being excluded disproportionately from the current RAE, to the detriment of their future careers. The report indicates that the number of staff classed as ‘research active’, together with the total number of submissions, are both down in 2001 by approximately 10% on the 1996 returns. Unions are reported to fear that the exercise this time will polarize teaching and research. Of course, if women are disproportionately excluded, then nurses probably are as well. Perhaps the most scandalous fact reported by THES is that some departments have not even told their staff that they are being excluded, leading one researcher to use the Data Protection Act in an attempt to identify any data held on him. One can only imagine the demoralization amongst nursing departments if, as a result of the 2001 RAE, many find themselves classified as ‘teaching only’ departments.
The scoring system for the RAE is currently a seven-point scale on which the international standing of the research activity in departments is taken into account: the greater the international standing in the eyes of the RAE panel members, the higher the score. It should be noted as mentioned above, however, that the fine detail has changed with every RAE, leaving academic staff to deal with ‘constantly moving goal posts’ in terms of the criteria used for assessment.
The exercise is maligned by some academics (Flowerdew 2001); it is certainly very time consuming. On the other hand, there appears to be a fatalistic acceptance by many academics that when vast sums of public money are spent on research, the outcomes should be monitored and that the ‘changing goal posts’ represents an attempt to improve the system.
For anyone not familiar with the RAE it requires an explanation. Rating research activity along the lines described above may seem initially to be quite straightforward. Just add up all the publications from a department, attribute them to the staff responsible and then make a comparison between departments. In fact, this is not how the system functions. Individuals who appear in the RAE submission for a UoA are allowed to submit a maximum of four pieces of evidence (usually, but not exclusively, peer reviewed research papers) in support of their academic department’s RAE submission. This takes care of the ‘salami slicers’ who eke out publications from projects when one significant paper would suffice. As mentioned above, departments are not obliged to return all members of staff: they may be selective and return only those who are likely to bring glory in RAE terms. This has the effect of levelling the playing field significantly and allows small departments to compete with larger ones.
An additional factor is that the amount of money subsequently gained by a department as a result of the assessment – and the exercise is about the disbursement of central UK government university research funds – depends on the percentage of academic staff returned. Therefore, a return of 10 individuals from a department of 50 people may be worth less than a return of nine people from one of 10 people, although in the final calculation the absolute numbers being returned are taken into account when money is allocated. The THES report referred to above cites the case of Oxford and Cambridge to illustrate the point:
In 1996, Oxford submitted 90% of its staff for assessment, 75% of whom attained the top 5* grade; the University of Cambridge submitted 98% of its staff, 62% of whom attained a 5* grade. Despite receiving cash for fewer staff, Oxford gets more research money from the funding council than Cambridge (Goddard 2001, p. 6).
It should be noted that considerable confusion still reigns over how the percentages of academic staff will be calculated for nursing departments with large numbers of academic-related staff (Sanders 2001).
Another aspect of the RAE is that research income expenditure as opposed to just research income is the financial indicator that is taken into consideration. The research income expenditure recorded in the RAE submission must be attributed to individuals who are being returned as being research active. Not all money is considered equally; research council money and income that has been gained through national competition is rated more highly than other funds. Again, this levels the playing field considerably: if large departments with many research active staff were allowed to record their total research income and not, necessarily, attribute it to those who were being returned then they would have an unfair advantage over small departments with fewer research active staff. The RAE is not about overall amounts and overall totalled up achievements, but about what departments are doing with the resources available to them, how they are creating and sustaining a research culture and how this makes a difference on the national and international scene. In the case of nursing, whether or not it helps anybody at the end of the day is also taken into account. Nowhere is this more apparent than in the textual commentary that accompanies the ‘harder’ details of the RAE. Whilst possibly the hardest piece of writing that most academics responsible for the RAE have had to do in the past 5 years, this part of the submission allows departments to explain what it is they are doing, what it is that they have achieved, and what they hope to achieve before the next RAE takes place in four year’s time. In response to complaints that there was insufficient guidance on how to make a proper case – especially in nursing – the nursing RAE panel provided comprehensive guidelines on what was expected this time around (Bond 1999).
The question being asked, inside and outside of nursing, is whether or not nursing will perform better in the 2001 RAE. Clearly, in nursing we hope that this will be the case. There is more than money at stake; reputation and even our place in higher education are also in the balance. Currently nurse education is a significant source of income for institutions of higher education. However, this will not be enough to ensure the survival of nursing in higher education as a research-based discipline if research performance continues to be poor. Unfortunately, the indications of how nursing might fare are not all promising. According to figures from the RAE administration, 10 more departments of nursing have indicated an intention to submit in 2001 than in 1996 and the additional staff to be entered averages 30 per additional department. Of course, these are only provisional indicators and the departments in question were not obliged to follow through and submit. Nor were any departments obliged finally to submit the number of staff originally indicated in provisional returns. The personal disadvantage experienced by individual nursing staff if their departments play this political game shrewdly has been illustrated above. How these tensions can be reconciled when the majority of British academic nurses are such late entrants to the University sector is not at all clear.
What is clear is that for nursing as a political collectivity, a strategic approach should be taken to the RAE. A smaller, high quality return of staff is likely to yield a higher score than a large return of poor staff – yet the latter is what is being indicated by the above figures for nursing; at least for some departments. It has been indicated by some departments that they are going to make an RAE return this time round just to ‘get involved’, and the numbers purporting to be returned by those with little or no research reputation is alarming. Any department submitting under that premise may do themselves no good and they may help to maintain the place of nursing at the bottom of the RAE league table. Ironically, it will do nothing to help individual research inactive staff either.
The RAE returns are now in the hands of the assessment panels submitted at the end of April 2001. The process is one of peer review and, despite a rigorous process of evaluation and assurances that nursing is not considered to be less worthy of proper research evaluation than other disciplines, a peer review process has greater degrees of freedom than a system that simply considers objective criteria. Much will depend upon how panel members evaluate individual pieces of evidence – the research papers and other submissions, which have been returned – and the impact of the textual submissions is, largely, an unknown factor.
In common with previous RAEs the results of the 2001 RAE will generate a number of analytical papers and commentaries (Robinson 1993, Tierney 1994, Kitson 1997). Given the copious guidance and the predominant culture of selectivity that has been generated in the run up to the current RAE, if nursing remains at the bottom of the RAE league table we will only have ourselves to blame.