Research jobs: how good is the training?

Authors


Mrs E. Hawkins, Harold Wood Hospital, Gubbins Lane, Romford RM3 0BE, UK.

Abstract

Objectives  To determine if a period in a research job improves confidence in research skills and to assess the quality of research training.

Design and setting  A questionnaire was designed and piloted to assess the content, structure and process of research training.

Population/Sample  All individuals who had passed MRCOG in the five years prior to 1999 and were residents of England or Wales.

Methods  Confidence scores were compared between those with and those without research experience. The availability and value of differing strategies for research training were compared.

Main outcome measures  Confidence in research skills and attitudes to training.

Results  Of the 532 usable questionnaires returned, 226 respondents had done or were doing research and these individuals had higher confidence on a variety of research skills than those with no experience of a formal research job. Confidence was patchy with less than 50% feeling confident at assessing bias in a case-controlled study, understanding the statistics used in a paper or assessing the power of a study. Of those who had done research, 50% or less felt their training had been good or excellent in any area. Self-directed learning and discussion with peers were felt to be the most useful strategies for research training. Short intensive courses were not available for many respondents, but were felt to be useful.

Conclusion  The high levels of dissatisfaction with the training in key skills required for research suggests that there is a need for a system for recognition of research posts. Reform of training in the research job should build on the current strength of encouraging self-directed learning.

INTRODUCTION

In common with other specialties, a period in research is considered essential for training in obstetrics and gynaecology.1,2 It is thought that research increases trainees' ability to critically analyse the literature,3 provides an in-depth understanding of a particular subject and improves confidence in analysing other people's results and conclusions.1,4 The clinical benefits of research training are less clearly articulated, but it is thought that research improves clinical reasoning and encourages lifelong learning.3 However, others feel that enforced research is a waste of research funds and is responsible for the large number of poor quality studies that are published.5 The assumption that clinicians do learn research methodology from undertaking research has been questioned,6 and the high rate of errors in published research7,8 suggests that learning research methodology through ‘doing’ may perpetuate errors.

Previous surveys have suggested that 76–90% of consultants and trainees in surgical specialties feel that research is beneficial. These studies have focussed on the global benefits of research and none has evaluated the extent to which research increases skills required for critical analysis.9–13 One previous study reported high levels of dissatisfaction with the process of research training, however, only 49 respondents were in obstetrics and gynaecology.14

Thus, despite the substantial commitment that many trainees make to research, its role in training in obstetrics and gynaecology has not been evaluated. The aims of this study were to determine if time spent in research improves confidence in research skills and comprehension of terms used in research papers. In addition, we wished to explore the views of those who had done or were doing research on their training in research methodology.

METHODOLOGY

Following a review of the literature, a questionnaire was designed focussing on the content, structure and process of research training, with respondents completing different sections depending on whether or not they had research experience.14 After piloting, the questionnaire was modified by clarification of ambiguous questions, and removal of questions that were poorly discriminatory or redundant. Content validity was established by telephone interviews with a sample of the pilot group. Concurrent validity was assessed by comparing the responses to the questions on research confidence with responses to a previously validated question on research training.15 In addition, the questions on research confidence and understanding of technical terms in evidence-based medicine showed a high degree of internal consistency (Cronbach's alpha 0.91). A copy of the final questionnaire is available from the author on request.

The questionnaire asked respondents to rate their agreement from strongly agree to strongly disagree on a variety of questions exploring the main purported benefits of research, the role of research in developing clinical skills and its importance in career progression.5,16,17 The extent to which specific research skills were learnt was determined by asking the respondents to rate their confidence on a five-point scale from ‘extremely confident’ to ‘not at all confident’ on key research skills18–20 and to rate their understanding of technical terms used in evidenced-based medicine.21 Confidence in research skills was compared between those who had done or were doing a research job and those who had no experience of a research job.

Those who had done, or were doing, a research job completed a second part of the questionnaire that explored the characteristics of their research, the extent to which the trainee contributed to the study and the time constraints that impinged on their research. They were asked to rate their training from ‘excellent’ to ‘no training’ in a variety of topics that had previously been identified as important.16 Possible preferred learning resources were identified from the literature5,16 and respondents were asked to rate these from ‘very useful’ to ‘not available’.

The final questionnaire was posted to all individuals who had passed part 2 MRCOG in the last five years and were currently resident in England or Wales. Second questionnaires were sent to non-responders one month later.

Responses to questions were entered onto Microsoft Access Database and analysed with SPSS version 9 for Windows. Dichotomous data were analysed with Fisher's exact test, Mann–Whitney U test was performed to compare groups with non-parametric data and the t test was used to compare normally distributed data. Multiple regression was used as appropriate to control for confounding between variables.

RESULTS

Questionnaires were sent in July 1999 to 1062 people, from whom 532 useable questionnaires were returned (50% response rate). Three hundred and seventeen of these questionnaires had been returned on the first mailing and 215 on the second mailing. Those who returned a late questionnaire were more likely to be in a non-training grade and to have qualified for longer than those who returned their questionnaire in response to the first mailing. There were no other differences between late and early responders.

Table 1 compares the characteristics of the 226 respondents who had done or were doing research with the 306 respondents with no experience of a formal research job, while Table 2 shows the respondents opinions about the potential benefits of a research job.

Table 1.  Characteristics of those who have done or are doing research compared with those with no research experience. Values are presented as n (%) or median [range].
 No research (n= 306)Research (n= 226)
Current career grade
Consultant in obstetrics and gynaecology5 (24)16 (76)
Specialist Registrar/lecturer196 (55)161 (45)
Research fellow 23 (100)
Senior House Officer21 (84)4 (16)
Locum33 (87)5 (13)
Staff grade/trust doctor35 (81)8 (19)
Other16 (64)9 (36)
 
Planned career intention
Academic consultant in obstetrics and gynaecology16 (29)39 (71)
Consultant in obstetrics and gynaecology211 (56)167 (44)
Staff grade/associate specialist in obstetrics and gynaecology35 (88)5 (12)
Other44 (75)15 (25)
 
Time since qualification (years)11 [3–29]10 [4–25]
 
No. of papers published
0156 (51)31 (14)
1–6138 (45)130 (57)
7 or more11 (4)65 (29)
Table 2.  Number of people agreeing or strongly agreeing with statements about the benefits of research (responses ranked according to strength of agreement; Freidman's test). Values are presented as n (%).
 Have research experience (n= 226)No research experience (n= 306)Total (n= 532)P value (Mann–Whitney)
Research improves doctors ability to critically analyse the literature199 (88.8)247 (81.2)446 (84.5)0.006
Most people do research so they can produce publications181 (80.4)255 (83.9)436 (82.4)0.031
There is a risk of losing clinical skills during a research job158 (70.2)226 (74.1)384 (72.4)0.9
Research jobs allow people to gain additional clinical skills163 (72.4)180 (59.0)343 (64.7)<0.001
Doing a research job is essential for career progression132 (58.7)175 (57.4)307 (57.9)0.51
The main benefit of doing research is to obtain a research qualification126 (56.3)172 (56.9)298 (56.7)0.94
People who have done research are better able to integrate research findings into clinical practice136 (60.4)113 (37.1)249 (47.0)<0.001
Research improves general problem solving skills113 (50.7)93 (30.6)206 (39.1)<0.001
Research jobs are the best way to develop specialist knowledge80 (35.6)71 (23.4)151 (28.5)0.005
Research improves peoples ability to solve complex clinical problems74 (33.0)70 (22.9)144 (27.2)0.01

Figure 1 shows the responses to the questions on confidence in research skills. Responses to this question were scored out of five and combined with a question on comprehension of technical terms (scored out of four), to give an overall mean confidence score. The possible score ranged from 1.00 to 4.61. Those who had no experience of a research job had lower confidence scores (mean 3.01, 95% confidence interval [CI] 2.95–3.07) than those who had done or were doing research (mean 3.52, 95% CI 3.47–3.59). Using multiple regression to control for confounding between factors, increasing confidence also correlated with male sex, increasing publications and planning a career in academic obstetrics and gynaecology.

Figure 1.

Responses to question on confidence in research skills comparing those with some experience of a research job and those without.

Of those who were doing research, 20 had less than six months experience, they had a mean confidence score of 3.42 (95% CI 3.20–3.64), this was not significantly different to the confidence scores of the 119 individuals who had at least two years experience (mean 3.57 95% CI 3.48–3.65). Of the 218 who responded to the question on the duration of their research job, 193 (88%) had spent or expected to spend at least a year in research. The median expected duration of research was two years (range 6 months–30 years).

Of the 226 people who had some research experience, 140 (61%) had completed their research job, 82 (39%) were doing a research job and 4 did not answer the question. Those who had completed research had done so a median of two years previously (range 1 month–20 years). The 15 people who had completed their research more than six years previously were excluded from the remainder of the analysis. When asked to classify their field of research, 137 selected a field in gynaecology and 65 a field in obstetrics, with two not answering the question and seven performing research outside of the specialty. The most popular field of research was reproductive medicine (27%) with intrapartum care and minimal access surgery as the least popular (each 3.8%). In all subjects except oncology research was clinical rather than laboratory based.

Respondents' views on the training they received are summarised in Table 3. The most useful strategies for research training were self-directed learning and discussion with peers, found to be very or reasonably useful for 162 (78%) and 155 (75%) of respondents, respectively. Formal tutorials with the supervisor were not available for 55 respondents (27%), however, when they were available, 94 respondents (62%) found them very or reasonably useful. More respondents (181) felt that informal teaching was available from their supervisor and it was reasonably or very useful for 121 (67%) of these. Short intensive courses in research methodology were only available for 86 respondents of who 62 (72%) felt they were useful. On the other hand, attending lectures was available for 141 respondents but only 63 (44%) thought they were useful.

Table 3.  Respondents' rating of their training in research skills. Values are presented as n (%).
 Rating of trainingTotal responses
Good or excellentFair or poorNo training
Laboratory or clinical techniques needed for the project101 (50)72 (36)28 (14)201
Reviewing the literature105 (50)69 (33)34 (16)208
Handling and storing data87 (42)85 (41)35 (17)207
Formulating a research question83 (40)94 (45)30 (14)207
Producing a thesis69 (35)82 (42)46 (23)197
Research ethics69 (33)98 (47)41 (20)208
Methods of statistical analysis58 (28)105 (51)44 (21)207
Writing grant proposals47 (23)82 (40)74 (36)203

Overall, respondents were pleased with their supervision, with 132 identifying some aspect of their supervision as being particularly helpful. Supervisor approachability and ease of contact were the most commonly appreciated aspects of supervision. However, 92 people felt that some aspect of their supervision seriously impeded progress. Most commonly, this was due to lack of availability of one or more of the supervisors, however, other issues such as poor structure of supervision, excessive workload, conflict between supervisor and trainee and a culture of criticism were also identified.

Stepwise multiple regression was used to determine the factors that might effect the rating of research training. A high mean training score was correlated with confidence in research skills, little interference in time due to administrative duties, but more interference in time with grant applications, increasing number of supervisors, increasing time with main supervisor, increased quality of supervision score and with finding tutorials with supervisor and discussions with other senior staff useful in learning about research methodology.

DISCUSSION

The low response rate was disappointing, and clearly is a flaw in this study. Extrapolating from the late responses suggests that there was a bias in response towards those who were still in the training grades and were more recently qualified. Length of the questionnaire may have been one of the reasons for the poor response rate. However, a similar poor response rate was noted in the recent survey of training.2

Career advancement has been identified as the main motivation for surgical trainees undertaking research.22 While high levels of agreement with questions relating to extrinsic motivation for research such as the production of publications suggests that this may also be true in obstetrics and gynaecology, it was encouraging to see that the greatest agreement was that research improved ability to critically review the literature. Although there was strong agreement that time spent in research might result in a loss of clinical skills, there was also agreement that it could provide an opportunity for gaining specialised skills. However, the most popular subspecialty for research was reproductive medicine, with intrapartum research being undertaken by a tiny minority. This is clearly contrary to the current drive to increase consultant involvement on delivery suite. This raises the possibility that continuing to use research as a necessary hurdle for career progression is resulting in individuals developing a special interest area that is not reflected in job opportunities or the needs of the NHS.

In this study, there was an association between spending time in a research job and improved confidence with research skills. The degree of confidence is patchy, with less than half the respondents feeling at least moderately confident at assessing bias in a case–control study, understanding the statistics used in a paper or at judging the power of a study. It could be argued that there is no need for medical researchers to understand statistics, as this is the role of professional statisticians. Yet people who had done research were more confident in performing statistical analyses than they were in understanding statistics as reported in literature. This may be a chance finding, or may be a reflection of the ease with which statistics can be produced from a computer program, without an understanding of the underlying assumptions.

There was particularly poor confidence in dealing with the ethical issue of fraudulent behaviour in a colleague. In 1991, the Royal College of Physicians suggested that each university department should nominate one person to receive allegations of scientific misconduct and that the name of this person should be publicised.19 Clearly, respondents to this questionnaire were unaware of these recommendations. In Britain, there is no independent investigating authority to police the scientific community.23 Thus, the lack of confidence in these issues may simply reflect general confusion within the scientific community.

There is a poor correlation between responses to self-completed questionnaires and behaviour.24 However, it is unlikely that respondents under-estimated their confidence in research skills. The relatively high confidence score of those with six months or less experience suggests that, either research skills can be obtained without the commitment of a two-year project, or that those who are successful in obtaining a research post are more confident at research techniques. Similarly, the more positive attitude of those who were doing or had done research may be a reflection of the reasons they chose to do it or a result of the research experience.

The most striking finding of this study was the high percentage of respondents who felt their training in research skills was deficient or absent. This high level of dissatisfaction was very similar to that found by Gale.14 Self-directed reading and discussion with peers were the most useful strategies for research training. Short intensive courses were useful when available, but were not available for the majority of respondents. Humanist theories of adult education emphasise the importance of self-directed learning,25 but respondents may not perceive it as a form of training. In a survey of postgraduate students at Reading University, most students identified working on one's own initiative and independent learning as important for doing a higher degree.26 Thus, if short courses in research methodology are introduced, they should be seen as a supplement to self-directed learning and not a replacement.

As in previous studies, supervision was found to be an important factor in determining satisfaction with research training.27 This study confirmed previous findings that frequency of meetings with supervisors and quality of supervision are related to satisfaction with training in research methodology.26 In general, approachability of supervisor was one of the most important factors in determining satisfaction with research training. However, a minority appreciated being allowed to work independently and for some respondents their supervisor's demands could be intrusive on the time available for their own work. There can be no set formulae for good supervision as there is considerable variation in the extent to which trainees depend on their supervisor during the research process.28

In conclusion, this study provides only weak support to the belief that spending time in a research job improves confidence in research skills. However, for many respondents, research jobs provided opportunities for reflection and personal development that were lacking in clinical jobs. There is an urgent need for reform of the system of training in research jobs, and consideration should be given to developing a system of recognition for such posts. Such reform of the research job should build on its current strength of encouraging self-directed learning.

Acknowledgements

This study was undertaken as part of an MSc in Medical Education at the University of Wales. During the early stages of the study, Mr Joseph Campbell gave invaluable assistance with the design of the questionnaire, and Mr Stephen Brigley gave advise and comments on earlier drafts of the paper.

The author would particularly like to thank all the respondents who took the time and effort to complete the questionnaire.

Accepted 27 March 2004

Ancillary