New methodology for using incognito standardised patients for telephone consultation in primary care

Authors


Hay Derkx, Dr Paul Janssenweg 149, 5026 RH Tilburg, The Netherlands. Tel: 00 31 13 594 2155; Fax: 00 31 13 467 9012; E-mail: hay.derkx@medicinfo.nl

Abstract

Context  Many countries now use call centres as an integral part of out-of-hours primary care. Although some research has been carried out on safety issues pertaining to telephone consultations, there has been no published research on how to train and use standardised patients calling for medical advice or on the accuracy of their role-play.

Objectives  This study aimed to assess the feasibility and validity of using telephone incognito standardised patients (TISPs), the accuracy of their role-play and the rate of detection. Further objectives included exploring the experiences of TISPs and the difficulties encountered in self-recording calls.

Methods  Twelve TISPs were trained in role-play by presenting their problem to a general practitioner and a nurse. They were also trained in self-recording calls. Calls were made to 17 different out-of-hours centres (OOHCs) from home. Of the four or five calls made per evening, one call was assessed for accuracy of role play. Retrospectively, the OOHCs were asked whether they had detected any calls made by a TISP. The TISPs filled in a questionnaire concerning their training, the self-recording technique and their personal experiences.

Results  The TISPs made 375 calls over 84 evenings. The accuracy of role-play was close to 100%. A TISP was called back the same evening for additional information in 11 cases. Self-recording caused extra tension for some TISPs. All fictitious calls remained undetected.

Conclusions  Using the method described, TISPs can be valuable both for training and assessment of performance in telephone consultation carried out by doctors, trainees and other personnel involved in medical services.

Introduction

Standardised patients (SPs) are lay persons taught to portray patients in a standardised and consistent fashion.1 They have been used as ‘unannounced’ or ‘incognito’ SPs (ISPs) for many years to assess the performance of students and health care professionals during face-to-face consultations. Performance assessment is defined as measuring what the observed person does in actual professional practice when he or she is unaware of being assessed. There is evidence that the ISP methodology is a powerful instrument for assessing clinical performance.2–4

In many countries doctors or nurses handle telephone requests for medical advice at call centres during the out-of-hours period.5–11 Several studies have described the use of telephone ISPs (TISPs) to assess safety issues in this context,12–19 but have not assessed the accuracy of their role playing.

There are several differences between face-to-face and telephone consultations in terms of the use of ISPs. A TISP cannot see the call handler and therefore all communication is strictly verbal and does not include the usual multitude of visual cues. In addition, a TISP does not need to fill in a checklist after the consultation as the call can be recorded and the items to be assessed can be studied later. Therefore, a good-quality audio-recording of the telephone consultation is required to make proper assessments later.20 Telephone ISPs can read the role documentation and can refer to the required information on paper during the call, but their voices should sound natural. Another difference is that a TISP can make a call from any geographic location, while pretending to be in the neighbourhood of the medical service that has been contacted. In addition, different roles can be played by the same TISP.

An important issue with every ISP is accuracy of role playing. Accuracy can be defined as the proportion of clinical features presented correctly during the consultation.21 Another important issue is whether call handlers will detect that a call has been made by a TISP rather than by a real patient (‘detection rate’).

In the literature we found no information on the validity, feasibility and preparation of TISPs, the accuracy of their role playing, detection rates or how calls were recorded for further assessment.

We designed a study to address the following research questions:

  • 1To what extent is it feasible to use TISPs within the primary care setting of an out-of-hours centre (OOHC), with emphasis on the preparation required? The study included assessment of the feasibility of self-recording at home by the TISPs, the accuracy of their role playing and detection rates.
  • 2What are the experiences of TISPs using this new methodology?

Methods

Selection of clinical cases

The research team developed seven clinical roles for the TISPs, validated by a group of 12 general practitioners (GPs) selected at random from different areas in the Netherlands. The seven roles were presented regularly by telephone to OOHCs. Role 1 concerned a 5-year-old child with fever; roles 2 and 3 concerned an adult with fever; roles 4 and 5 concerned a 5-year-old child with vomiting, and roles 6 and 7 concerned an adult with epistaxis.22–24 Roles 2 and 3, 4 and 5, and 6 and 7, respectively, involved similar scenarios with small variations. For each case all GPs agreed that the required disposition might be either self-care advice or a consultation at the OOHC on the same evening. None of the cases would require immediate care or a home visit.

Selection and training of the TISPs

The University of Maastricht has used face-to-face ISPs in medical student training for many years. A group of ISPs was selected from the university’s pool of ISPs, based on experience and clarity of speech. Three scenarios were to be played by a mother calling about her 5-year-old child; the other four were to be played by male or female adults.

To lessen the chance that a call handler might recognise a TISP’s voice, every scenario was to be presented by two different TISPs, one of whom would call the same OOHC 6 weeks after the other.

Each pair of TISPs with the same role was trained together in two 1.5-hour sessions by two nurses and a GP, all of whom had previous experience of working at a medical call centre. The TISPs were instructed and trained to provide further information only on request. To help the TISPs speak naturally during the consultation, extra training was provided on the expression of non-verbal signals such as surprise, disappointment or anxiety. The training included giving attention to the prescribed opening and closing sentences and how to react to a request to come to the OOHC the same evening, which was obviously impossible. The TISPs were trained by presenting their problem to the GP and a nurse. In cases where a return call was made to the TISP’s mobile phone, they were trained to make notes about what was discussed directly after the call. Midway through the project, all TISPs returned to the research centre for refresher training lasting 1 hour. No changes were made in the existing information or the TISPs’ roles.

Description of scenarios

For each case a scenario was developed with three kinds of information: general, case-specific and local information. The general information described all personal patient data, including family name, date of birth, home address and the TISP’s mobile phone number.

The case-specific information described the opening and closing sentences, the reason for calling (e.g. medication or consultation), the personal background of the caller (e.g. caller is worried) and the answers to questions from the call handler about medical issues.

The local information contained information about OOHCs (addresses, telephone numbers), an area map and a picture of the TISP’s fictitious residence so that the TISP had an idea about the area of the OOHC.25 All locations were visited by the research co-ordinator before the training started to ensure that locations were appropriate and so that relevant local information could be passed on to the TISPs during training.

Selection of call centres and calls

Previous research has shown that a sufficient time lapse between the announcement of an SP visit and the actual visit can prevent detection.5 Therefore, 1 year before the actual calls were made, all 103 OOHCs in the Netherlands were asked for permission for TISP calls and for the recording of the calls. (This is not normal practice because under Dutch law no ethical approval is needed to phone or record calls made by SPs to an OOHC once permission has been given by the management of the OOHC.) From the 98 centres that gave permission, 17 were selected at random. The call handlers at these OOHCs were not informed that their centres had been selected and thus were unaware of when they would be called or the names that would be used by the TISPs. Each case was to be presented three times to each of the 17 OOHCs; thus 357 calls were scheduled.

Recording of calls

As the TISPs lived within a radius of 20–30 km of the research institute, we decided to allow them to make the calls from their own homes. We found a system that met our standards for the adequate self-recording of calls. (Detailed information on the equipment used can be supplied on request.)

Pilot telephone calls

To be sure that the TISP s had all the required information about the OOHCs and that they were ready to play their respective roles, two pilot studies were performed. Pilot 1 aimed to ensure that the TISPs had all the required and correct general, local and case-specific information they needed. Therefore, the trainers called every selected OOHC for medical advice using fictitious personal data.

Pilot 2 aimed to check the accuracy of role playing and the self-recording of calls by the TISPs. For this, each TISP called a medical call centre that had been forewarned, taking on his or her assigned role, with fictitious personal data, and self-recorded the call. The call handlers at the medical call centre were aware that they were being called by a TISP as the fictitious name was flagged as representing a TISP in their system. They were instructed to handle this trial call as they normally would do and to report on their general impression of the TISP’s performance. These trial calls were also assessed by the researcher and trainers.

Assessment of accuracy

During the research project each TISP called four to five different OOHCs per evening. The TISPs were free to choose in which order they called the OOHCs on a particular evening. The third call was selected to measure accuracy of role playing.

Each TISP assessment was performed by two assessors who used the following four criteria developed by the research team:

  • • Was the conversation opened according to the instructions?
  • • Did the TISP give the correct answers to questions asked by the call handler?
  • • Did the TISP give extra information only if asked for?
  • • Was the conversation closed according to the instructions?

The assessors received a copy of the recorded call on a CD, along with the information the TISPs had received about their respective roles. They could respond to each of the criteria with Yes, No or Unsure. The TISPs were informed that their role-play would be assessed but were not told which of their calls would be selected for assessment.

Detection of SPs by the OOHC

After all calls had been made, we sent a questionnaire to the 98 OOHCs that had given permission for the calls. They were asked to answer questions about:

  • • any indications they had been called by an SP; if so:
  • • on what date and for what reason;
  • • whether the SP involvement had been discovered during or after the call and how sure the call handler was that he or she had been called by an SP (0–100%), and
  • • whether the discovery had influenced the handling of the call.

Evaluation by the TISPs

The TISPs were asked to fill in a questionnaire about their experiences, both quantitatively and qualitatively. Questions addressed issues such as training and preparation and the self-recording. Personal impressions and experiences were also sought. Furthermore, the TISPs were interviewed by the researcher and a trainer about their personal experiences.

Results

Selection and training of TISPs

Of the 14 selected and trained TISPs, two dropped out for personal reasons very shortly before the research project started. Their task was taken over by the TISP who had already been trained for the same role; thus the performance of 12 TISPs was assessed.

Recording the call

We scheduled 357 calls, but the TISPs had to make 375 calls as 18 were not recorded correctly because of technical problems that occurred during the recording. The TISP concerned received a new set of fictitious personal data to make an extra call a few weeks later.

It took the TISPs 1.5–2 hours per evening to make four to five calls. All calls were made between March 2006 and March 2007. On 11 occasions a TISP received a return call on the same evening from the GP on call at the OOHC. On three occasions the TISP was called by the administration of a centre with some additional questions 2–3 weeks after the TISP had made his or her initial call. Of these 14 return calls, 11 were answered by the TISP and notes were made of the ensuing conversations. Three calls were not answered for different reasons. The calls were assessed by 12 different assessors, all of whom agreed on the excellent quality of the recordings.

Pilot telephone calls

From the first pilot we learned that some centres ask for the postal code of the residence. This information was added to the local information. The second pilot showed that the TISPs played their roles convincingly. Some of the TISPs received advice about refining their presentations of their opening sentence as they tended to read out this sentence instead of delivering it in a personal style.

Assessment of accuracy

Using the procedures described, a total of 84 calls were available for assessing the accuracy of role-play. The results for the four criteria as assessed by two assessors are displayed in Table 1.

Table 1.   Results of assessments of performance of role-play per item (168 assessments for 84 calls)
 Opening sentenceCorrect answersExtra information on requestClosing sentence
  1. Yes = delivered according to instructions; No = not delivered according to instructions

Yes162166166149
Unsure5229
No10010

Detection

The 17 OOHCs that had been called stated that they had not detected any calls from TISPs. Two OOHCs that had not been called by one of our TISPs stated that they had received a call from a TISP.

Experiences of TISPs

Training and preparation

Although not every situation encountered was covered in the training, the TISPs were able to handle difficult situations because they had rehearsed their roles in great detail. The TISPs stated that the training, which involved presenting their case to a GP and a nurse, had been very helpful, especially as the return calls on the same evening were made by GPs. Making the try-out call to the medical call centre gave them self-confidence. They felt that the extra attention that had been focused on playing the role using only verbal communication had helped them to play the role more naturally.

Information

None of the TISPs were confronted with a question for which they were unprepared. Some TISPs were aware that they had not finished the conversation with the correct closing sentence, mainly because some call handlers finished the consultation in a way that took the TISP by surprise. Having a picture of a fictitious residence together with an area map had helped the TISPs to enter into their roles.

Calling and self-recording

The TISPs appreciated that they had been able to make the calls from home. Sometimes the self-recording had caused more tension than making the call itself. Both role playing and recording to a high standard required intense concentration with no distractions. Five calls was considered to be the maximum number that could be made per evening.

Personal experiences

The most frequent comments were: ‘I missed the eye contact with the care adviser,’ and ‘I was glad that I was trained to play my role using merely my voice.’

Other comments included: ‘You have to stay alert for a return call, not only for the rest of the evening, but even for many days later [so that you can] [re]act immediately and identically as in your role.’

Discussion

To our knowledge, this study describes the first research into both the feasibility of using TISPs in a health care setting and assessment of the accuracy of their role playing. Therefore, many of the procedures described are the result of a very careful step-by-step approach to the introduction of TISPs.

Although some members of the research team had expertise in training SPs for use in ISP studies pertaining to actual visits with face-to-face contact, we did not merely rely on this experience in our efforts to provide the optimal training and preparation of TISPs to call OOHCs for medical advice. From this perspective, we believe our results are innovative.

For example, we provided the TISPs with pictures of the residences in which they were supposed to live. The study confirmed our expectation that this extra information is needed to help TISPs play their roles convincingly. In a new study, we would collect this type of information in other ways (e.g. from websites) that are less time-consuming.26 However, this information needs to be accurate and up-to-date to prevent detection.

Although TISPs who are recording calls themselves do not need to be trained to fill in a checklist after a consultation, they face other challenges. Their experiences showed that thorough training for role-play is needed to prepare them for handling the many different situations they might encounter. Most unexpected were the calls made by some OOHCs 2 or 3 weeks later, during the day, when the administration of a centre requested where to send the bill for a telephone consultation. This situation was very troublesome for the TISPs as they were suddenly confronted with questions regarding personal data relating to the fictitious person they had played.

Although the return calls were not recorded, we believe that no information regarding the consultation was lost as the TISPs were trained to make notes of the conversation directly after the call. Telephone ISPs should be instructed to answer all telephone calls following their role-plays so that no return call from an OOHC goes unanswered.

Role playing and recording the calls at the same time caused extra tension for some of the TISPs, but the fact that they could record the calls at home rather than having to travel somewhere else in the evening was preferred by all the participants. There is a slight risk of failure in self-recording the call, but the advantage of consulting by telephone is that a new call can easily be made by the same TISP if he or she adjusts the personal data used.

As we had no experience with using TISPs we decided to present only cases that did not require a home visit or an emergency consultation as such cases are more likely to incur detection. Further research into using TISPs for all types of clinical cases, including emergency cases, will be our next challenge.

We were pleased to see that the TISPs played their roles with great accuracy. As not all TISPs adhered completely to the instructions given for the delivery of the opening and closing sentences, this issue requires special attention during training. The method of using TISPs is feasible in any country, but privacy regulations about recording calls may cause restrictions in some countries.

We conclude that it is feasible to use SPs on the telephone in both the training and assessment of performance in telephone consultation of doctors, trainees and other personnel involved in medical services, as long as SPs are well prepared and sufficiently trained in role-play. Self-recording to a high standard of calls made from home is also feasible.

Contributors:  all authors were actively involved in the design of the research project and contributed to the corrections of this article. All authors meet the criteria for authorship.

Acknowledgments

Acknowledgements:  we thank all telephone standardised patients who participated in this study. Special thanks go to Estelle Billet for her excellent training.

Funding:  this research was funded by CZ and VGZ, both of which are health care insurance companies in the Netherlands.

Conflicts of interest:  this work was performed independently of any activity on the part of the funders.

Ethical approval:  in the Netherlands ethical approval is not required for this type of study, however, permission was granted by all the Out-of-Hours Centres that participated.

Ancillary