Computer-assisted assessment of obsessive-compulsive disorder in young people: a preliminary evaluation of the Development and Well-Being Assessment

Authors


Abstract

Background

Paediatric obsessive-compulsive disorder (OCD) often goes undetected, delaying access to evidence-based treatment. This study aimed to assess the utility of a computerised diagnostic tool, the Development and Well-Being Assessment (DAWBA), in detecting OCD and comorbidity in youth.

Method

A total of 51 young people referred to a specialist OCD service between September 2007 and July 2008 completed the DAWBA prior to clinical assessment. Computer-rated and clinician-rated DAWBA diagnoses were compared with those assigned by the specialist clinic.

Results

The computer-rated and clinician-rated DAWBA correctly classified OCD in 71% and 77% of cases respectively. Compared to consensus diagnoses, the computer-rated DAWBA tended to over-diagnose comorbidity, while the clinician-rated DAWBA diagnoses of comorbidity corresponded well with the consensus.

Conclusions

The DAWBA has potential value in detecting OCD as well as comorbid disorders, and could be a cost-effective method for diagnosing OCD and related problems.

Key Practitioner Message

  • The current study represents the first investigation into the utility of the DAWBA in detecting OCD and comorbid disorders in young people
  • Even among a group of young people with complex and unusual OCD, the clinician-rated DAWBA correctly classified 77% of cases
  • A positive diagnosis of OCD on the DAWBA has high accuracy. A negative diagnosis of OCD may be less accurate and therefore OCD should not be ruled out on this basis alone
  • The clinician-rated DAWBA may be helpful in highlighting when comorbid conditions are present. Computer ratings alone may produce a high number of false positives, particularly of other anxiety disorders and externalising disorders

Background

Obsessive-compulsive disorder (OCD) is a serious and potentially debilitating condition that is estimated to affect 0.25–4% of young people (Douglass, Moffitt, Dar, McGee & Silva, 1995; Flament et al., 1988; Heyman et al., 2001). It typically follows a chronic course if left untreated, and can have a profound impact on functioning in all domains, including family life, peer relationships, and education (Piacentini, Bergman, Keller & McCraken, 2003; Stewart, Geller & Jenike, 2004). Although effective treatments for paediatric OCD exist, namely cognitive behaviour therapy (CBT) and serotonin reuptake inhibitor medications (SSRIs), recent studies indicate that OCD is underdetected and the majority of young people with OCD suffer for long periods before they access treatment (e.g. Chowdhury, Frampton & Heyman, 2004). Underdetection of OCD may partly reflect clinical practice, and the failure of clinicians to successfully screen for and assess obsessive-compulsive symptoms. Although OCD is usually associated with a characteristic set of symptoms, it can nevertheless present a diagnostic challenge due to its heterogeneity and topological overlap with other psychiatric and neurodevelopmental disorders, such as generalised anxiety disorder, tic disorder and autism spectrum disorders (ASD; Merlo, Storch, Murphy, Goodman & Geffken, 2005).

A wide range of measures have been developed for the assessment of OCD (Merlo et al., 2005), including brief self-report questionnaires and more lengthy clinician-administered measures and diagnostic interview schedules. Diagnostic interview schedules, such as the Anxiety Disorders Interview Schedule for DSM-IV (ADIS; Silverman & Albano, 1996) and the Schedule for Affective Disorders and Schizophrenia for School-age Children (K-SADS; Kaufman, Birmaher & Brent, 1997) have many positive qualities including the fact that they can be used to assign diagnoses and determine differential diagnosis. The main disadvantage of these instruments is that administration is time consuming and not practical in many clinical settings, and hence they tend to be primarily used in research contexts (Suppiger et al., 2009). The Development and Well-Being Assessment (DAWBA; Goodman, Ford, Richards, Gatward & Meltzer, 2000) offers a promising solution to this problem. The DAWBA consists of a collection of questionnaires and rating techniques that can be used to generate a range of psychiatric diagnoses in children and adolescents based on ICD-10 (World Health Organization, 1992) and DSM-IV (American Psychiatric Association, 1994) criteria. It was originally developed as a face-to-face interview but more recently has been adapted into an online format, meaning that parents and children can complete it without an interviewer being present. The computer program generates diagnostic predictions based on the parent/child responses, which can be reviewed by a clinical rater, in conjunction with the original parent/child responses, in order to assign final diagnoses. To our knowledge, the DAWBA is currently the only computerised diagnostic assessment for paediatric OCD.

The DAWBA has a number of important advantages. First, as already noted, it is convenient for families to complete and places relatively little demand on clinician's time. Second, both parent and child reports are systematically collected and integrated, which may enhance diagnostic accuracy, particularly in OCD where the agreement between parent and child report has been shown to be poor and the need for multiple informants has been emphasised (Canavera, Wilkins, Pincus & Ehrenreich-May, 2009). Third, it screens for all common Axis I disorders, thereby informing differential diagnosis and allowing identification of the primary disorder. Again, this is particularly relevant to the assessment of OCD where rates of comorbidity are high, with up to 80% of young people meeting diagnostic criteria for an additional diagnosis (Pediatric OCD Treatment Study (POTS) Team, 2004; Storch et al., 2008), and where symptoms can overlap with other psychiatric and neurodevelopmental disorders.

The DAWBA has been shown to have good psychometric properties in various settings. In a large-scale study involving a community sample and a clinic sample, the DAWBA was able to discriminate well between the two groups in terms of rates of diagnoses (Goodman et al., 2000). Furthermore, within the clinic sample, there was substantial agreement between the DAWBA diagnoses and case note diagnoses. The DAWBA has since been used in several countries to establish rates of child psychiatric disorder in epidemiological samples (Ford, Goodman & Meltzer, 1999; Heiervang et al., 2007; Mullick & Goodman, 2005). Disorder-specific sections of the DAWBA have also been evaluated in clinical settings. The eating disorder section has been shown to have excellent agreement with clinical diagnoses (Moya et al., 2005), and to be superior to the Eating Disorder Examination in detecting the presence of an eating disorder among adolescents referred to a specialist eating disorder service (House, Eisler, Simic & Macali, 2008). Similarly, the attention-deficit hyperactivity disorder (ADHD) section has been shown to have moderate-to-substantial agreement with clinical diagnosis among a secondary care sample, which is comparable to existing face-to-face diagnostic instruments (Foreman, Morton & Ford, 2009). Thus, overall the current findings suggest that the DAWBA has considerable potential as an efficient diagnostic measure in both clinical and research settings. To date, no study has specifically evaluated the OCD section of the DAWBA.

The potential advantages of the DAWBA as an assessment instrument for paediatric OCD indicate the value in establishing its psychometric properties in the detection of OCD. This naturalistic study sought to provide an initial evaluation of the DAWBA in assessing paediatric OCD, in the context of a specialist OCD clinic. This was considered to be an appropriate setting in which to evaluate the OCD section of the DAWBA, given that the clinic tends to receive referrals of young people with more unusual and complicated OCD, and in that sense it provided a stringent test of the diagnostic properties of the DAWBA. In particular, the current study aimed to establish the validity of computer-rated and clinician-rated DAWBA diagnoses of OCD, by comparison with diagnoses assigned by the specialist OCD clinic. DAWBA misdiagnoses of OCD were examined in detail to understand the nature of the errors. A secondary aim was to explore the utility of the DAWBA in detecting comorbid disorders among young people with a diagnosis of OCD.

Method

Participants

Participants were young people and their parents, who were consecutive attendees for an initial assessment at the OCD and Related Disorders Clinic for Young People at the Maudsley Hospital, London. Young people are referred to the clinic from across the United Kingdom, most commonly with suspected OCD but sometimes with related disorders such as body dysmorphic disorder and Tourette Syndrome. They tend to be a relatively severe or treatment-refractory group, or have complexities regarding diagnosis.

Families who attended the clinic for an assessment between September 2007 and July 2008 were asked to complete the online DAWBA prior to their appointment. Fifty-four families were approached, of whom three did not complete the DAWBA. The final sample therefore comprised 51 participants, of whom 29 were male and 22 were female. The mean age was 14 years 11 months (SD = 2.44). The majority of the sample were White British (= 47), with the remainder being Mixed Asian (= 1), Tamil (= 1), Chinese British (= 1) and Other White (= 1).

Measures

The online version of the DAWBA was used in the current study, which is designed to be completed by parents and children over 11 years (only parent report is used for younger children). Parents and young people access their separate sections using different passwords, meaning that each informant can answer questions confidentially. Within each diagnostic category, initial screening items are presented. If these are not endorsed then the subsequent questions are disregarded and the informant can skip to the next diagnostic category.

The OCD section of the DAWBA includes a combination of 6 open and 20 closed questions, and probes the following symptom domains: washing; checking; repeating; touching; arranging and counting. In addition to establishing the presence of obsessions and compulsions in these domains, the DAWBA assesses associated impairment, distress, egodystonia, resistance and insight. These questions map specifically onto the DSM-IV and ICD-10 diagnostic criteria.

Once the DAWBA has been completed by parent and child, it is submitted online and the computer generates diagnostic predictions using responses to closed questions. The computer predictions for each diagnostic category are organised according to six levels (termed ‘bands’), which correspond to the probability that the disorder is truly present, and the highest two prediction bands can be counted as a positive diagnostic prediction (Goodman, Heiervang, Collishaw & Goodman, 2011).1 The preliminary computer predictions are then reviewed, in conjunction with the original parent and child responses, by an experienced clinical rater who generates the final DAWBA diagnoses. The DAWBA can be used to generate diagnoses based on ICD-10 or DSM-IV criteria. For comparison with the specialist clinic diagnoses, only the ICD-10 diagnoses were considered in the current study.

Procedure

The diagnostic procedures used in the current study are summarised in Figure 1. Young people who were offered an assessment appointment were sent a letter containing instructions of how to access the online DAWBA, and separate login details for the young person and parents. They were asked to complete the DAWBA as a matter of routine prior to attending their appointment at the specialist clinic.

Figure 1.

Summary of diagnostic procedures. (Computer diagnoses of obsessive-compulsive disorder (OCD) were generated using the ≥15% prediction band; computer diagnoses of comorbid disorder were generated using the ≥50% prediction band)

The computer-rated diagnostic predictions were reviewed by members of the specialist clinic before the assessment, in conjunction with other information that had been obtained from the referrer and other agencies. At the assessment, the young person was interviewed using the Children's Yale-Brown Obsessive-Compulsive Scale (Scahill et al., 1997) to screen for the presence of obsessions and compulsion, and to establish symptom severity. In addition, a mental state examination was conducted, as well as an interview with the parents to obtain a full clinical history. The specialist clinical team then assigned diagnoses based on ICD-10 criteria.

Independent of the specialist clinic assessment, parent and child responses to closed and open-ended questions on the DAWBA, and the preliminary computer-rated diagnoses, were reviewed by two external clinical raters who were both highly experienced in diagnostic classification and trained in DAWBA ratings. The clinical raters discussed each case and reached an agreement on the final DAWBA diagnoses.

In cases where there was disagreement between the clinician-rated DAWBA diagnoses and the specialist clinic diagnoses, case notes and DAWBA responses were reviewed by a senior clinician from the specialist clinic and a DAWBA clinical rater, and consensus diagnoses were agreed.

Results

Clinical diagnoses

The primary diagnoses made by the specialist clinic were OCD (= 37), tic disorder/Tourette Syndrome (= 3), ASD (= 3), specific phobia (= 2), trichotillomania (= 1), anorexia nervosa (= 1), bulimia nervosa (= 1), and body dysmorphic disorder (= 1). Two young people did not meet diagnostic criteria for any psychiatric disorder at clinical assessment. In addition to the 37 cases of primary OCD, one young person met diagnostic criteria for OCD occurring as a secondary problem.

Overall agreement of OCD diagnoses

The agreement between the computer- and clinician-rated DAWBA diagnoses with the specialist clinic diagnoses are shown in Table 1. Both the computer- and clinician-rated DAWBA methods correctly classified the majority of young people with respect to OCD (36 and 39 cases of 51, respectively). Both DAWBA procedures had good positive predictive values but poor negative predictive values.

Table 1. Agreement between the computer-rated and clinician-rated DAWBA diagnoses, and clinical diagnoses
DAWBA diagnostic procedure Specialist clinic diagnosisPsychometric indices
OCDNo OCDCC (%)PPV (%)NPV (%)
  1. Note: DAWBA, Development and Well-Being Assessment; CC, correct classification; PPV, positive predictive value; NPV, negative predictive value.
Computer-rated diagnosisOCD28570.684.844.4
No OCD108
Clinician-rated diagnosisOCD32676.584.253.8
No OCD67

Reasons for disagreement in OCD diagnoses

The clinician-rated DAWBA diagnoses did not identify OCD in six young people who were clinically assessed as cases (i.e. false negatives). In three of these cases, OCD symptoms were detected by the DAWBA but were classified as being sub-clinical by the rater, and indeed clinical assessment had established that symptoms were very mild in two of these cases, only just reaching threshold for diagnosis. A further two OCD cases were not identified by the DAWBA due to incomplete parent/child responses. The final unidentified OCD case was a misclassification error whereby OCD symptoms were rated as being part of generalised anxiety disorder. However, it is of note that this case presented with particularly unusual OCD symptoms (obsessional worries about acquiring unwanted characteristics, termed transformation obsessions; Volz & Heyman, 2007), which had in fact lead to diagnostic confusion in the mental health services in which the young person had previously be seen.

The clinician-rated DAWBA diagnoses identified OCD as being present in six young people who were not considered to be cases at clinical assessment (i.e. false positives). Five of these cases were diagnosed by the specialist clinic as having another obsessive compulsive spectrum disorder, namely tic disorder (= 2), trichotillomania (= 1), body dysmorphic disorder (= 1), and anorexia nervosa (= 1). The sixth case was diagnosed with an ASD at clinical assessment. In all six cases repetitive, stereotyped behaviours were present and reported on the DAWBA.

Detection of comorbidity in OCD cases

Rates of comorbidity were calculated among the cases that were assessed as having OCD by the specialist clinic. This revealed that one or more comorbid disorder was present in 29% of cases according to the clinical diagnoses, 79% of cases according to the computer-rated DAWBA, 58% of cases according to the clinician-rated DAWBA, and 50% of cases according to the consensus diagnoses. Assuming the consensus diagnoses had highest accuracy, these findings suggest that specialist clinic tended to underdiagnose comorbid disorders, while the computer-rated DAWBA tended to over-diagnose comorbidity in young people with OCD. An analysis by diagnostic category indicated that the computer particularly tended to over-diagnose other anxiety disorders and externalising disorders (see Figure 2). The clinician-rated diagnoses corresponded well to the consensus ratings, although there was still some tendency to over-diagnose other anxiety disorders and externalising disorders. Depression was most reliably detected by both DAWBA methods, with consistent rates determined by the preliminary computer diagnosis, clinician-rated diagnosis and consensus diagnosis (26%, 21% and 21% respectively). Lower rates of depression were identified at clinical assessment (5%) suggesting that it was underdiagnosed in routine practice.

Figure 2.

Rates of comorbid disorders in obsessive-compulsive disorder (OCD) cases according to the different diagnostic procedures. Note: ODD = oppositional defiant disorder; CD = conduct disorder; ASD = autism spectrum disorder

Discussion

The present study sought to establish the validity of the DAWBA in the diagnosis of OCD in youth, and to explore its utility in detecting comorbid disorders in this population. Despite the complexity of the OCD cases in the current sample, the level of agreement between the DAWBA and the specialist clinic was fair, with both the computer- and clinician-rated DAWBA correctly diagnosing OCD in over 70% of cases. For both DAWBA methods, a positive diagnosis of OCD on the clinician-rated DAWBA had high accuracy but a negative prediction had poorer accuracy. However, the false negatives were most commonly due to incomplete responses from the child and/or parent, or mild OCD symptoms being detected but rated as sub-clinical. False positives tended to occur when repetitive behaviours were present in the context of another disorder (e.g. tic disorder, ASD, BDD). Such cases often create diagnostic confusion even among experienced clinicians, and therefore it is perhaps not surprising that DAWBA classification errors occurred. It has previously been suggested that probing for more ‘unusual’ symptoms, such as obsessions and compulsions, may be prone to produce false positives (Breslau, 1987). It is likely that the current sample contained higher rates of atypical OCD presentations than would be observed in non-specialist services, which may have therefore resulted in more false positives.

In relation to our second aim, the current study found that both DAWBA, the computer- and clinician-rated DAWBA, had a tendency to over-diagnose comorbidity, an observation that has been made in other studies (Goodman et al., 2000). This was particularly true for anxiety and externalising disorders, and in reviewing these cases it was clear that in most instances false positives were due to OCD symptoms being reported and rated in separate diagnostic sections. With respect to anxiety disorders, in many cases obsessions (e.g. fear of contamination) were reported by children and parents as specific fears in the phobia section, and as worries in the generalised anxiety disorder section. With respect to externalising disorders, it is not uncommon for young people with OCD to become verbally and physically aggressive if their rituals are disrupted by others (Storch et al., 2007), and parents often reported such behaviour in the oppositional defiant disorder and conduct disorder sections of the DAWBA. In most cases, these ‘spill-over’ errors were easily identified by clinical raters reviewing the original DAWBA responses, and the diagnosis was overturned accordingly. As a consequence, the clinician-rated DAWBA diagnosed lower rates of comorbidity than the computer (58% versus 79%). With the clinician-rated DAWBA procedure, rates were only slightly inflated relative to the consensus diagnoses (58% versus 50%), indicating that accuracy was greatly enhanced by the addition of the clinical rater, compared to the computer alone.

These findings have a number of clinical implications. First, they suggest that the computer-rated DAWBA diagnoses may be sufficiently accurate to positively predict OCD in a sample of young people with suspected obsessive-compulsive symptoms. This could potentially improve the cost-effectiveness of assessment and expedite access to treatment, by removing the need to conduct a full clinical assessment to ascertain diagnosis. However, the current findings also suggest that cases who receive a negative diagnosis of OCD on the DAWBA may warrant further clinical assessment and a diagnosis of OCD should not be ruled out on the basis of the DAWBA results. Second, these findings suggest that the DAWBA may assist in identification of comorbidity in paediatric OCD, which could improve treatment delivery by indicating a need to adapt treatment (e.g. modifying CBT for OCD in an individual with an ASD) and/or a need for additional adjuncts to treatment (e.g. parent management training for comorbid behavioural problems). The current findings demonstrate that the specialist clinic diagnosed lower rates of comorbidity than the consensus method and the DAWBA. This may reflect the fact that in routine clinical practice a psychiatric assessment may fail to be fully systematic, and the clinician may focus on the presenting problem, potentially overlooking other difficulties that might be detected with the use of a structured instrument (Angold & Costello, 2009). The DAWBA offers a promising solution by undertaking a comprehensive screen of common psychiatric disorders with minimal clinician input.

It is important to note that the lower rates of comorbity found in the clinical diagnoses in this study may also reflect a more hierarchical, pragmatic approach to diagnosis. The DAWBA does not generate a hierarchy of diagnoses but rather uses the operationalised criteria of ICD-10 or DSM-IV to generate diagnoses according to an algorithm, whereas clinical judgment may subsume one set of symptoms within another diagnosis. For example, many OCD patients in the current sample reported low mood, fatigue, poor concentration and other symptoms of depression, and according to the DAWBA and consensus ratings, over a fifth of the sample met ICD-10 diagnostic criteria for a depressive episode. However, at clinical assessment depressive symptoms were often considered to be a consequence of the impairment and distress caused by OCD, not an independent problem requiring treatment, and therefore depression was not coded as a separate disorder in three quarters of the cases that operationally met diagnostic criteria. This decision was a pragmatic one designed to focus and simplify treatment, with the assumption being that additional symptoms would be targeted if they did not resolve with treatment of the presumed ‘primary’ diagnosis.

The present study has several limitations that must be considered when interpreting the results. First, the sample size is relatively small and was comprised mainly of White British families. Future studies should seek to replicate the current findings in larger and more ethnically diverse populations given that previous research has demonstrated cultural sensitivity in OCD measures (Garnaat & Norton, 2010). Second, this study was conducted among referrals to a national specialist OCD clinic, where the prevalence of OCD is significantly higher than in generic, local community child and adolescent mental health services. Both the positive and negative predictive values are dependent on the prevalence of the disorder to be detected, and therefore the properties of the DAWBA in detecting OCD that are reported in this study need to be established in other clinic settings. Furthermore, the OCD cases referred to a specialist clinic may not be representative with respect to severity, nature of symptoms and comorbidity. However, we would hypothesise that the DAWBA would demonstrate higher positive and negative predictive values in relation to more classical OCD presentations, as might be found in non-specialist settings. A third limitation of the current study is that the clinical team reviewed the preliminary computer diagnoses before undertaking the full clinical assessment and assigning diagnoses. This reflects the naturalistic nature of the study; the DAWBA information was reviewed by the specialist team in order to enhance the clinical assessment. It is possible that the computer results influenced the clinical team's assessment and formulation, thereby artificially inflated the agreement rate in diagnoses. A more stringent test would be for the clinical diagnoses to be made independently of the DAWBA.

In summary, the current study represents the first investigation of the use of a computerised instrument, the DAWBA, to assess and diagnose OCD and related disorders in young people. The results are promising and suggest that the clinician-rated DAWBA could potentially be a useful diagnostic instrument for OCD and comorbidity in young people, although further research is needed to investigate the validity of the DAWBA in other clinical and non-clinical settings and with more diverse populations. Currently, the DAWBA is the only computerised diagnostic measure of OCD and common related disorders, and has a major advantage over traditional clinician-administered measures in that it involves minimal clinician time and training. It therefore has the potential for large-scale outreach and by enhancing diagnostic procedures in community settings, the DAWBA could assist in expediting access to evidence-based treatment.

Acknowledgements

We thank Professor Robert Goodman for providing help with consensus ratings and useful discussions regarding the manuscript. The authors have declared that they have no competing or potential conflicts of interest.

  1. 1

    These diagnostic predictions are available to the clinical team but not the parent or young person.

Ancillary