Reliability and validity of the Japanese version of the Cognitive Therapy Awareness Scale: A scale to measure competencies in cognitive therapy

Authors


Daisuke Fujisawa, MD, PhD, Psycho-Oncology Division, National Cancer Center East, 6-5-1 Kashiwanoha, Kashiwa-shi, Chiba 277-8577, Japan. Email: dfujisaw@east.ncc.go.jp

Abstract

Aim:  This study aimed to test reliability and validity of the Japanese-version Cognitive Therapy Awareness Scale, a self-rating scale that measures basic knowledge of cognitive behavioral therapy (CBT) concepts and methodology. Furthermore, we explored optimal cutoff score to demonstrate required standard of CBT competency.

Methods:  The reliability and validity of the scale was tested among 252 mental health professionals. The participants' number of readings relevant to CBT, as well as their experience in training and practice in CBT were used as external standards of competency.

Results:  The scale has good internal consistency (the Kuder-Richardson formula 20 = 0.76). The total score of the scale was significantly correlated with experience in training and practice in cognitive behavioral therapy, and the number of relevant readings (Spearman's rho = 0.27, 0.28 and 0.44, respectively, P < 0.001). No significant correlation was found between the total score of the scale and experience in general psychosocial care (Spearman's rho = −0.02, P = 0.76). The receiver operating characteristics curve analysis produced the area under curve (AUC) of 0.77 (SD = 0.05, P < 0.001), and a cutoff score of 31/32 produced sensitivity of 0.81 and specificity of 0.64 to discriminate trainees who have experience of 40 h or more of CBT training and five or more of relevant readings, from those who do not meet this standard.

Conclusion:  The Japanese-version Cognitive Therapy Awareness Scale was successfully validated. It has been demonstrated that the scale specifically assess knowledge in cognitive behavioral therapy. This scale would serve as a rough guide to assess competencies among beginner cognitive behavior therapists.

UNDERPINNED BY ITS robust research evidence, cognitive behavioral therapy (CBT) has become an indispensable psychological treatment in psychiatric practice.1,2 In the United States (US), CBT has become a mandatory competency in residency training.3 In the United Kingdom, a government-funded movement toward improvement of public accessibility to CBT has been taken place since 2007.4 In Japan, CBT has come to be covered by the national health insurance in 2010.5 Acquisition of basic knowledge in CBT is important not only for the therapists who deliver CBT themselves, but is important for all the mental health professionals because every mental health professional is responsible for providing evidence-based medical information to the patients, meaning that they are expected to introduce CBT to the patients to whom it is applicable.

As is the case with other psychotherapies, treatment outcome of CBT is influenced by therapists' competence.6,7 Therefore, ensuring quality of therapy is pivotal when incorporating CBT in healthcare policy. In the UK and the US, guidelines of qualification of therapists have been established.8,9

In principle, acquisition of CBT competencies starts with acquisition of knowledge, followed by acquisition of skills and clinical applications tailored to individual cases.10 In CBT training, most studies emphasize a minimum of 30 h of didactic lecture, along with a 6- to 12-month period of clinical work with patients under regular supervision.11

For assessing competencies in psychotherapy, several methodologies have been proposed and have been used. The methods include a written examination including multiple-choice questionnaire, case logs, checklist evaluation, case report, chart-stimulated oral examination, direct and indirect supervision, role-plays, and didactic discussion.10 Among these, multiple-choice questionnaires are universally used for in-training examinations and for initial certification. In the field of psychodynamic psychotherapy, the Columbia Psychodynamic Competency Psychotherapy Test has been widely used for competency assessment.12 In the field of CBT, the Cognitive Therapy Awareness Scale (CTAS) has been proposed as a method of assessing CBT knowledge.11

The CTAS is a self-rating questionnaire which measures basic knowledge of cognitive therapy concepts and methodology.13 The CTAS contains 40 true/false questions on topics such as definitions of automatic thoughts and schemas; description of thought records, activity schedules, and other commonly used treatment methods; and identification of maladaptive thinking in case illustration. The maximum score on the CTAS is 40, and the higher score reflects the greater amount of knowledge.

The CTAS has been used in the US residency training programs as a standardized pre- and post-measure for changes in knowledge associated with participation in CBT courses. The US psychiatric residents typically have mean CTAS scores in the mid-20s to lower 30s before starting formal training in cognitive therapy, and most of them have a substantially higher CTAS score after completing a comprehensive course in CBT.11 However, the CTAS has not been studied systematically so far, therefore the optimal score that trainee-therapist should aim for has yet been established.

The aim of this study is to develop the Japanese-version of the CTAS, and to test its reliability and validity. Furthermore, we aimed to explore its optimal cutoff score (required standard).

METHOD

Development of the Japanese-version CTAS

First, we developed the Japanese-version CTAS (CTAS-J; available from the first author upon request). To maintain equivalence in a cross-cultural adaptation of the CTAS, we basically followed the guidelines proposed by Beaton et al.14,15 The translation procedure comprised front-translation by two translators, translation synthesis, back-translation, expert review and pre-testing. The 40 items of the CTAS were translated into Japanese by two of our authors (DF and AN), and discussion was made on translation discrepancies, then the transcripts were synthesized into one translation. It was back-translated by a bilingual translator who was unfamiliar with the original CTAS. The translation and back-translation were reviewed by the research group and were tested among seven clinicians to probe understanding of the items. This pre-final version of the translation was sent to Dr Jesse Wright, the developer of the original version of the CTAS, to confirm its equivalence. Disagreements and subsequent amendments by Dr Wright were reviewed. The equivalency of the English and the Japanese version was confirmed after repeating this procedure until there were no disagreements.

Participants

The CTAS-J was tested for its reliability and validity among the following sample. Study participants were recruited during the period from August to October 2009 from among the following four groups of mental health professionals: the members of Keio University CBT Study group, which consisted of a group of clinicians who were interested in CBT and has regular study experience of CBT (sample 1); the participants of a 6-hour CBT workshop held in Keio University, which targeted psychiatric residents and psychiatric nurses (sample 2); the participants of a 3-hour workshop for beginner CBT therapists held in an annual meeting of the Japanese Association of Cognitive Therapy (sample 3); and the participants of a 3-hour CBT workshop for local counselors (sample 4).

The eligibility criteria for study participation were engagement in mental health care, fluency in Japanese, not having administered CTAS-J before the study, and submission of written informed consent. The exclusion criteria were being aged above 80 years.

Among a total of 330 potential participants, six participants had a history of past CTAS-J administration and were excluded. Among the rest of the participants, written consent was obtained from 312 participants (response rate: 96.3%). Of them, 60 responses were excluded because of missing data. Finally a total of 252 responses (77.8%) were included in the analysis.

The participants' demographic data are shown in Table 1. The participants mainly comprised psychiatrists, certified clinical psychologists, and non-certified counselors. The non-certified counselors include professionals and paraprofessionals who provide generic psychosocial care in a variety of settings (usually in educational or occupational settings).

Table 1.  Demographic and educational background of the participants
  1. CBT, cognitive behavioral therapy; SD, standard deviation.

Mean age (years) (SD; range)37.9 (11.6; 21–70)
Male/female92/160
Type of occupation 
 Psychiatrist57
 Other physicians10
 Clinical psychologist56
 Nurse17
 Psychiatric social worker5
 Other non-certified couselor114
Experience in mental health care (years) (SD; range)4.8 (6.0; 0–30)
Experience in CBT training (hours)n%
None11545.6
1–57228.5
6–10218.3
11–20156
21–40114.4
40 or more187.1
Experience in CBT practice (years)n%
None16264.3
15421.4
2156
3 or more218.3
Number of readings relevant to CBTn%
None5220.6
14116.3
24417.5
33915.5
4166.3
5 or more6023.8

Procedure

After full description of the study, the participants were asked to complete the CTAS-J. The sample 1 participants completed the CTAS-J in a regular conference. The sample 2–4 participants were asked to complete the CTAS-J at the beginning of the workshop.

Along with the administration of the CTAS-J, the participants were asked for their demographic and educational backgrounds described as follows: age, gender, type of professional, their years of experience in mental health care, years of experience in CBT practice, hours of experience in CBT training, and the number of books that they had read on CBT topics.

These items were selected because they are considered to be relevant to competency in CBT. Most studies of CBT training emphasize a minimum of 24–30 h of didactic lecture, along with at least 2 hours per week of clinical work with patients and 2 hours per week of supervision for a 6- to 12-month period of time. The Academy of Cognitive Therapy,9 an authorized society that involves training and certification of cognitive therapists in the US, requires the following condition as minimum requirements for trainees to be certified: (i) at least 40 h of training specific to cognitive therapy, (ii) treatment of at least 10 patients using the cognitive model, (iii) satisfaction of at least five readings in CBT, and (iv) utilization of the cognitive model in treatment for at least 1 year prior to application. As described below, we adopted a part of these requirements as the external standard in defining ‘experienced’ trainees.

This study was conducted under participants' submission of written informed consent after full description of the study. Since this study only involved voluntary professionals, it was considered as out of range of medical ethical board.

Statistical analysis

The reliability of the CTAS-J was tested using the Kuder-Richardson Formula 20.16 This statistic is a variation of Cronbach's alpha utilized for binary data. The concurrent validity was tested by measuring correlation of the total CTAS score with the number of readings relevant to CBT, years of experience in CBT training, and years of experience in CBT practice, using Spearman's correlation coefficient. Divergent validity was tested comparing correlation between the total CTAS score and experience in general psychosocial therapy not specific to CBT.

To explore a score to discriminate experienced and non-experienced CBT therapists, the receiver operating curve (ROC) analysis was performed. We defined the participants who have had training experience of 40 h or and more and five or more readings relevant to CBT, as ‘experienced’ trainee-therapists, as suggested as a minimum requirement for certification in Academy of Cognitive Therapy.9

RESULTS

The total score of CTAS-J ranged from 18 to 39, with the mean score of 30.0 (SD = 4.3). The Kuder-Richardson formula 20 for the data was 0.76, showing good internal consistency.17

The total score of the CTAS-J was moderately correlated with the number of relevant readings (Spearman's rho = 0.44, P < 0.001), and was weakly correlated with experience in CBT education and experience in CBT practice (Spearman's rho = 0.27 and 0.28, respectively; P < 0.001) (Figs 1–3). No significant correlation was found between the total score of CTAS-J and experience in general psychosocial care (Spearman's rho = −0.02, P = 0.76: Fig. 4).

Figure 1.

The CTAS score and hours of CBT training. CTAS, The Cognitive Therapy Awareness Scale; CBT, Cognitive Behavioral Therapy. n = 252. Spearman's ρ = 0.27 (P < 0.001).

Figure 2.

The CTAS score and experience in CBT practice. CTAS, The Cognitive Therapy Awareness Scale; CBT, Cognitive Behavioral Therapy. n = 252. Spearman's ρ = 0.28 (P < 0.001).

Figure 3.

The CTAS score and number of CBT readings. CTAS, The Cognitive Therapy Awareness Scale; CBT, Cognitive Behavioral Therapy. n = 252. Spearman's ρ = 0.44 (P < 0.001).

Figure 4.

The CTAS score and experience in general mental health care. CTAS, The Cognitive Therapy Awareness Scale. n = 252. Spearman's ρ = −0.02 (P = 0.76).

The ROC curve analysis produced the area under curve (AUC) of 0.77 (SD = 0.05, P < 0.001). As shown in Table 2, a cutoff score of 31/32 produces sensitivity of 0.81 and specificity of 0.64 to differentiate trainees who have experience of 40 h or more of CBT training and have five or more readings, from those who do not meet this standard.

Table 2.  Sensitivity and specificity of the CTAS
CTAS scoreSensitivitySpecificity
  1. External standard: 40 h or more of training and five or more of relevant readings.

  2. CTAS, cognitive therapy awareness scale.

26/271.000.23
27/280.940.30
28/290.940.37
29/300.940.46
30/310.880.56
31/320.810.64
32/330.630.71
33/340.560.78
34/350.500.84
35/360.310.91
36/370.190.93
37/380.190.97
38/390.001.00

DISCUSSION

The Japanese-version Cognitive Therapy Awareness Scale was developed, and was shown to have good reliability and satisfactory validity.

The total score of the CTAS-J was moderately correlated with the number of readings relevant to CBT and weakly correlated with duration of experience and training in CBT. This is considered reasonable, because the CTAS-J is designed to test knowledge, which can be gained with reading, and is not supposed to test skills, which are expected to be gained through active participation in the training andpractice of CBT. The lack of correlation between the score of the CTAS-J and experience in general mental health care (not specific to CBT) demonstrated that the CTAS-J specifically assesses knowledge in CBT, inferring divergent validity of the scale.

The ROC analysis demonstrated that the CTAS-J has relatively weak power to differentiate experienced trainees from non-experienced trainees. The AUC of 0.77 and the specificity of 0.64 (at the score of 31/32) indicate the limitation of the cutoff. Therefore, this scale is not suitable for a rigid criterion for certification. Rather, it should be used as a self-assessment tool for trainee-therapists, it should be used with the purpose of facilitating learning in educational settings, or it should be used in assessing effectiveness of an education program.

Our study has some limitations. First, in assessing validity, the external standard relied upon therapists' self-reported training experience, and the therapists' skill was not assessed with objective measures. However, there has been no validated objective measure to assess cognitive therapists' competencies in Japan. Therefore, this issue should be considered as a future research implication. Second, test-retest reliability of the CTAS-J was not examined. However, it should be difficult because the nature of this scale measures knowledge in CBT among CBT therapists. Administration of this scale naturally prompts participants to gain knowledge.

Despite these limitations, the current study has a significant meaning not only for Japanese trainees but for English-spoken CBT trainees, because, to the best of the authors' knowledge, the optimal standard for the original CTAS score has not been established previously.

Training in CBT takes place through reading, didactic lectures, clinical shadowing and role-plays with simulated patients, which would be followed by a practice with real patients under clinical supervision.18 In order to minimize the risk of initiating treatment with lack of basic knowledge, assessment of knowledge of core concepts should be done before trainee CBT therapists embark into practice with real patients. For this purpose, the CTAS could serve as the first toll gate for CBT trainees. We propose that trainee-CBT therapists should score 32 or above on the CTAS, before they see real patients.

So far, systematic postgraduate training in psychotherapy has been lacking in Japan and this is considered as a barrier for dissemination of quality psychotherapy.19 Introducing the CTAS-J in Japanese postgraduate training can be a crucial first step for assessment-based rigorous CBT training.

In conclusion, the Japanese-version CTAS was successfully validated. Although this self-administrative questionnaire cannot be a sole method to assess competencies in CBT, it provides a rough guideline on the knowledge of CBT that trainee-CBT therapists should follow.

ACKNOWLEDGMENTS

This study was fully supported by the Grand-in-aid for from Japan Ministry of Health, Labor and Welfare to Y.O. The authors report no conflicting interests with the exception of Jesse H. Wright, MD, PhD who receives royalties from sales of books (Simon and Schuster, Guilford Press, American Psychiatric Publishing, Inc) and software for CBT (Mindstreet). The authors express gratitude for the members of the Keio University Cognitive Therapy Study Group for data collection.

Ancillary