A survey of accuracy of nurses’ clinical judgement of cutaneous graft‐versus‐host disease in Japan

Abstract Aim We examined accuracy of nurses’ clinical judgement of graft‐versus‐host‐disease (GVHD) symptoms and related factors using Common Terminology Criteria for Adverse Events (CTCAE) for patients who developed chronic cutaneous GVHD after haematopoietic stem cell transplants. Design Cross‐sectional design using nationwide survey. Methods A questionnaire survey based on Tanner's clinical judgement model to assess patients with chronic cutaneous GVHD using CTCAE was used. Free‐text descriptions and statistical analyses of relationship between correct responses and demographic data were performed. Results The rate of correct responses for main symptoms of skin GVHD was < 50%; there was no statistical significance between correct responses and demographic data, knowledge about GVHD and collaborative practice with physicians. The accuracy of cutaneous GVHD clinical judgements was not directly related to nurses’ background. Educational opportunities that reinforce nurses’ abilities to reflect on knowledge and experiences to interpret patient symptoms are essential for improving accuracy of clinical judgement.

been increasing annually (Niederwieser et al., 2016). Among these, approximately 5,500 HSCTs are performed annually in Japan, ranking second after the United States in terms of frequency (The Japanese Data Center for Hematopoietic Cell Transplantation/The Japan Society for Hematopoietic Cell Transplantation, 2017).
The 5-year survival rate after HSCT has increased to 53.2% with autologous transplantation (The Japanese Data Center for Hematopoietic Cell Transplantation/The Japan Society for Hematopoietic Cell Transplantation, 2019), although approximately 55% of transplant recipients develop GVHD (Rodrigues et al., 2018), thereby contributing to the decline in patients' survival and prognoses as well as quality of life.
At present, there is no definitive treatment for GVHD; immunosuppressive drugs and skin care are used to alleviate symptoms. In addition, chronic skin GVHD symptoms may persist for several years, resulting in physical and psychological distress among patients (Yokota et al., 2011). The appropriate monitoring of skin symptoms by nurses is critical for the early detection and treatment of chronic GVHD and is also expected to contribute to the alleviation of skin symptoms through the implementation of appropriate skin care (Flowers & Martin, 2015). To detect/treat cutaneous GVHD early and choose appropriate skin care for alleviating symptoms, nurses' accurate clinical judgements using multidisciplinary indicators are required.
The process of nurses' clinical judgements was modelled by Tanner (Tanner, 2006); the model includes four aspects: "Noticing," "Interpreting," "Responding," and "Reflecting" (Figure 1). According to this model, "Noticing" is the first aspect where skin symptoms and the status of patients are observed to gain an overall understanding of the situation. The next aspect is "Interpreting," where symptoms are correlated to information such as clinical knowledge and the nurses' own experiences to obtain sufficient understanding, followed by "Responding," wherein clinical judgements and skin care method decisions are made. The final aspect is "Reflection," wherein patients' responses are observed, and a decision is made regarding whether or not the interpretation of information and judgement were correct.
The model states that the nurses' background, such as clinical knowledge and experiences, the political and social context, interdisciplinary relationships and disproportionate relationships, particularly with physicians, influence clinical judgements (Tanner, 2006).

One of the indicators to evaluate cutaneous GVHD is the Common
Terminology Criteria for Adverse Events (CTCAE). CTCAE was developed at the National Cancer Institute and is recommended for use across multiple job functions (Japan Clinical Oncology Group [J.C.O.G., 2017]) as an indicator for evaluating unexpected symptoms associated with cancer treatment. CTCAE has also been used in symptom assessment in clinical studies and many studies have been conducted by nurses (Nagao et al., 2016;Oki et al., 2016;Yabuki et al., 2016). However, there have been no studies evaluating cutaneous GVHD using CTCAE.

| Research question
This study aimed to: (a) determine the accuracy of nurses' clinical judgements of skin symptoms using CTCAE for patients who developed chronic skin GVHD after HSCT; and (b) explore factors related to the accuracy of nurses' clinical judgements of skin symptoms using CTCAE.

| Design
Cross-sectional design using a nationwide postal and web survey and content analysis of free descriptions.

| Questionnaire development
The questionnaire development process is shown in Figure 2. The developed case is shown in Figure 3. The questionnaire asked the F I G U R E 1 Clinical judgment model (Tanner, 2006) following questions; 1) Personal Factors measured included demographic data: (a) years of clinical experience; (b) years of HSCT nursing experience; (c) presence or absence of experience in caring for patients with cutaneous GVHD; (d) job role; (e) educational attainment; (f) use of CTCAE to assess cutaneous GVHD); and the "interdisciplinary relationships" was measured using the Japanese version F I G U R E 2 Questionnaire development process The collaborators (nurses with 10 years of clinical HSCT experience who had master's degrees and clinical educational roles) and authors selected images showing the most typical symptoms of chronic skin GVHD from a series of images obtained with the consent of patients for record keeping and teaching.
Following approval from the institutional review boards of the respective collaborators' institutions, consent to use the images in this study was obtained from the patients.
The authors and collaborators prepared example patients by referring to multiple previous patients.
Example patients were corrected based on the advice of 2 experts (a chief hematologist at a cancer center and a hematologist with >10 years of clinical experience) and content validity was ensured.
One dermatologist in charge of the treatment of patients with GVHD at a university hospital evaluated the validity of the grades assigned by the two hematologists. Finally, 7 of the 17 terms (skin induration, pruritus, erythroderma, maculopapular rash, dry skin, nail loss, and nail ridging) were classified as Grade 1 for the example patients.
Each hematologist graded the 17 terms as "Grade 0: does not apply to the example patient" or Grades 1-4.
Among all 794 CTCAE terms, items clearly not applicable to the example patients (e.g., alopecia) were excluded from the 34 terms under "skin and subcutaneous tissue disorders" and the remaining 17 terms were used for grading.
of the Collaborative Practice Scales-Version for Nurses (hereinafter, CPS). The scale consists of a total of nine items, including two subscales for measuring the self-assertiveness towards physicians, with four items on "expert knowledge and asserting opinions," and five items for "clarifying each other's expectations of joint responsibilities." This was evaluated on a six-point Likert scale, the lowest score indicating "not practiced at all" (one point) to the highest score indicating "always practiced" (six points); the higher the total score, the more collaborative practice was carried out with physicians.
Cronbach's α for the Japanese version of this scale was reported to be 0.92 and it has been confirmed that the Japanese version of CPS for nurses was consistent with the original version. The Japanese version of CPS for nurses' reliability and validity has been ensured (Komi et al., 2010).
3) The clinical knowledge test regarding cutaneous GVHD was internally prepared via the following procedures. Fifty questions were prepared to obtain knowledge on the pathophysiology of GVHD, skin assessments and skin care with reference to the 51 questions on GVHD and skin care from the "Clinical and educational ladder for nurse to be engaged in haematological cancer nursing including HSCT" created by The Japan Society for Hematopoietic Cell Transplantation. The questions were reviewed by two haematologists, one dermatologist (an expert in cutaneous GVHD treatment), one clinical nurse with a master's degree and nursing experience in HSCT patients and one clinical nurse with a master's degree in nursing to investigate content validity. A total of 25 questions that may be prioritized as knowledge that nurses should have was selected by these internal experts. Questions were written in the form of "yes" "no" questions. Correct answers scored 1 point each for a maximum score of 25 points.

| Questionnaire distribution
Paper-and web-based questionnaires were prepared to record symptom assessments using free text. Both paper-and web-based questionnaires included same questions and the participants could choose either one as per their convenience. Questionnaire pretesting was performed on a total of six participants, two of whom were nurses with HSCT nursing experience (one was a certified nurse spe-

cialist [CNS] in oncology nursing) and four without such experience,
to ensure face validity.
Questionnaires were sent to target participants using the following methods. Registered nurses (RNs) were sent questionnaires by mail to the directors of the nursing departments of the target sites along with a document explaining the purpose of the study and an access method manual for the web-based questionnaire. CNSs and certified nurses (CNs) were sent questionnaires directly to their prospective sites with a document explaining the purpose. The participants responded by returning the questionnaire or submitting responses online for paper-and web-based questionnaires. Submitted responses were considered consent to participate in the study.

| Participants
The paper-or web-based questionnaires that included the same questions were distributed to a total of 3,022 participants, includ-

F I G U R E 3
Summary of a chronic GVHD example patient and assessment of skin disorders using CTCAE. A female patient in her 40s and a housewife. Aplastic anemia observed 90 days after HLA-identical unrelated allogeneic HSCT. Disease history: Engraftment was confirmed 7 days after transplantation, and the patient left the clean room on Day 40 after transplantation. The patient is currently hospitalized in the general ward. She presented with Grade 2 GVHD symptoms from Day 21 after transplantation and extensive skin induration from the upper arms to the fingers of both hands from approximately Day 80 after transplantation. Skin rupture of the wrist joint with exudate and skin desquamation were observed, which partially restricted her ADL. Symptoms were limited to the area shown in the image. No ointment or skin care has been used. At present, Prograf (graceptor) is being administered to the patient, the compliance with oral administration is good, and she is scheduled to be discharged from the hospital shortly. Social background: Family of 4 including a husband (40 years old), son (10 years old) and daughter (4 years old), and the patient's mother is living in their neighborhood. The patient mentioned "being worried about returning home (with hands) looking like this," and "her children getting scared upon seeing her hands," and she appeared discouraged. CTCAE: Erythroderma Grade 1, dry skin Grade 1, nail loss Grade 1, nail ridging Grade 1, pruritus Grade 1, maculopapular rash Grade 1, skin induration Grade 1 Japanese CNSs are advanced practice nurses who have a master's degree. CNs are those who have obtained qualifications after completing training for 6 months at a training centre certified by the Japanese Nursing Association (Japanese Nursing Association, 2016).
There were no restrictions on age, sex, number of years of nursing experience, or type of employment.

| Statistical analysis
To verify if the CTCAE terms corresponding to events related to example patients were correctly selected, when "Grade 1 or higher" (symptoms present) was selected for the seven terms corresponding to events related to example patients or when Grade 0 (without symptoms) was selected for the terms not corresponding to events related to example patients, 1 point was given; on the other hand, responses not fitting the above conditions were given 0 points and scores were obtained for each response (CTCAE scores).
Consequently, CTCAE scores were correlated with the number of years of nursing experience, CPS and the clinical knowledge test.
In addition, t-tests were performed for each pair of nurses with/ without CN/CNS qualifications, nurses with/without experience in providing care for patients with GVHD and using or not using CTCAE for assessment.
Subsequently, to determine whether there was a significant difference in the number of correct responses among the CTCAE terms and if responses of Grade 1 or higher were given for the seven terms corresponding to events related to example patients, the frequency of responses of Grade 0 among the 10 terms not corresponding to events related to example patients was obtained. Thereafter, the Friedman test was performed to assess the difference between events. When significant differences were found, pairwise comparisons were performed with multiple comparisons and Bonferroni's correction.
Statistical tests were performed using SPSS Statistics Base ver.
26 and p < .05 was considered statistically significant. Priori analyses were performed using G*Power 3.1. The required sample size for G*Power 3.1 was 136 (effective size: 0.4, power: 0.8).

| Comparison of free-text entries
To clarify the differences in skin assessment, the content of freetext entries was analysed using methods based on content analysis by Krippendorff. A group with higher CTCAE scores (higher CTCAE score group) and another with lower CTCAE scores (lower CTCAE score group) were created and free-text entries regarding each skin assessment were used as raw data, with one sentence or each related sentence regarded as one unit and coded. Codes were summarized and classified based on similarities and differences in semantic content, and categories were created. The categories and codes in the study groups were compared. The validity of the classification of free-text entries was evaluated by four nursing researchers.

| Ethics
This study was conducted in accordance with the Declaration of

| Providing care and CTCAE use status
Approximately 50% participants had experience of providing care to patients with chronic skin GVHD. Of these, 23.6% responded with "symptoms assessed using CTCAE for skin disorders" (Table 2).
There were no significant differences in CTCAE use between CNSs/ CNs and RNs.

| Relationship between CTCAE scores and nursing experience or advanced practice nursing qualifications
The mean CTCAE score was 11.9 (SD = 2.05) points (range: 6-17) of a total of 17 points and the mean score when converted to a total of 100 points was 69.9 (SD = 12.1) points.

| Correlations among knowledge test, collaborative practice scales and CTCAE scores
The mean score of the knowledge test was 18.20 of 25 points [SD = 2.14, range 11-23] and the mean total CPS score was 3.49 of Correlations among CTCAE scores, CPS and knowledge tests were r < 0.1 (p > .05).

| Accuracy of CTCAE grading
The frequency of each grade for the seven terms including symptoms of chronic GVHD is shown in Table 3. The rate of correct responses for all terms was < 60% and that for maculopapular rash and erythroderma (monitoring indices) was only 10%-20% (Table 4).

| Free-text entry of assessments
Responses of 27 and 43 participants in the lower and higher (6-9 and 14-17 points, correct responses: 35.3%-52.9% and 82.4%-100%), respectively, were extracted. There were no significant differences between these groups in terms of the codes for "dry skin/epidermolysis," "scleroderma," and "change in skin colour," which are symptoms of chronic skin GVHD. However, regarding the code for "skin care based on the assessment," the higher CTCAE score group indicated the skin care purpose as a means "to not exacerbate symptoms" and the content of skin care was listed in detail according to symptoms like dryness and nail protection. The entered content in the lower CTCAE score group did not specifically describe some characteristics such as affected activities of daily living (ADL) and there was no specificity in the content of skin care based on skin assessment (Table 5).

| D ISCUSS I ON
The recovery rate for the study questionnaire was low at 7.8% (effective response rate: 91.1%). Since the enactment of the Act on the Protection of Personal Information in 2003, a decline has been observed in the recovery rate of questionnaires via mail in Japan (Go, Hiroyuki, & Satoshi, 2006), with the highest rate estimated to be 20% (Hayashi, 2016). This survey showed that the institution type, number of transplants performed, basic educational history of participants, sex and care history of patients with skin GVHD are adequately reflected among nurses employed at HSCT sites in Japan.
The required number of samples was achieved to obtain sufficient statistical power analysis.

| Differential diagnosis of chronic skin GVHD and grading with CTCAE
The grading of characteristic symptoms such as maculopapular rash and skin induration are considered based on "effects on daily life," which is roughly classified under "instrumental ADL," and "selfcare ADL,"; however, clear criteria have not been specified even in CTCAE ver. 5.0. There is a possibility that these weak points may be related to the low CTCAE score. In this study, the rate of selection for main symptoms of skin GVHD, that is maculopapular rash and erythroderma, were < 50%. Approximately 75% of nurses could not correctly differentiate between the main symptoms and erythema multiforme/purpura. The accuracy of the assessment of skin GVHD did not merely on experience in providing care or advanced practice nursing qualifications. A previous study (Peuvrel et al., 2018) identified: "(a) the selection of appropriate symptom categories; and (b) assigning grade" as concerns in assessing skin symptoms using CTCAE. The accuracy of assessments using CTCAE is also important to improve (c) the accuracy of differential diagnoses for skin eruptions and diseases associated with skin colour changes.
On the other hand, in the free-text entries on assessments of subjective skin symptoms, such as itchiness, the higher CTCAE score group described many free-text entries of such symptoms in relation to specific ADL and the hypothesis is that an assessment based on this relationship leads to accurate differential diagnoses. In this study, nurses with higher scores made an evaluation of concrete skin care methods based on patient ADL in addition to subjective symptoms with the objectives of "preventing the lowering of ADL." ADL of patients with skin GVHD rely on hands and joint mobility, which has a close relationship with subjective symptoms of skin GVHD.
There is a possibility that nurses who gave higher scores diagnosed subjective skin symptoms more carefully to choose appropriate skin care methods for preventing the lowering of ADL and this may help in grading CTCAE more accurately. *Items in which "degree to which symptoms affect activities of daily living (ADL)" was used as criterion for grading TA B L E 5 Examples of free-text entries in the higher CTCAE score group (n = 43)

| Factors related to clinical judgement
The skin of the finger joint is broken, there is exudate, the defence mechanism of the skin is dysfunctional, and the skin is in an infection-prone condition. In addition, finger numbness and skin tightness in the wrist interferes with activities of daily life (ADL). The attending physician is considering discharge. Patients understand the benefit of Prograf and agree to continue the drug. However, patients seem to be anxious about ADL/IADL restrictions due to the present state of his/her fingers and the reactions of his/her family to the skin condition. Skin tightness and scleroderma-like symptoms are present. There is a need to start performing sufficient skin care because it is not being performed. It is necessary to promote skin softening. Is there speculation of whether patients can take the tablet by themselves (take the tablet and consume)? Is it necessary to administer medical treatments such as ointments other than oral medications? Is there anything that can be used to treat itching? → There is a need to confirm the necessity of medical measures for skin symptoms and to give guidance on self-care for daily life. not be measured. However, the total CPS score and subscale scores of participants in this survey were higher than the mean for the Japanese version reported by Komi et al. (2010) and "disproportionate relationships with physicians" that affected clinical judgements were minimal.
Interestingly, in the current survey, clinical knowledge and experiences, which are believed to improve clinical judgements, were also not directly related to CTCAE scores. According to the CJM, grasping the situation by "Noticing," then relating and analysing the knowledge and information in the process of "Interpreting" are important aspects to ensure accurate judgements are made (Tanner, 2006). In the higher CTCAE score group, a large number of free-text statements about ADL were given and these analyses may have helped not only in the selection of skin care methods but also in grading CTCAE as a narrative analysis in the CJM. The lack of direct correlations between collaborative relationships with physicians, clinical knowledge and nurses' experiences with clinical judgements supported the CJM concept that "background factors" such as clinical knowledge and experiences do not translate into correct clinical judgements if the appropriate interpretation is not made. Various educational interventions using CJM are now being considered (Nielsen, 2016;Timbrell, 2017); educational interventions to improve the clinical judgement of cutaneous GVHD using CTCAE are essential.

| Limitations
Although the target participants seemed to reflect the actual conditions of nurses who work at HSCT sites in Japan, based on the low recovery rate, the possibility that only nurses who had a high interest in skin GVHD actively responded cannot be excluded.

| Conclusion
• Nurses were not able to discriminate and grade key symptoms that serve as monitoring indicators for clinical judgements using CTCAE for cutaneous GVHD.
• The accuracy of cutaneous GVHD clinical judgements was not directly related to nurses' background.
• To make accurate judgements, the mastery of experiences and knowledge and the process of "Interpreting," wherein knowledge is correlated to patient information and appropriate analyses are performed, are crucial.
• Continuing education opportunities to increase the accuracy of interpretation and improve clinical judgements are needed.

ACK N OWLED G M ENT
The abstract of this study was presented at the International

CO N FLI C T O F I NTE R E S T
None of the authors have any conflicts of interest or any financial ties to disclose.

DATA AVA I L A B I L I T Y S TAT E M E N T
The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to restrictions, for example their containing information that could compromise the privacy of research participants.