SEARCH

SEARCH BY CITATION

Abstract

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. AUTHOR CONTRIBUTIONS
  8. Acknowledgements
  9. REFERENCES

Objective

To determine the minimal clinically important differences (MCIDs) of validated measures of systemic lupus erythematosus (SLE) disease activity in childhood-onset SLE.

Methods

Childhood-onset SLE patients (n = 98) were followed every 3 months for up to 7 visits (n = 623 total visits). Disease activity measures (European Consensus Lupus Activity Measure, Systemic Lupus Erythematosus Disease Activity Index, Systemic Lupus Activity Measure, British Isles Lupus Assessment Group, and Responder Index for Lupus Erythematosus [RIFLE]) were completed at the time of each visit. Physician-rated changes in the disease course (clinically relevant improvement, no change, clinically relevant worsening) between visits served as the criterion standard.

Results

MCIDs defined by mean change scores with improvement and worsening, or those based on the standard error of measurement with stable disease, were both small and did not discriminate well between disease courses (detection rates for improvement or worsening were all <55%). MCIDs based on discriminant and classification analyses yielded similar results. Alternative MCIDs, defined by a 70% predicted probability of improvement or worsening as per the discrimination analysis, were larger but underestimated the proportion of patients with change. The RIFLE only correctly identified 26% and 8% of episodes of clinically important worsening and improvement of childhood-onset SLE, respectively.

Conclusion

The MCIDs of childhood-onset SLE disease activity measures are often small but similar to those reported for adults with SLE. Therefore, even small changes in disease activity scores can be clinically relevant. Low correct detection rates of these MCID thresholds for changes in disease course support the notion that worsening and improvement with childhood-onset SLE, or its response to therapy, is unlikely to be captured adequately by validated measures of disease activity alone.


INTRODUCTION

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. AUTHOR CONTRIBUTIONS
  8. Acknowledgements
  9. REFERENCES

Systemic lupus erythematosus (SLE) is a complex, chronic multisystem autoimmune inflammatory disease that targets young women and men (1, 2). The up to 20% of SLE patients who are diagnosed during childhood, i.e., prior to age 16 years, tend to experience a more severe disease course than those with disease onset later in life (3–5). Various measures of global disease activity have been developed for SLE in adults and subsequent validation confirmed that these indices have concurrent validity for measuring disease activity of childhood-onset SLE (6–8). These consist of the Systemic Lupus Erythematosus Disease Activity Index (SLEDAI) (9), the Systemic Lupus Activity Measure (SLAM) (10), the European Consensus Lupus Activity Measurement (ECLAM) (11), and the British Isles Lupus Activity Group (BILAG) Index (12). Conversely, the Responder Index for Lupus Erythematosus (RIFLE) was developed specifically to define treatment response, i.e., clinically meaningful change in SLE over time (13), but has not been used in childhood-onset SLE.

The concept of a minimal clinically important difference (MCID) was introduced by Jaeschke et al, who defined the MCID as “the smallest difference in a score of a disease measure of interest that patients perceive as beneficial and that would mandate, in the absence of side-effects, a change in the patient's management” (14). Since then, the Outcome Measures in Rheumatology Clinical Trials group has explored the concept of MCIDs in depth (15, 16). There are different types of MCIDs, depending on whether improvement or worsening is considered and what external standard is employed. MCIDs constitute threshold values for clinically relevant change, i.e., special features of the responsiveness to change of a disease activity index. Any amount of change greater than the MCID threshold can be considered clinically meaningful or important, whereas any change score smaller than the MCID, irrespective of statistical significance, is clinically irrelevant.

Various statistical approaches have been suggested to calculate the MCID for an outcome measure (17). In addition to determining the MCID itself, it is also of interest to assess how well an index can discriminate patients in whom a clinically important change has occurred from others.

The objective of this study was to determine the MCIDs of validated measures of disease activity when used in childhood-onset SLE from a physician's and parent's perspective, using previously proposed statistical methods. We also wished to assess the ability of the disease activity indices to identify patients in whom a clinically relevant change of childhood-onset SLE has occurred and to study the usefulness of the RIFLE for capturing clinically relevant change in childhood-onset SLE.

MATERIALS AND METHODS

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. AUTHOR CONTRIBUTIONS
  8. Acknowledgements
  9. REFERENCES

Patients.

Children (n = 98) fulfilling the American College of Rheumatology (ACR) classification criteria for SLE (2) were recruited at 7 pediatric rheumatology centers and studied every 3 months for up to 18 months. Disease activity and change in the course of childhood-onset SLE were measured at each study visit.

Measures of disease activity.

In addition to the SLAM (10), SLEDAI (9), and ECLAM (11), the BILAG Index (12, 18) was completed. To convert the alphabetical domain scores of the BILAG Index to numerical childhood-onset SLE activity scores, 3 alternative schemes were considered, as suggested by Gladman et al (A = 4, B = 3, C = 2, D = 1, and E = 0) (19), Liang et al (A = 10, B = 6.7, C = 3.3, D = 0, and E = 0) (10), and Stoll et al (A = 9, B = 3, C = 1, D = 0, and E = 0) (20). None of these schemes has been well validated in childhood-onset SLE or SLE. For all of the abovementioned disease activity indices, a score of 0 reflects inactive disease.

RIFLE.

The RIFLE is a 60-item questionnaire to measure change of disease activity using a 5-point Likert scale: not present, partial response, complete response, present, or worsening (13, 21). It has been suggested that clinically important worsening of SLE is present if there are at least 3 RIFLE items with “worsening” conditions; similarly, clinically important improvement occurs when there are at least 4 RIFLE items with “partial response” and/or “resolution” conditions. This tool has not yet been validated for use in childhood-onset SLE.

Global rating of disease activity.

The managing pediatric rheumatology professional completed a visual analog scale (VASMD; where 0 = inactive and 10 = very active) presented with the sentence stem: “Considering the findings at today's visit, the overall disease activity of the patient is (Please circle the number that appears most appropriate).” All of the raters underwent detailed and repeated training in completing the abovementioned disease activity measures.

Measures of change in SLE course: criterion standards.

In response to the sentence stem, “Compared to the last study visit three months ago and the patient's overall disease, the patient experienced a…,” the managing pediatric rheumatology professional rated the change in the disease course between consecutive visits on a 5-point Likert scale as follows: major flare of disease, minor flare of disease, no change in disease, minor improvement of disease, or major improvement of disease. All pediatric rheumatology professionals who provided the above ratings for the course of childhood-onset SLE, i.e., information about the external standard used for this validation exercise, were board certified or board eligible and, on average, see 20 patients with childhood-onset SLE per week in their academic center and have a 10-year experience in treating childhood-onset SLE.

In the secondary analysis, we assessed how the scores of the disease activity indices changed with the course of childhood-onset SLE as reflected by the family's perspective. Therefore, the parent rated the change of their child's disease on a 5-point Likert scale (much worse, somewhat worse, unchanged, somewhat improved, or much improved) that was presented with the sentence stem: “Compared to the last study visit three months ago, and when considering medications, school, work, life at home, doctor visits, pains and feelings the overall well-being is … .”

Statistical analysis.

Various approaches to assessing a measure's MCID have been proposed (22, 23). After review of the medical literature, we considered statistics that appeared to be most commonly employed to measure MCIDs: first, we used mixed models in longitudinal analyses that considered that each patient had up to 7 study visits, and the mean score change with improvement of disease activity was determined by assessing the changes in scores of the disease indices in patients who were rated either as showing “minor improvement” or “major improvement.” Similarly, the mean score change with worsening of disease activity considered changes of the scores of the disease indices in patients who were rated as having a “minor flare” or “major flare,” respectively. When assessing for significant differences in change scores between groups (here: clinically relevant improvement, no change in disease, or clinically relevant worsening), P values were corrected under the mixed-model framework using a Tukey procedure. A random effect was used, i.e., patients were introduced in the mixed-effects model to account for within-patient correlation caused by the repeated measurement.

Alternatively, the MCID can be defined based on the standard error of measurement (SEM) of changes in the disease activity scores with stable disease, i.e., patients rated as having “no change in disease” between consecutive visits; the MCID is then based on the so-called one-SEM criterion proposed by Wyrwich et al (24, 25). As done in the past, the SEM was defined as the square root of the within-episodes mean square error variance (calculated from an analysis of variance model), using both episodes (nested in patients) and visits as fixed effects and accounting for within-patient (or between-episodes) correlation using a generalized estimating equation method in computation (26). Besides the traditional MCID thresholds at ±1 SEM (or, equivalently, the 63% confidence interval [63% CI]), we explored alternative MCID thresholds at ±1.645 SEM (or, equivalently, the 90% confidence interval [90% CI]). Furthermore, to assess the diagnostic accuracy of the MCID with clinically important improvement, we calculated the detection rate, i.e., the proportion of correctly identified episodes of improvement among the total episodes of improvement (as per the criterion standard), for each disease index. Accordingly, the detection rates with clinically worsening or stable disease courses were calculated.

In a third approach to determining the MCID, each disease activity measure was assessed for its ability to discriminate between the disease courses (improved versus no change versus worsening) using classification and discrimination modeling (27). Linearized discrimination functions were done to calculate predicted probabilities for clinically important improvement, no change in disease, and clinically important worsening of disease for a given change score of the disease activity measure under the classification and discrimination model framework.

Similar to what has been suggested by the ACR Ad Hoc Committee on SLE Response Criteria (26), clinically relevant changes of indices may be defined as the change score of disease activity measures with a 70%, 80%, or 90% predicted probability of an event to have occurred (here: clinically relevant improvement or clinically relevant worsening), i.e., each observation is assigned a probability of belonging to a given group based on the distance of its discriminant function from that of each class mean.

We also calculated intraclass correlation coefficients (ICCs) to assess chance-corrected agreement of the activity measures' change scores with stable courses (28), using a similar approach as detailed above.

For the RIFLE, kappa statistics were calculated to assess its agreement with the criterion standards. Like kappa coefficients, ICC values can be interpreted as follows: ICCs <0.4 = poor agreement, ICCs ≥0.4 to 0.75 = fair to good agreement, and ICCs ≥0.75 = excellent agreement (28).

In the secondary analysis, the analysis detailed above was repeated, using the parents' ratings of the patients' changes in well-being between visits (instead of the physician's assessment of childhood-onset SLE disease course) as the criterion standard for determining the MCIDs of the SLE disease activity measures. All of the analyses were done using SAS software, version 9.2 (SAS Institute). P values less than 0.05 were considered statistically significant.

The study was approved by the institutional review boards of the participating pediatric rheumatology centers. Informed consent was obtained from all of the parents and, as appropriate, assent was given by the participants prior to the study procedures.

RESULTS

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. AUTHOR CONTRIBUTIONS
  8. Acknowledgements
  9. REFERENCES

Characteristics and disease course of patients with SLE.

The demographics and disease features of the patients with childhood-onset SLE are summarized in Table 1. Data from a total of 623 visits (or 526 between-visit intervals) of 98 children were available for analysis. There were 39 patients with biopsy-proven lupus nephritis. As per the managing pediatric rheumatology professionals, the courses of childhood-onset SLE between consecutive study visits consisted of 89 episodes of clinically relevant worsening (12 major worsening, 77 minor worsening), 348 episodes of stable disease between visits, and 89 episodes of improvement (14 major improvements, 75 minor improvements).

Table 1. Demographics and systemic lupus erythematosus features at baseline*
ParameterNMean ± SD
  • *

    SDI = Systemic Lupus International Collaborating Clinics/American College of Rheumatology Damage Index; SLAM = Systemic Lupus Activity Measure; SLEDAI = Systemic Lupus Erythematosus Disease Activity Index; BILAG = British Isles Lupus Assessment Group; ECLAM = European Consensus Lupus Activity Measurement.

  • World Health Organization classification of lupus nephritis: class II (n = 2), class III (n = 8), class IV (n = 21), and class V (n = 8).

  • Where 0 = no damage.

  • §

    Visual analog scale of disease activity as rated by the pediatric rheumatologist (VASMD; range 0–10, where 0 = inactive disease and 10 = very active disease).

Age, years9815.3 ± 2.85
Disease duration, years981.5 ± 2.0
Current medications  
 Prednisone, mg/day8215.1 ± 1.8
 Azathioprine, mycophenolate mofetil,  methotrexate47 
 Cyclophosphamide6 
 Hydroxychloroquine73 
 Nonsteroidal antiinflammatory drugs24 
 At least 1 antihypertensive medication38 
Biopsy-proven lupus nephritis39 
Disease damage: SDI score980.42 ± 0.1
Physician assessment of overall disease activity (VASMD)§982.5 ± 1.95
Disease activity98 
 SLAM 7.64 ± 6.01
 SLEDAI 5.18 ± 4.35
 Stoll et al BILAG Index 5.31 ± 5.44
 Liang et al BILAG Index 11.8 ± 8.81
 Gladman et al BILAG Index 8.7 ± 3.57
 ECLAM 1.85 ± 1.75

From the families' perspectives, there were 59 episodes of worsening of well-being (9 major worsening, 50 minor worsening), 253 episodes of stable well-being, and 202 episodes of improved well-being (108 minor improvements, 94 major improvements). For 12 between-visit intervals, no family ratings were available.

MCID: mean changes in the scores of activity measures with improvement or worsening.

Between-visit changes of the VASMD and the scores of the disease indices are summarized in Table 2. Despite statistical significance, but irrespective of the index, the mean change scores were all small and close to the smallest possible difference in score, which is 1 for each of these tools. Therefore, increases of the ECLAM scores as small as 1 appeared to be clinically relevant, whereas for the Liang et al BILAG Index, increases of 3 could be considered as clinically important. With clinically relevant improvement of childhood-onset SLE, decreases in the scores of the disease activity indices were somewhat larger.

Table 2. Change in disease activity scores with physician-rated and parent-rated childhood-onset systemic lupus erythematosus (SLE) course*
Change in core set variablesWorsening (n = 89 episodes)No change (n = 348 episodes)Improvement (n = 89 episodes)Worsening vs. no changeWorsening vs. improvementNo change vs. improvement
  • *

    Values are the mean ± standard error. NS = not significant; see Table 1 for additional abbreviations.

  • P values are based on mixed models using the Tukey procedure for post hoc correction.

Physician-rated change in childhood-onset SLE course      
 VASMD1.23 ± 0.12−0.18 ± 0.06−1.01 ± 0.12< 0.001< 0.001< 0.001
 ECLAM0.97 ± 0.160.02 ± 0.08−1.12 ± 0.16< 0.001< 0.001< 0.001
 SLEDAI1.56 ± 0.33−0.09 ± 0.17−2.27 ± 0.33< 0.001< 0.001< 0.001
 SLAM1.78 ± 0.76−0.01 ± 0.38−1.78 ± 0.78< 0.1< 0.01NS
 Liang et al BILAG Index3.23 ± 0.70−0.62 ± 0.35−4.61 ± 0.70< 0.001< 0.001< 0.001
 Gladman et al BILAG Index1.18 ± 0.23−0.08 ± 0.12−1.22 ± 0.23< 0.001< 0.001< 0.001
 Stoll et al BILAG Index1.85 ± 0.39−0.35 ± 0.20−2.79 ± 0.39< 0.001< 0.001< 0.001
Parent-rated change in patient well-being      
 VASMD0.42 ± 0.18−0.03 ± 0.09−0.24 ± 0.09< 0.1< 0.01NS
 ECLAM0.65 ± 0.220.03 ± 0.10−0.25 ± 0.11< 0.05< 0.001NS
 SLEDAI1.04 ± 0.45−0.03 ± 0.21−0.66 ± 0.23< 0.1< 0.01NS
 SLAM1.83 ± 0.98−0.29 ± 0.46−0.04 ± 0.50NSNSNS
 Liang et al BILAG Index2.98 ± 0.93−0.34 ± 0.44−1.81 ± 0.47< 0.01< 0.001< 0.1
 Gladman et al BILAG Index1.22 ± 0.300.02 ± 0.14−0.43 ± 0.15< 0.01< 0.001< 0.1
 Stoll et al BILAG Index1.53 ± 0.53−0.29 ± 0.25−0.97 ± 0.27< 0.01< 0.001NS

MCID: SEM with stable disease course and chance-corrected agreement.

Disease measures most often remained unchanged, with disease courses rated as “no change in disease” by the managing pediatric rheumatology professional (Table 2). Chance-corrected agreement of change score of activity index with a stable disease course was “excellent” for the VASMD (ICC 0.76) and “good” for all other disease indices (all ICCs ≥0.47) (Table 3).

Table 3. MCID based on the one-SEM criterion using physician-rated change in childhood-onset systemic lupus erythematosus (SLE) as the external standard*
Disease activity measureICCMCID definitionMCID for clinically important improvementMCID for clinically important worseningDetection rate for clinically important improvement, %§Detection rate for no change, %Detection rate for clinically important worsening, %Overall correct detection rate of all childhood-onset SLE courses, %
  • *

    MCID = minimal clinically important difference; see Table 1 for additional abbreviations.

  • The intraclass correlation coefficient (ICC) presented considers random effects in mixed-effect models (28). High ICC values (range 0–1) indicate a high level of chance-corrected agreement of the changes of disease activity scores among patients with stable disease course (or unchanged disease) between consecutive visits.

  • The standard error of measurement (SEM) is the square root of the “within-subject variability or within-subject mean square error” (24). The one-SEM criterion assumes that the MCIDs for worsening and improvement have the same absolute values. In addition to the one-SEM criterion, i.e., the 63% confidence interval (63% CI) at ±1 SEM around the mean change of the scores of disease activity measures in patients with stable disease, an alternative 90% confidence interval (90% CI) at ±1.645 SEM is shown that more accurately detects patients with a stable disease course.

  • §

    Defined as the number of improved patients as per the MCID value considered divided by the number of the truly improved patients as per the criterion standard (physician assessment of change in disease). The detection rates for worsening and unchanged disease courses were calculated accordingly.

VASMD0.7663% CI of SEM−0.7+0.771616563
  90% CI of SEM−1.2+1.229903571
ECLAM0.4763% CI of SEM−0.9+0.949515552
  90% CI of SEM−1.5+1.538753963
SLEDAI0.6363% CI of SEM−1.9+1.958524552
  90% CI of SEM−3.1+3.131773362
SLAM0.5763% CI of SEM−3.8+3.829743660
  90% CI of SEM−6.3+6.318841762
Liang et al BILAG Index0.6263% CI of SEM−4.2+4.240724061
  90% CI of SEM−6.9+6.930862166
Gladman et al BILAG Index0.7763% CI of SEM−1.4+1.440704060
  90% CI of SEM−2.3+2.330842966
Stoll et al BILAG Index0.5863% CI of SEM−2.0+2.051654960
  90% CI of SEM−3.3+3.334862767

Alternatively, as is shown in Table 3, the MCID can be based on the one-SEM criterion, assuming that important improvement or worsening has occurred if the change score exceeds −1 SEM or +1 SEM, respectively. We also tested a more stringent MCID definition, i.e., the 90% CI around the mean change score or ±1.645 SEM (Table 3). As is reflected by the respective detection rates, the tighter the confidence interval limit was set, the more accurately patients with a stable disease course could be discriminated from those who experienced clinically relevant change (Figure 1), but this occurred at the expense of decreased rates of correctly identified patients with clinically relevant change in disease (Table 3).

thumbnail image

Figure 1. Changes in the Systemic Lupus Erythematosus Disease Activity Index (SLEDAI) scores and the physician global rating of disease activity on a visual analog scale (VASMD) with improved, worsening, and stable disease course. For the VASMD and the SLEDAI, the minimal clinically important difference (MCID) as defined by a ±1 standard error of measurement (SEM) or ±1.645 SEM threshold are depicted (A), as well as their diagnostic accuracy (B). A, Scatter plots of the change ratings on the VASMD and change scores of the SLEDAI under consideration of different disease courses. Two different sets of broken lines represent thresholds at ±1 SEM (or 63% confidence interval) and ±1.645 SEM (or 90% confidence interval). B, The probabilities of correctly detecting the disease course (or detection rate) are on the y-axis under consideration of different MCID (or SEM) thresholds on the x-axis. The larger the confidence interval (CI) around the mean change score of the disease measure of patients with stable disease, the more likely will patients who have improved or worsened be misclassified as having “no change in disease.”

Download figure to PowerPoint

For example, when setting the MCID value of the SLEDAI to the one-SEM (63% CI) boundary, only 56% of the patients with a stable disease course would be correctly classified, whereas at the 90% CI mark, 77% of the patients with “no change in disease” would be correctly identified as having a stable disease course. However, the 90% CI mark for defining the MCID would have greatly underestimated the frequency of patients in whom change truly had occurred.

Of note, detection rates using the 63% CI thresholds (one-SEM criterion) were again small and quite similar to those using the mean change scores as shown in Table 2.

MCID defined by optimal cutoffs as per the discrimination and classification analysis.

The discrimination and classification analysis provides cutoff values of disease change scores that best discriminate the 3 groups of patients (worsening, no change, and improvement). Such cutoff values could be considered as alternative MCID thresholds. The MCID cutoff values (detection rates) for physician-rated worsening and improvement, respectively, were at +0.6 (65%) and −0.5 (71%) for the VASMD, +0.6 (64%) and −0.5 (49%) for the ECLAM, +1.2 (48%) and −0.9 (58%) for the SLEDAI, +0.9 (57%) and −0.9 (57%) for the SLAM, +2.6 (55%) and −1.3 (61%) for the Liang et al BILAG Index, +0.7 (56%) and −0.6 (60%) for the Gladman et al BILAG Index, and +1.6 (60%) and −0.8 (52%) for the Stoll et al BILAG Index. Irrespective of the measure of disease activity considered, none of the MCID cutoffs determined by this statistical approach correctly classified >64% of all of the episodes of the 3 disease courses. Figure 2A depicts the representative results of these analyses for the VASMD and the SLEDAI.

thumbnail image

Figure 2. Discrimination analysis for determining the minimal clinically important difference (MCID) of disease activity measures. A, The predicted probabilities of disease course (y-axis) based on the changes of scores of the physician global assessment of disease activity on a visual analog scale (VASMD) and the Systemic Lupus Erythematosus Disease Activity Index (SLEDAI) based on classification and discrimination analysis are shown on the x-axis. The cutoff values for the VASMD at −0.6 and +0.5 were best suited to discriminate between groups of patients with improved, stable, and worsening childhood-onset systemic lupus erythematosus (SLE) as rated by the managing pediatric rheumatologists. For the SLEDAI, the respective cutoff values were at −1.2 and +0.75. Broken vertical lines represent these thresholds under the discrimination methods. Additionally, the 70% predicted probability of improvement (or worsening) is marked (solid horizontal line) to provide an example of how the values shown in Table 4 were determined. B, The predicted probabilities of improved, worsening, or stable well-being to provide the family's perspective on the change of childhood-onset SLE over time. The cutoff values for the VASMD at −0.15 and +0.2 were best suited to discriminate between groups of patients with improved, stable, and worsening patient well-being as rated by the caregiver. For the SLEDAI, the respective cutoff values were at −0.35 and +0.5. Broken vertical lines show thresholds under the discrimination methods. These plots support the facts that 1) using a 70% predicted probability for defining the MCID for worsening or improvement of childhood-onset SLE will misclassify a larger proportion of patients with stable disease as having worsening or improvement of childhood-onset SLE as compared with the cutoff values suggested by the classification and discrimination analysis, and 2) neither the VASMD nor the SLEDAI were well suited to reflect the family's perspective of clinically relevant change of the patient.

Download figure to PowerPoint

Table 4. Minimal clinically important differences considering certain predicted probabilities for worsening or improvement*
 70%80%90%
  • *

    Values are the predicted changes of the disease activity scores based on discrimination or cluster analysis. Rounding to the next integer was done (one decimal for VASMD). See Table 1 for abbreviations.

Physician-rated improvement   
 VASMD−2.1−2.9−4.2
 ECLAM−3−4−5
 SLEDAI−6−8−12
 SLAM−3−5−7
 Liang et al BILAG Index−14−19−28
 Gladman et al BILAG Index−5−7−10
 Stoll et al BILAG Index−8−11−14
Physician-rated worsening   
 VASMD+1.6+2.1+2.8
 ECLAM+3+4+6
 SLEDAI+7+10+14
 SLAM+3+5+7
 Liang et al BILAG Index+13+19+27
 Gladman et al BILAG Index+5+6+9
 Stoll et al BILAG Index+7+10+15

Of note, MCID thresholds defined by discrimination and classification analysis were somewhat smaller but again similar to those defined by the one-SEM criterion and comparable with those using the mean change scores for defining the MCID.

MCID at the 70% predicted probability of worsening or improvement.

It has been suggested (26) that clinically relevant change in disease activity indices may be defined based on a certain desired probability to correctly detect patients with change of disease. The results of such analyses for predicted probabilities of 70–90% for improvement and worsening of childhood-onset SLE, as would be predicted by discrimination analysis and the physician-rated change in childhood-onset SLE as an external standard, are summarized in Table 4 and Figure 2.

Clinically relevant changes in disease indices defined by such high predicted probabilities were much larger than the MCID defined by any of the other approaches to estimating the MCID.

MCID of disease activity scores: family's perspective.

When considering the ratings of the families (external standard: parent's rating of change of the patient's well-being), the mean change scores of the disease measures were often even smaller than when physician ratings of change in childhood-onset SLE were used as an external standard.

MCIDs defined by the one-SEM criterion for the 63% CI were as follows: VASMD at ±0.7, ECLAM at ±0.9, SLEDAI at ±1.9, SLAM at ±3.8, Liang et al BILAG Index at ±4.2, Gladman et al BILAG Index at ±1.4, and Stoll et al BILAG Index at ±3.3.

Using discrimination and classification analysis, the MCID cutoffs (detection rates) that discriminated best between the groups of patients with different disease courses were as follows for worsening and improvement, respectively: +0.2 (42%) and −0.1 (37%) for the VASMD, +0.1 (55%) and −0.2 (44%) for the ECLAM, +0.4 (40%) and −0.2 (42%) for the SLEDAI, +0.2 (16%) and −0.9 (63%) for the SLAM, +1.1 (53%) and −0.6 (44%) for the Liang et al BILAG Index, +0.2 (58%) and −0.4 (44%) for the Gladman et al BILAG Index, and +0.6 (55%) and −0.3 (45%) for the Stoll et al BILAG Index. This is shown for the VASMD or the SLEDAI in Figure 2B. Families often rated the patients' well-being as unchanged even when large changes in the VASMD and the SLEDAI had occurred. Regardless of the activity measure considered, none of the MCID cutoffs using this statistical approach were able to correctly classify >47% of all of the episodes of the 3 disease courses.

For reaching a 70% predicted probability of clinically important worsening of well-being to have occurred, the VASMD had to have increased by 5.2, the ECLAM by 6, the SLEDAI by 13, the SLAM by 4, the Liang et al BILAG Index by 20, the Gladman et al BILAG Index by 6, and the Stoll et al BILAG Index by 12.

Similarly, for achieving 70% predicted probabilities of patients whose well-being importantly improved, the respective MCID thresholds were at −8.2 for the VASMD, at −9 for the ECLAM, at −18 for the SLEDAI, at −5 for the SLAM, at −32 for the Liang et al BILAG Index, at −10 for the Gladman et al BILAG Index, and at −21 for the Stoll et al BILAG Index.

Accuracy of the RIFLE for classifying disease courses.

The RIFLE correctly identified 26% and 8% of the episodes of disease worsening and disease improvement, respectively. The kappa coefficient ± standard error of the RIFLE was only 0.06 ± 0.02. Alternative criteria for defining improvement or worsening (instead of clinically relevant worsening: “worsening” of at least 3 RIFLE items; clinically important improvement: “partial response” and/or “resolution” occurring in 4 or more RIFLE items) did not improve the accuracy of the RIFLE for capturing childhood-onset SLE disease courses (data not shown).

DISCUSSION

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. AUTHOR CONTRIBUTIONS
  8. Acknowledgements
  9. REFERENCES

The MCIDs, i.e., the smallest changes of measures that have clinical relevance for disease activity indices in childhood-onset SLE, are much sought after but difficult to ascertain. Differences smaller than the MCIDs are regarded as clinically irrelevant, irrespective of whether the difference is statistically significant or not (23). In the world of statistics, a significant difference is simply a difference that is unlikely to have occurred by chance and has a mathematical basis for such a claim. In the realm of health care, a difference may be statistically significant based on a simple numerical value, yet may at the same time be of little or no importance to the health or quality of life of the patients.

There is not a single generally accepted mathematical approach to calculating the MCID. We present various alternative strategies to determining the MCIDs for disease activity measures using previously proposed statistical approaches. Irrespective of the global disease activity index considered, the methodologic approach chosen, and statistical significance, the MCIDs of the disease activity indices in childhood-onset SLE were often very small, confirming previous observations in adults with SLE (29, 30). This means that in childhood-onset SLE, even small changes in disease activity may be clinically relevant. Such small MCIDs of disease activity measures are problematic because they bear the risk of erroneously classifying a patient as having improved or worsened by an important amount when, in fact, no such clinically relevant change has occurred. Because groups of patients with clinically relevant differences in childhood-onset SLE disease courses cannot be well separated using the MCID thresholds, changes in disease activity indices alone appear unlikely to suffice for correctly approximating childhood-onset SLE courses. Therefore, similar to SLE in adults, clinically relevant improvement and worsening of childhood-onset SLE can unlikely be defined based on changes in the scores of the tested disease activity indices alone (31, 32).

More generous MCID thresholds set at the level of 70% predicted probability for detecting patients with clinically relevant change in childhood-onset SLE (improved or worse) were similar to those proposed for adults with SLE (26).

Increasing the MCID thresholds of these high predicted probability rates occurs at the expense of sensitivity, i.e., will result in a large number of patients whose disease has truly worsened or improved to be rated as having experienced no change in their disease course. This again supports the notion that clinically relevant changes in the course of childhood-onset SLE may not be adequately captured solely by the disease activity indices assessed in this study.

When considering the families' perspectives of the course of childhood-onset SLE, the above is also true. Using the MCIDs as cutoff values, disease activity indices appeared even less suited to discriminate patients whose well-being had improved from those where it had remained unchanged or even worsened. Of note, this was true irrespective as to whether the disease index under consideration included items that account for patient symptoms (instead of only objectively measurable childhood-onset SLE signs).

We do not think that errors in completing the disease activity tools were the basis for the small MCID values determined in this study because all of the participating investigators were repeatedly trained to complete the disease activity indices.

It is noteworthy that the MCID thresholds based on the mean change scores, the one-SEM criterion, and the discriminant analysis were all quite similar, supporting the validity of our findings. Moreover, additional data (laboratory values, physical examination, and patient symptom reports) were collected to allow for data-driven confirmation of the disease activity scores provided, and our findings were in line with those seen in adults with SLE (26, 30).

Based on our study, the RIFLE appears to be less useful for the assessment of childhood-onset SLE than for SLE. Additional studies will be necessary to explore in more depth why the RIFLE did not perform as well in children, as would have been expected based on previous studies in adults with SLE (33).

This study has to be interpreted in light of certain limitations. As has been done by others (26, 30), we used the physician's and the parent's, rather than the patient's, assessment of change in childhood-onset SLE as the criterion standard for defining the MCID. Because of the complexity of the underlying construct, expert opinion may differ widely as to whether important improvement or worsening of childhood-onset SLE has occurred or not (33). We did not consider alternative criterion standards such as change in therapy or prednisone doses because medication change reflects the physician's perception of the patient's change in disease activity. Prior research suggests that pediatric rheumatologists, similar to adult rheumatologists, differ widely in their treatments of childhood-onset SLE and SLE (34, 35). Moreover, the inclusion of episodes of major changes in disease, rather than only of minor improvement or worsening of childhood-onset SLE, might have led to an overestimation of the MCID, which would not have changed the conclusions of this study.

In this study, using various statistical approaches, we found the MCIDs of the SLEDAI, SLAM, ECLAM, and BILAG Index for clinically important improvement or worsening of childhood-onset SLE to be small, suggesting that even small changes in their scores can have clinical relevance. Based on the commonly used SEM approach to estimating MCIDs, increases or decreases of the ECLAM score by 1, the SLEDAI, Stoll et al BILAG Index, or Gladman et al BILAG Index by 2, the SLAM by 4, or the Liang et al BILAG Index by 5 can be clinically significant. More generous MCID thresholds based on predicted probabilities using discriminant and classification analysis bear the risk of both underestimating the response to therapy and the occurrence of flares in children with childhood-onset SLE.

AUTHOR CONTRIBUTIONS

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. AUTHOR CONTRIBUTIONS
  8. Acknowledgements
  9. REFERENCES

All authors were involved in drafting the article or revising it critically for important intellectual content, and all authors approved the final version to be submitted for publication. Dr. Brunner had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study conception and design. Brunner, Ying.

Acquisition of data. Brunner, Higgins, Klein-Gitelman, Lapidus, Olson, Onel, Punaro.

Analysis and interpretation of data. Brunner, Ying, Giannini.

Acknowledgements

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. AUTHOR CONTRIBUTIONS
  8. Acknowledgements
  9. REFERENCES

We acknowledge the following investigators for data collection: Drs. Robert Colbert, T. Brent Graham, Murray Passo, Thomas Griffin, Alexi Grom, and Daniel Lovell (Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio); Dr. Robert Rennebohm (Nationwide Children's Hospital, Columbus, Ohio); Dr. Linda Wagner-Weiner (University of Chicago Comer Children's Hospital, Chicago, Illinois); Shirley Henry, PNP (Texas Scottish Rite Hospital, Dallas, Texas); and Drs. James Nocton and Calvin Williams and Elizabeth Roth-Wojicki, PNP (Medical College of Wisconsin and Children's Research Institute, Milwaukee, Wisconsin). We also acknowledge the following people: Shannen Nelson (study coordinating), Jamie Meyers-Eaton (site coordinator), Lukasz Itert (database management), Kristina Wiers (data collection), and Cincinnati Children's Hospital Medical Center Biomedical Informatics (Web-based data management application development; Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio); Becky Puplava (site coordinator; University of Chicago Comer Children's Hospital, Chicago, Illinois); Dina Blair (site coordinator; Children's Memorial Hospital, Chicago, Illinois); Marsha Malloy (data collection and site coordinator), Jeremy Zimmermann, Joshua Kapfhamer, and Noshaba Khan (data collection; Medical College of Wisconsin and Children's Research Institute, Milwaukee, Wisconsin); and Drs. AnneMarie Brescia and Carlos Rosé (Alfred I. DuPont Hospital for Children, Wilmington, Delaware).

REFERENCES

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. AUTHOR CONTRIBUTIONS
  8. Acknowledgements
  9. REFERENCES
  • 1
    Klein-Gitelman M, Reiff A, Silverman ED. Systemic lupus erythematosus in childhood. Rheum Dis Clin North Am 2002; 28: 56177.
  • 2
    Hochberg MC, for the Diagnostic and Therapeutic Criteria Committee of the American College of Rheumatology. Updating the American College of Rheumatology revised criteria for the classification of systemic lupus erythematosus [letter]. Arthritis Rheum 1997; 40: 1725.
  • 3
    Brunner HI, Gladman DD, Ibanez D, Urowitz MD, Silverman ED. Difference in disease features between childhood-onset and adult-onset systemic lupus erythematosus. Arthritis Rheum 2008; 58: 55662.
  • 4
    Hiraki LT, Benseler SM, Tyrrell PN, Hebert D, Harvey E, Silverman ED. Clinical and laboratory characteristics and long-term outcome of pediatric systemic lupus erythematosus: a longitudinal study. J Pediatr 2008; 152: 5506.
  • 5
    Kasitanon N, Magder LS, Petri M. Predictors of survival in systemic lupus erythematosus. Medicine (Baltimore) 2006; 85: 14756.
  • 6
    Brunner HI, Freedman M, Silverman ED. Close relationship between systemic lupus erythematosus and thrombotic thrombocytopenic purpura in childhood. Arthritis Rheum 1999; 42: 234655.
  • 7
    Brunner HI, Silverman ED, To T, Bombardier C, Feldman BM. Risk factors for damage in childhood-onset systemic lupus erythematosus: cumulative disease activity and medication use predict disease damage. Arthritis Rheum 2002; 46: 43644.
  • 8
    Brunner HI, Silverman ED, Bombardier C, Feldman BM. European Consensus Lupus Activity Measurement is sensitive to change in disease activity in childhood-onset systemic lupus erythematosus. Arthritis Rheum 2003; 49: 33541.
  • 9
    Gladman DD, Ibanez D, Urowitz MB. Systemic Lupus Erythematosus Disease Activity Index 2000. J Rheumatol 2002; 29: 28891.
  • 10
    Liang MH, Socher SA, Larson MG, Schur PH. Reliability and validity of six systems for the clinical assessment of disease activity in systemic lupus erythematosus. Arthritis Rheum 1989; 32: 110718.
  • 11
    Bencivelli W, Vitali C, Isenberg DA, Smolen JS, Snaith ML, Sciuto M, et al, and the European Consensus Study Group for Disease Activity in SLE. Disease activity in systemic lupus erythematosus: report of the Consensus Study Group of the European Workshop for Rheumatology Research. III. Development of a computerised clinical chart and its application to the comparison of different indices of disease activity. Clin Exp Rheumatol 1992; 10: 54954.
  • 12
    Hay EM, Bacon PA, Gordon C, Isenberg DA, Maddison P, Snaith ML, et al. The BILAG Index: a reliable and valid instrument for measuring clinical disease activity in systemic lupus erythematosus. Q J Med 1993; 86: 44758.
  • 13
    Petri M, Kasitanon N, Singh S, Link K, Magder L, Bae SC, et al, for the Systemic Lupus International Collaborating Clinics. Systemic Lupus International Collaborating Clinics renal activity/response exercise: comparison of agreement in rating renal response. Arthritis Rheum 2008; 58: 178995.
  • 14
    Jaeschke R, Singer J, Guyatt GH. Measurement of health status: ascertaining the minimal clinically important difference. Control Clin Trials 1989; 10: 40715.
  • 15
    Wells G, Anderson J, Beaton D, Bellamy N, Boers M, Bombardier C, et al. Minimal clinically important difference module: summary, recommendations, and research agenda. J Rheumatol 2001; 28: 4524.
  • 16
    Beaton DE, Bombardier C, Katz JN, Wright JG, Wells G, Boers M, et al, for the OMERACT MCID Working Group. Looking for important change/differences in studies of responsiveness. J Rheumatol 2001; 28: 4005.
  • 17
    Copay AG, Subach BR, Glassman SD, Polly DW Jr, Schuler TC. Understanding the minimum clinically important difference: a review of concepts and methods. Spine J 2007; 7: 5416.
  • 18
    Brunner HI, Feldman BM, Bombardier C, Silverman ED. Sensitivity of the Systemic Lupus Erythematosus Disease Activity Index, British Isles Lupus Assessment Group Index, and Systemic Lupus Activity Measure in the evaluation of clinical change in childhood-onset systemic lupus erythematosus. Arthritis Rheum 1999; 42: 135460.
  • 19
    Gladman DD, Goldsmith CH, Urowitz MB, Bacon P, Bombardier C, Isenberg D, et al. Sensitivity to change of 3 systemic lupus erythematosus disease activity indices: international validation. J Rheumatol 1994; 21: 146871.
  • 20
    Stoll T, Stucki G, Malik J, Pyke S, Isenberg DA. Further validation of the BILAG disease activity index in patients with systemic lupus erythematosus. Ann Rheum Dis 1996; 55: 75660.
  • 21
    Burt RK, Traynor A, Statkute L, Barr WG, Rosa R, Schroeder J, et al. Nonmyeloablative hematopoietic stem cell transplantation for systemic lupus erythematosus. JAMA 2006; 295: 52735.
  • 22
    Beaton DE, Boers M, Wells GA. Many faces of the minimal clinically important difference (MCID): a literature review and directions for future research. Curr Opin Rheumatol 2002; 14: 10914.
  • 23
    Hays RD, Woolley JM. The concept of clinically meaningful difference in health-related quality-of-life research: how meaningful is it? Pharmacoeconomics 2000; 18: 41923.
  • 24
    Wyrwich KW, Tierney WM, Wolinsky FD. Further evidence supporting an SEM-based criterion for identifying meaningful intra-individual changes in health-related quality of life. J Clin Epidemiol 1999; 52: 86173.
  • 25
    Wyrwich KW, Nienaber NA, Tierney WM, Wolinsky FD. Linking clinical relevance and statistical significance in evaluating intra-individual changes in health-related quality of life. Med Care 1999; 37: 46978.
  • 26
    American College of Rheumatology Ad Hoc Committee on Systemic Lupus Erythematosus Response Criteria. The American College of Rheumatology response criteria for systemic lupus erythematosus clinical trials: measures of overall disease activity. Arthritis Rheum 2004; 50: 341826.
  • 27
    Anderson TW. An introduction to multivariate statistical analysis. 3rd ed. Hoboken (NJ): Wiley-Interscience; 2003.
  • 28
    Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull 1979; 86: 4208.
  • 29
    Wollaston SJ, Farewell VT, Isenberg DA, Gordon C, Merril JT, Petri MA, et al. Defining response in systemic lupus erythematosus: a study by the Systemic Lupus International Collaborating Clinics Group. J Rheumatol 2004; 31: 21904.
  • 30
    Petri M. Disease activity assessment in SLE: do we have the right instruments? Ann Rheum Dis 2007; 66 Suppl 3: iii614.
  • 31
    Ruperto N, Ravelli A, Oliveira S, Alessio M, Mihaylova D, Pasic S, et al, for the Pediatric Rheumatology International Trials Organization (PRINTO) and the Pediatric Rheumatology Collaborative Study Group (PRCSG). The Pediatric Rheumatology International Trials Organization/American College of Rheumatology provisional criteria for the evaluation of response to therapy in juvenile systemic lupus erythematosus: prospective validation of the definition of improvement. Arthritis Rheum 2006; 55: 35563.
  • 32
    Strand V. Clinical trial design in systemic lupus erythematosus: lessons learned and future directions. Lupus 2004; 13: 40611.
  • 33
    Renal Disease Subcommittee of the American College of Rheumatology Ad Hoc Committee on Systemic Lupus Erythematosus Response Criteria. The American College of Rheumatology criteria for proliferative and membranous renal disease in systemic lupus erythematosus clinical trials. Arthritis Rheum 2006; 54: 42132.
  • 34
    Brunner HI, Klein-Gitelman MS, Ying J, Tucker LB, Silverman ED. Corticosteroid use in childhood-onset systemic lupus erythematosus-practice patterns at four pediatric rheumatology centers. Clin Exp Rheumatol 2009; 27: 15562.
  • 35
    Abrahamowicz M, Fortin PR, du Berger R, Nayak V, Neville C, Liang MH. The relationship between disease activity and expert physician's decision to start major treatment in active systemic lupus erythematosus: a decision aid for development of entry criteria for clinical trials. J Rheumatol 1998; 25: 27784.