Drs. van der Helm-van Mil and Huizinga are named as investigators on the patent “Systems and methods for predicting an individual's risk of developing rheumatoid arthritis.”
A prediction rule for disease outcome in patients with Recent-onset undifferentiated arthritis: How to guide individual treatment decisions
Article first published online: 30 JAN 2007
Copyright © 2007 by the American College of Rheumatology
Arthritis & Rheumatism
Volume 56, Issue 2, pages 433–440, February 2007
How to Cite
van der Helm-vanMil, A. H. M., le Cessie, S., van Dongen, H., Breedveld, F. C., Toes, R. E. M. and Huizinga, T. W. J. (2007), A prediction rule for disease outcome in patients with Recent-onset undifferentiated arthritis: How to guide individual treatment decisions. Arthritis & Rheumatism, 56: 433–440. doi: 10.1002/art.22380
- Issue published online: 30 JAN 2007
- Article first published online: 30 JAN 2007
- Manuscript Accepted: 30 OCT 2006
- Manuscript Received: 18 MAY 2006
In patients with undifferentiated arthritis (UA), methotrexate is effective for inhibiting symptoms, structural damage, and progression to rheumatoid arthritis (RA). However, 40–50% of patients with UA experience spontaneous remission. Thus, adequate decision-making regarding treatment of patients with early UA requires identification of those patients in whom RA will develop.
A prediction rule was developed using data from the Leiden Early Arthritis Clinic, an inception cohort of patients with recent-onset arthritis (n = 1,700). The patients who presented with UA were selected (n = 570), and progression to RA or another diagnosis in this group was monitored for 1 year of followup. The clinical characteristics with independent predictive value for the development of RA were selected using logistic regression analysis. The diagnostic performance of the prediction rule was evaluated using the area under the curve (AUC). Cross-validation controlled for overfitting of the data (internal validation). An independent cohort of patients with UA was used for external validation.
The prediction rule consisted of 9 clinical variables: sex, age, localization of symptoms, morning stiffness, the tender joint count, the swollen joint count, the C-reactive protein level, rheumatoid factor positivity, and the presence of anti–cyclic citrullinated peptide antibodies. Each prediction score varied from 0 to 14 and corresponded to the percent chance of RA developing. For several cutoff values, the positive and negative predictive values were determined. The AUC values for the prediction rule, the prediction model after cross-validation, and the external validation cohort were 0.89, 0.87, and 0.97, respectively.
In patients who present with UA, the risk of developing RA can be predicted, thereby allowing individualized decisions regarding the initiation of treatment with disease-modifying antirheumatic drugs in such patients.
Making individualized decisions regarding treatment is one of the most important challenges in medicine. To this end, several studies have associated clinical variables or gene expression profiles with disease outcome, thereby providing help for clinicians making treatment decisions in several diseases, e.g., breast cancer, Hodgkin's disease, and lymphoma (1–4). For the past decennium, treatment of rheumatoid arthritis (RA) has been characterized by early, aggressive treatment with disease-modifying antirheumatic drugs (DMARDs), because this treatment strategy prevents joint damage and functional disability (5–7).
In rheumatology practices, the majority of patients who present with recent-onset arthritis have undifferentiated arthritis (UA), which is a form of arthritis that does not fulfill the classification criteria for a more definitive diagnosis. Based on results from several inception cohort studies, it is known that ∼40–50% of patients with UA experience spontaneous remission, whereas RA develops in one-third of patients with UA (8–10). Recent evidence indicates that treatment with methotrexate in patients with early UA hampers progression to RA and progression of joint damage (11), underscoring the need for guidance when starting treatment with a clinically beneficial but potentially harmful drug in UA. Ideally, only the patients with UA in whom RA develops would be treated with DMARDs, excluding those in whom UA remits spontaneously.
At present, although several risk factors for the development of RA have been identified (8, 12), a model that predicts the disease course specifically in patients with recent-onset UA is lacking. In the present study, we aimed to develop a model that predicts progression from UA to RA, using clinical variables that are easily assessed in clinical practice. The derived prediction rule was internally validated, controlling for overfitting of the data, and was subsequently externally validated in an independent cohort of patients with UA.
PATIENTS AND METHODS
The prediction rule was derived using the Leiden Early Arthritis Clinic (EAC) cohort. This inception cohort comprises more than 1,900 patients with recent-onset arthritis, of whom ∼1,700 have completed at least 1 year of followup. The EAC cohort began in 1993 at the Department of Rheumatology of the Leiden University Medical Center, the only referral center for rheumatology in a health care region of ∼400,000 inhabitants in The Netherlands (13). General practitioners were encouraged to refer patients directly when arthritis was suspected; patients were included if a physical examination revealed arthritis.
At the first visit, the rheumatologist completed a questionnaire regarding the presenting symptoms, as reported by the patient: type, localization and distribution of initial joint symptoms, symptom duration, and course of the initial symptoms. The patient's smoking history and family history were assessed. Patients rated morning stiffness on a visual analog scale (VAS; range 0–100 mm). For the present study, the severity of morning stiffness was used instead of the duration of morning stiffness, because the former variable has been proven to be a better discriminator (14, 15). The Health Assessment Questionnaire (16) was used to provide an index of disability. A 44-joint count for tender and swollen joints was performed, scoring each joint on a 0–1-point scale (17). Compression pain in the metacarpophalangeal and metatarsophalangeal joints was recorded.
Baseline blood samples were obtained for determination of the erythrocyte sedimentation rate (ESR), the C-reactive protein (CRP) level, the presence of IgM rheumatoid factor (RF), as determined by enzyme-linked immunosorbent assay (ELISA), and the presence of antibodies to cyclic citrullinated peptide 2 (anti-CCP), as determined by ELISA (Immunoscan RA Mark 2; Euro-Diagnostica, Arnhem, The Netherlands). The cutoff level for anti-CCP positivity was 25 arbitrary units. Radiographs of the hands and feet were obtained and scored according to the Sharp/van der Heijde method (18). Patients provided informed consent, and the local ethics committee approved the protocol.
Assessment of disease status after 1 year.
Two weeks after being included in the study, when results of laboratory investigations and radiography were known, 570 patients were determined to have a form of arthritis that could not be classified according to American College of Rheumatology (ACR; formerly, the American Rheumatism Association) criteria (19) and were documented as having UA. After 1 year of followup, the disease status of all patients with UA was examined to determine whether RA or another specific type of arthritis had developed, based on fulfillment of the ACR criteria. Inherent in the design of an inception cohort, the duration of followup differed within the study population, and at the moment of analysis (July 2005), the majority of patients with UA (94%) had been followed up for more than 1 year (mean ± SD followup 8 ± 3 years).
External validation cohort.
Patients included in the placebo arm of the Probaat (PROMPT) trial, a double-blind, placebo-controlled, randomized trial in which patients with recent-onset UA were treated with either methotrexate or placebo, were used for validation (n = 55) (11). Exclusion of the patients with UA who were also included in the EAC cohort resulted in 36 independent patients with UA. Two of these patients were lost to followup. For each patient, the progression score at baseline was calculated, and the development of RA after 1 year of followup was assessed (11).
Patients with UA in whom RA developed were compared with those in whom RA did not develop, using the chi-square test for nominal variables and Student's t-test for continuous variables. Symptom duration was categorized. Subsequently, all clinical variables were entered as possible explanatory variables in a logistic regression analysis, with disease outcome (RA or non-RA) at 1 year of followup as the dependent variable. Using a backward selection procedure, the most significant independent variables were identified, using a P value greater than 0.10 as the removal criterion.
In the logistic regression model, the predicted probability of RA was related to the covariates via the following prognostic index: B1 × x1 + B2 × x2 + B3 × x3 … Bk × xk. The regression coefficient (B) of the covariate indicates an estimate of the relative magnitude of the prognostic power of a specific variable. Using the prognostic index, we calculated the predicted probability of RA developing for every patient. For continuous variables (age, VAS score, tender and swollen joint counts, CRP level), the effect was studied both as a continuous variable and as a categorized variable. Categories were created using clinically applicable cutoff levels and percentiles. Categories were pooled if corresponding regression coefficients were similar. Data were missing for some patients, as follows: morning stiffness score on a 100-mm VAS (n = 160 patients), anti-CCP antibody level (n = 64 patients), disease duration (n = 22 patients), tender joint count (n = 5 patients), swollen joint count (n = 4 patients), CRP level (n = 1 patient), and presence of RF (n = 1 patient). To prevent exclusion of these patients from the logistic regression analysis, the median values for these variables were imputed.
To obtain a simplified prediction rule, the regression coefficients of the predictive variables were rounded to the nearest number ending in .5 or .0, resulting in a weighted score; subsequently, the values for the independent predictive variables were summed. The calculated prediction scores were compared with the observed percentage of patients who experienced progression to RA. The positive and negative predictive values were determined for several cutoff values of the prediction scores. To evaluate the diagnostic performance of the rule, a receiver operating characteristic (ROC) curve was constructed. The area under the ROC curve (AUC) values provided a measure of the overall discriminative ability of a model.
For internal validation, cross-validation was performed to control for overfitting (20). Cross-validation mimics the prediction situation and for each observation yields a prediction score based on the other (n − 1) observations (20). To validate the model, a ROC curve was made using the cross-validated predictions as well as the external validation cohort. SPSS version 10.0 software (SPSS, Chicago, IL) was used.
During the first year of followup, RA developed in 177 of the 570 patients with UA, other rheumatologic disease developed in 94 patients, and 150 patients achieved clinical remission, defined as discharge from the outpatient clinic because of the absence of arthritis while not receiving DMARD treatment. For further analysis, patients with another rheumatologic diagnosis or UA and those who achieved remission were assembled as the non-RA group (n = 393).
The characteristics of patients with UA in whom RA developed and those in whom RA did not develop are compared in Table 1. In the univariate analyses, all variables except smoking were significantly associated with progression to RA.
|Characteristic||No progression to RA (n = 393)||Progression to RA (n = 177)||P|
|Age, mean ± SD years||48.6 ± 17.0||56.3 ± 15.3||<0.001|
|Female sex||208 (53)||121 (68)||0.001|
|Positive family history of RA||81 (21)||54 (31)||0.01|
|Course of starting symptoms|
|Acute for <24 hours||116 (30)||36 (20)|
|Subacute for >24 hours||123 (31)||51 (29)|
|Gradual||141 (36)||86 (49)|
|Intermittent||13 (3)||4 (2)||0.02|
|Symptom duration at inclusion|
|<6 weeks||103 (27)||18 (11)|
|6 weeks to 3 months||80 (21)||43 (25)|
|3–6 months||89 (23)||47 (28)|
|>6 months||107 (28)||61 (36)||<0.001|
|Localization of affected joints|
|Small joints||171 (44)||95 (54)|
|Large joints||165 (42)||32 (18)|
|Small and large joints||57 (15)||50 (28)||<0.001|
|Localization of affected joints|
|Symmetric||147 (37)||118 (67)||<0.001|
|Localization of affected joints|
|Upper extremities||177 (45)||71 (40)|
|Lower extremities||139 (35)||22 (12)|
|Upper and lower extremities||77 (20)||84 (47)||<0.001|
|Morning stiffness, mean ± SD score on a 100-mm VAS||35.5 ± 30.0||53.3 ± 30.1||<0.001|
|Compression pain in MCP joints||159 (40)||116 (66)||<0.001|
|Compression pain in MTP joints||134 (34)||103 (58)||<0.001|
|Number of tender joints, median (IQR)||3 (2–7)||8 (4–12)||<0.001|
|Number of swollen joints, median (IQR)||2 (1–4)||4 (2–7)||<0.001|
|C-reactive protein level, median (IQR) mg/liter||8 (3–21)||14 (7–43)||<0.001|
|ESR, median (IQR) mm/hour||17 (8–38)||32 (19–53)||<0.001|
|Rheumatoid factor positivity||56 (14)||84 (47)||<0.001|
|Anti-CCP positivity||38 (11)||83 (51)||<0.001|
|HAQ score, mean ± SD||0.7 ± 0.6||1.0 ± 0.7||<0.001|
|Smoking||187 (48)||84 (47)||1.0|
|Erosive disease||29 (7)||29 (16)||0.001|
Multivariate analyses and derivation of the prediction rule.
In the logistic regression analysis, the independent predictive variables for development of RA were age, sex, localization of joint symptoms (small/large joints, symmetric/asymmetric, upper/lower extremities), morning stiffness, tender and swollen joint counts, CRP level, and the presence of RF or anti-CCP antibodies (Table 2).
|Localization in small joints hand/feet||0.6||1.8||1.1–3.1||0.024||0.5|
|Localization in upper extremities||0.8||2.1||1.1–4.4||0.04||1|
|Localization in both upper and lower extremities||1.3||3.5||1.7–7.5||0.001||1.5|
|Morning stiffness score on 100-mm VAS|
|Number of tender joints|
|Number of swollen joints|
|CRP level, mg/liter|
Because age was more predictive as a continuous variable than as a categorized variable, age was not categorized. The other variables were categorized. For the resulting model, the fraction of explained variation (Nagelkerke's R2) was 0.57; when using a predicted probability of 0.5 as the cutoff value, the outcomes for 83% of patients were predicted correctly. The coefficients for the simplified prediction score are listed in Table 2. Figure 1 presents a form that can be used to easily calculate the prediction score. The range of the prediction scores is 0–14, with a higher score indicating a greater risk of developing RA.
A prediction score was calculated for every patient with UA. Figure 2 shows the predicted risk of RA as a function of the prediction score (obtained from a logistic regression model with score as the independent variable). Table 3 shows the observed percentage of patients who experienced progression to RA in relation to the calculated prediction score. None of the patients with UA who had a prediction score ≤3 progressed to RA during the 1-year followup period, and all of the patients with UA who had a score ≥11 did experience progression to RA. Among the patients with scores of 4–10 who experienced progression to RA, the frequency of such progression increased with rising scores.
|Prediction score||No progression to RA (n = 387)||Progression to RA (n = 175)|
|0||1 (100)||0 (0)|
|1||8 (100)||0 (0)|
|2||42 (100)||0 (0)|
|3||58 (100)||0 (0)|
|4||78 (93)||6 (7)|
|5||73 (85)||13 (15)|
|6||63 (74)||22 (26)|
|7||37 (49)||38 (51)|
|8||16 (33)||33 (67)|
|9||6 (14)||36 (86)|
|10||5 (23)||17 (77)|
|11||0 (0)||8 (100)|
|12||0 (0)||1 (100)|
|13||0 (0)||1 (100)|
|14||0 (0)||0 (0)|
Table 4 shows the percentage of patients in whom RA developed, according to several cutoff values of the prediction score. For example, when the scores 5.0 and 9.0 were chosen as cutoff values, 97% of patients with UA who had a score ≤5.0 did not develop RA, and a score ≥9.0 was associated with progression to RA in 84% of patients. When cutoff values of 6.0 and 8.0 were used, 91% of patients with UA who had a score ≤6.0 did not develop RA (negative predictive value 91%, 95% confidence interval [95% CI] 88–94%), and a score ≥8.0 corresponded to progression to RA in 84% of patients (positive predictive value 84%, 95% CI 75–91%). Using these cutoff values, 145 patients with UA (25%) had a score between 6.0 and 8.0, indicating that no adequate prediction could be made for these patients. Twenty-five patients with UA did not fulfill the 1987 ACR criteria for RA after 1 year of followup, but RA developed later in the disease course. These patients had a median prediction score of 5.7 (interquartile range [IQR] 4.8–6.2); this value is between the scores for the patients with UA in whom RA developed and those in whom RA did not develop during the first year of followup (median score 7.7 [IQR 6.6–8.8] and median score 4.6 [IQR 3.3–5.9], respectively).
|Cutoff values||No progression to RA||Progression to RA|
|Score ≤4.0||145 (99)||1 (1)|
|4.0–10.0||240 (60)||159 (40)|
|≥10.0||2 (12)||15 (88)|
|Score ≤5.0||223 (97)||8 (3)|
|5.0–9.0||157 (55)||131 (46)|
|≥9.0||7 (16)||36 (84)|
|Score ≤6.0||296 (91)||28 (9)|
|6.0–8.0||76 (52)||69 (48)|
|≥8.0||15 (16)||78 (84)|
The discriminative ability of the logistic regression model and the prediction rule were evaluated with a ROC curve (Figure 3). Both the logistic regression model and the prediction rule had a mean ± SD AUC value of 0.89 ± 0.014. The finding that the AUC values for the logistic regression model and the prediction rule were equal indicates that the derivation of the prediction rule from the logistic regression model had not introduced a loss of discriminative ability.
Cross-validation was used to control for overfitting. This procedure yielded a value for the predicted probability of RA for every patient, based on results of model-fitting on the other patients (20). The AUC value of the cross-validated predictions nearly equaled the mean ± SD AUC value of the prediction score (0.87 ± 0.015) (Figure 3), indicating that overfitting was not a major problem.
In the validation cohort, 47% of patients with UA had experienced progression to RA after 1 year of followup. The prediction scores for the patients with UA in whom RA did develop and those in whom RA did not develop are presented in Figure 4. Among patients with UA who experienced progression to RA, the median prediction score was 8.0 (IQR 6.1–9.1); among patients in whom RA did not develop, the median prediction score was 4.6 (IQR 3.5–5.5). Ninety-four percent of the patients with a prediction score ≤6.0 had not experienced progression to RA, and the development of RA was observed in 83% of patients with a score >6. All patients with a score ≥8.0 had progressed to RA, and 78% of patients with a score <8 did not develop RA. In the validation cohort, 17% of the patients with UA had a prediction score between 6 and 8; RA had not developed in two-thirds of these patients and had developed in one-third of them. If treatment decisions were based on the prediction rule using cutoff levels of ≥8 for initiating treatment and ≤6 for withholding treatment, treatment would have been withheld inaccurately from only 6% of the patients, and no patient would have received treatment inaccurately. The mean ± SD AUC value for the validation cohort was 0.97 ± 0.024.
The currently developed rule predicts the development of RA in patients with UA, using 9 clinical variables that are commonly assessed during the first visit: sex, age, localization of joint symptoms, morning stiffness, tender and swollen joint counts, the CRP level, and the presence of RF and anti-CCP antibodies. The resulting prediction score corresponds to a chance for progression to RA. The positive and negative predictive values of the prediction score depend on the chosen cutoff values. The discriminative ability was excellent, with an AUC value of 0.89 and with a value of 0.87 after internal validation correcting for overfitting. Subsequent validation in a small independent cohort revealed an AUC value of 0.97. Because the prediction rule is accurate and can be easily determined in daily clinical practice, the present model is an important step forward in achieving individualized treatment in patients with recent-onset UA.
Because current evidence regarding treatment of RA is based on results of large trials involving patients fulfilling the ACR 1987 revised criteria for RA (19), fulfillment of these criteria was used as the outcome for the current study. Alternative outcome measurements such as disease persistence or remission can be considered, but no generally accepted definitions of these disease states are available, and no trials of patients with these disease states are available to provide guidance when making treatment decisions. Nevertheless, the use of fulfillment of the ACR criteria as outcome may lead to circularity, because use of the items included in the ACR criteria is expected to result in the identification of predictive variables. However, several studies have shown that the ACR criteria themselves have low discriminative value in patients with UA (12, 21–25), and only some of the variables used for the present prediction rule are among the ACR criteria. In the end, it will most likely not make a large difference whether the outcome of a prediction rule is the diagnosis RA or disease persistence, because the ACR criteria are formulated based on patients with longstanding/persistent RA (mean disease duration 8 years), and the reported remission rate in these patients is low (10–15%) (26, 27).
Misclassification may have occurred when patients who presented with UA were treated with any drug that hampered the progression to RA. In such cases, patients who normally would have progressed to RA would be classified as non-RA. Exclusion of such misclassified patients, who supposedly would have high prediction scores because they were prone to the development of RA, will result in an increased discriminative ability of the current prediction rule.
The presence of erosions on radiographs of the hands and/or feet is reported to have high specificity (but low sensitivity) for discriminating between self-limiting and persistent disease (25). Although in univariate analysis, the presence of erosions was significantly increased in UA patients in whom RA developed compared with patients in whom RA did not develop (16% versus 7%), multivariate regression analysis revealed that the presence of erosions was not an independent prognostic variable. The presence of erosions appeared to be associated with a higher median age (64 years in patients with erosive disease versus 49 years in those with nonerosive disease), a higher median number of swollen joints (5 joints in patients with erosive disease versus 2 joints in those with nonerosive disease), and the presence of RF (46% of patients with erosive disease versus 23% of those with nonerosive disease). Because the presence of erosions was not identified as a variable with an independent predictive value, data on erosions were not included in the prediction rule.
A model for predicting self-limiting, persisting, or erosive arthritis exists (25). For the development of that model, all consecutive patients with arthritis who were referred were incorporated, including patients in whom a definite diagnosis was made during the first weeks. Decisions regarding the initiation of DMARDs are seldom problematic in such patients. At present, support is needed in making treatment decisions for patients with recent-onset UA (28), because the disease outcome in patients with UA is variable. For the present study, we selected patients with UA from a total of 1,700 consecutive patients and developed a prediction rule specifically for UA.
The positive and negative predictive values of the prediction score depend on the chosen cutoff level. When the upper and lower cutoff values were 8.0 and 6.0, the corresponding positive predictive value and negative predictive value were 84% and 91%, respectively. In the original cohort, 25% of patients had a prediction score between 6.0 and 8.0; in these patients, the chance of RA developing or not developing was equal. Apparently, the clinical characteristics of patients with intermediate scores are insufficient to predict disease outcome. It is possible that genotype data are helpful in these patients. Patients were also typed for HLA–DRB1 shared epitope (SE) alleles and PTPN22. In the multivariate analysis, the presence of SE alleles was not identified as an independent predictive variable; this might be attributable to the fact that the SE alleles are associated with the presence of anti-CCP antibodies (29), which are already included in the prediction rule. Also, the presence of the PTPN22 T allele did not result in a better ability to predict progression from UA to RA. This is understandable, because the PTPN22 T allele confers risk of both UA and RA (30). In the validation cohort, 17% of patients had a prediction score between 6.0 and 8.0; for treatment decisions in these patients the observed risk of progression to RA can be weighted against the individual risk profile for treatment toxicity.
The prediction score discriminated even better in the validation cohort than in the initial cohort: 100% of patients with a score ≥8.0 had progressed to RA, and 94% of patients with a score ≤6.0 did not develop RA. This indicates that if treatment decisions were based on the prediction rule using the cutoff levels of ≥8 for initiating treatment and ≤6 for withholding treatment, treatment would be inaccurately withheld in only 6% of patients, and no patient would receive treatment inaccurately. Although the validation cohort is relatively small and the current prediction rule should be evaluated in other early arthritis cohorts, we believe that the current model allows physicians and patients to make an evidence-based choice regarding whether or not to initiate DMARDs in the majority of patients presenting with UA.
Dr. van der Helm-van Mil had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study design. Drs. Breedveld, Toes, and Huizinga.
Acquisition of data. Drs. van der Helm-van Mil and van Dongen.
Analysis and interpretation of data. Drs. van der Helm-van Mil, le Cessie, van Dongen, and Huizinga.
Manuscript preparation. Drs. van der Helm-van Mil, le Cessie, Toes, and Huizinga.
Statistical analysis. Drs. van der Helm-van Mil and le Cessie.
- 6Clinical and radiographic outcomes of four different treatment strategies in patients with early rheumatoid arthritis (the BeSt study): a randomized, controlled trial. Arthritis Rheum 2005; 52: 3381–90., , , , , , et al.
- 11Treatment of patients with undifferentiated arthritis with methotrexate: a double-blind, placebo-controlled, randomized clinical trial to prevent evolvement into RA [abstract]. Arthritis Rheum 2005; 52: 4058., , , , , , et al.
- 17EULAR handbook of clinical assessments in rheumatoid arthritis. Alphen aan den Rijn: Van Zuiden Communications; 2000. p. 10–1., , .
- 26Sustained remission in a cohort of patients with RA: association with absence of IgM-rheumatoid factor and absence of anti-CCP antibodies. Int J Adv Rheumatol 2004; 2: 4–6., , , , .
- 29The HLA–DRB1 shared epitope alleles are primarily a risk factor for anti–cyclic citrullinated peptide antibodies and are not an independent risk factor for development of rheumatoid arthritis. Arthritis Rheum 2006; 54: 1117–21., , , , , .