SEARCH

SEARCH BY CITATION

Abstract

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. AUTHOR CONTRIBUTIONS
  8. REFERENCES
  9. Supporting Information

Objective

To evaluate data reporting related to external validity from randomized controlled trials (RCTs) assessing pharmacologic and nonpharmacologic treatment for hip and knee osteoarthritis (OA).

Methods

All RCTs assessing pharmacologic treatments and nonpharmacologic treatments for hip and knee OA indexed between January 2002 and December 2006 were selected. A sample of 120 articles were randomly selected: 30 each assessing pharmacologic treatments, surgery or technical interventions, rehabilitation, and nonimplantable devices.

Results

The country was clearly reported in 25 (21%) reports, the setting described in 40 (33%) reports, and the number of centers in 54 (45%). Details about the centers (volume of care) were given in 24 (20%) reports. Rates were lower for surgical trials for the country (3%), the setting (3%), the number of centers (13%), and details about the centers (7%). The intervention was adequately described in all pharmacologic reports and in >80% of rehabilitation reports. The technical procedure was given in all surgical intervention trial reports, but the type of anesthesia was reported in 4 (13%), preoperative care in 2 (7%), and postoperative care in 15 (50%). The device was described in 93% of device trial reports, but the manufacturer was reported in only 33%.

Conclusion

There is low reporting of data related to external validity in reports of RCTs assessing pharmacologic and nonpharmacologic treatments for hip and knee OA.


INTRODUCTION

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. AUTHOR CONTRIBUTIONS
  8. REFERENCES
  9. Supporting Information

Well-conducted randomized controlled trials (RCTs) are adopted as the gold standard for evaluating medical interventions (1–4). For results to be clinically useful, RCTs must take into account the internal validity (i.e., the extent to which systematic errors or bias are avoided) and the external validity (sometimes called applicability, i.e., whether the results of a trial can be reasonably applied or generalized to a definable group of patients in a particular setting in routine practice) (5, 6).

Historically, internal validity has been considered a priority for research. Several publications have identified methods to avoid bias (7, 8). The Consolidated Standards of Reporting Trials (CONSORT) statements, endorsed by many major medical journals, improved the reporting of data related to internal validity (1, 9). Tools (10–13) have been developed mainly to evaluate internal validity in reports of trial results included in systematic reviews (14).

Funding agencies and journals have tended to be more concerned with the scientific rigor of interventions studied than with the applicability of the results. Consequently, external validity has been frequently neglected (6, 15–17). This neglect has probably contributed to the failure to translate research into clinical practice. Lack of external validity is frequently advocated as the reason why interventions found to be effective in clinical trials are underused in clinical practice (5). However, assessing the external validity of a trial to turn research into action supposes that information is adequately reported in published articles. Further, as highlighted by the extension of the CONSORT statements to nonpharmacologic treatment, assessing external validity is probably more difficult for trials assessing nonpharmacologic treatments (e.g., surgery, technical interventions, rehabilitation, psychotherapy, devices) than pharmacologic treatments (e.g., oral drugs) (18, 19).

The aim of this study was to evaluate and compare the reporting of external validity in RCTs assessing pharmacologic and nonpharmacologic treatments for hip and knee osteoarthritis (OA). We chose these conditions because they are highly prevalent and can result in disability and reduced quality of life. Further, international guidelines require the use of a combination of pharmacologic and nonpharmacologic treatments for the optimal management of patients with these conditions (20, 21).

MATERIALS AND METHODS

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. AUTHOR CONTRIBUTIONS
  8. REFERENCES
  9. Supporting Information

Search strategy and selection of reports.

We identified all English-language reports of RCTs indexed between January 2002 and December 2006 in PubMed using the search terms “osteoarthritis hip” OR “osteoarthritis knee,” with a limitation to RCTs in Medline via PubMed and to articles published in English. A similar search strategy was used in a previous study on internal validity (22).

Eligibility criteria and screening process.

We collected the electronic records in an EndNote data file (Thomson Reuters, New York, NY). One author (NA) assessed each report by screening the title and abstract to identify relevant studies. A second author (IB) checked for adequate selection of the abstracts. Articles were included if the study was identified as an RCT assessing pharmacologic or nonpharmacologic treatment for hip or knee OA in a parallel-group or crossover design. We excluded reports of cluster RCTs, nonrandomized trials, observational studies (cohort and case–control studies), extended followup trials (i.e., extended followup of patients included in an RCT beyond the last outcome assessment), nontherapeutic trials (metrologic studies, epidemiologic studies), pathophysiologic studies, letters, ancillary studies of an RCT such as a subgroup analysis, cost-effectiveness evaluation, systematic review, and/or meta-analysis. We also excluded reports of trials assessing the organization of the health care system or interventions provided to care providers. We excluded reports with these designs because we wanted to have a relatively homogeneous sample.

The selected abstracts were classified according to the category of treatment assessed: pharmacologic treatments, surgery or technical interventions (e.g., joint lavage), rehabilitation, or nonimplantable devices.

For each category of treatment, we used a computer-generated list to randomly select 30 articles and then retrieved the full-text articles. Articles not fulfilling the inclusion criteria were replaced by a random selection of articles in the corresponding category. We chose a total of 120 articles for practical reasons, mainly to provide enough articles describing each category of treatment, and enough randomly selected articles to avoid selection bias.

Data collection.

To assess external validity as well as internal validity of the selected reports, we reviewed the literature and generated a standardized data extraction form (available from the corresponding author upon request). We used items related to external validity proposed by the CONSORT statement for RCTs (1), the extension of the CONSORT statement for nonpharmacologic trials (18, 19), and Rothwell et al (5). Before data extraction, as a calibration exercise the standardized form was tested independently by 2 authors (NA, IB) on a separate set of 20 reports. A meeting followed in which the ratings were reviewed and any disagreements were resolved by consensus. One author (NA) independently completed all of the data extraction. A random sample of 20 articles was reviewed for quality assurance.

The data extraction form covered the following data: the characteristics of the selected studies, including the year of publication, journal, medical area of the study (hip OA, knee OA, or hip and knee OA), type of treatment (pharmacologic treatment, surgical intervention, rehabilitation or education, or nonimplantable device), type of control intervention (active intervention, placebo, or usual care), funding sources (public, private, both, no funding, not reported, or unclear), study design (parallel-group or crossover), and sample size.

Internal validity of the selected reports was assessed with use of specific criteria recommended by the Cochrane Collaboration and by quality tools for assessing the results of pharmacologic and nonpharmacologic trials (10, 12), including allocation sequence generation; allocation concealment; blinding of patients, care providers, and outcome assessors; and intent-to-treat (ITT) analysis.

The reporting of data related to external validity was also evaluated.

Recruitment

Data on the method of recruitment (i.e., referral from a rheumatologist or general physician, self-selection of patients through advertisement) and duration of recruitment were evaluated.

Patients

We evaluated each study's criteria for patient eligibility (as defined in a previous work [23]), inclusion (i.e., criteria governing entry or recruitment of individuals into the trial and describing the medical conditions of interest), and exclusion (all other criteria limiting the eligibility of individuals) (23). The exclusion criteria were classified as strongly justified, potentially justified, or poorly justified reasons for excluding individuals from an RCT according to the classification proposed by van Spall et al (23). Exclusion criteria were considered strongly justified if an individual or substitute decision-maker was unable to grant informed consent, if the intervention or placebo would likely be harmful, if the intervention would likely be ineffective, or if the effect of the intervention would be difficult to interpret.

Data on the number of eligible patients, the number of patients not meeting inclusion criteria, and the number of patients refusing to participate were collected. We also checked whether the article reported baseline characteristics of excluded patients, as well as essential data on baseline characteristics of randomized patients (i.e., age, sex, weight/body mass index [BMI], ethnicity, coexisting diseases or comorbidities, duration of the disease, measure of function status, level of pain, description of radiographic evidence of damage, and use of nonsteroidal antiinflammatory drugs).

Center and care provider

We collected data on the number of centers/care providers, expertise of centers/care providers, and details about the centers (name, sources, organization, and expertise). The reporting of the number of patients recruited in each center or by each care provider was recorded.

Intervention

We collected data on whether and how details on the interventions were reported. For pharmacologic treatments, we evaluated the route of administration, dose, duration, frequency of treatment, and patient compliance. For rehabilitation, we evaluated the number, timing, duration, and content of each session; mode of delivery; whether there was supervision; and patient compliance. For surgical interventions, we evaluated the type of anesthesia, preoperative care, postoperative care, description of the technical procedure, and surgeons' compliance with the planned procedure. For nonimplantable devices, we evaluated the reporting of the manufacturer, description of the devices, and patient compliance.

Abstract and discussion sections

We collected information related to external validity reported in abstracts (i.e., country where the trial took place, setting, number of centers, number of eligible patients, number of patients randomized, length of recruitment, length of followup, and data on care providers), and noted whether the external validity was discussed in the discussion section of the study as is recommended by the CONSORT statement (1).

Global assessment of external validity

Quantitative assessment of external validity reporting may offer complementary information. Although it is difficult to specify which aspect of external validity is the most important, we decided to focus on 3 important components that are probably indispensable to assessing the external validity of a trial: the participants, the description of the experimental treatment, and the context of care (centers, setting, care providers' expertise). For each component, we identified items that were considered essential to an adequate assessment of the external validity of a published trial. These items are described in Supplemental Appendix A (available in the online version of this article at http://www3.interscience.wiley.com/journal/77005015/home). The quantitative assessment of external validity was evaluated by the percentage of the selected items that were adequately reported for each component.

Statistical analysis.

Data were analyzed using SAS software, version 9.1 (SAS Institute, Cary, NC). We used descriptive statistics for continuous variables: mean, SD, median (lower quartile, upper quartile), and minimum and maximum values. Categorical variables were described with frequencies and percentages. The results were adjusted for the potential journal clustering effect as has been recommended (24). The reporting of data related to external validity, according to category of treatment, was compared by a linear mixed-effects model, with the percentage of items with external validity as the dependent variable, fixed effects for the treatment category, and journal as a random effect.

RESULTS

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. AUTHOR CONTRIBUTIONS
  8. REFERENCES
  9. Supporting Information

Articles selected.

Our electronic search identified 388 citations, of which 123 were excluded. Among the 265 included reports, we randomly chose 120 reports, 30 for each category of treatment. After obtaining and reviewing the full texts, 11 articles were replaced. The flow of articles through the study is presented in Supplemental Appendix B (available in the online version of this article at http://www3.interscience.wiley.com/journal/77005015/home).

Characteristics of the selected studies.

Characteristics of the included studies are reported in Table 1. The 120 articles were indexed in 53 journals. Among them, 13 (11%) were published in a general medical journal with a high impact factor and 107 (89%) in a general medical journal with a low impact factor or in a specialized medical journal. Most trials (n = 118 [98%]) had a parallel-group design. Three-quarters of the reports assessed knee OA (n = 90). The source of funding was described as public in 45 (38%) articles and as completely or partially private in 25 (21%). A funding source was not reported in 50 (42%) reports.

Table 1. Characteristics of selected reports of trials of pharmacologic and nonpharmacologic treatments for hip and knee OA*
 All treatment (n = 120)Pharmacologic treatment (n = 30)Nonimplantable devices (n = 30)Rehabilitation (n = 30)Surgery (n = 30)
  • *

    Values are the number (percentage) unless otherwise indicated. OA = osteoarthritis; IQR = interquartile range.

Type of journal     
 General medical journal, high impact  factor13 (11)4 (13)4 (13)4 (13)1 (3)
 Special medical journal, or general  medical journal with low impact factor107 (89)26 (87)26 (87)26 (87)29 (97)
Medical area     
 Hip OA18 (15)1 (3)1 (3)5 (17)11 (37)
 Knee OA90 (75)23 (77)27 (90)21 (70)19 (63)
 Hip and knee OA12 (10)6 (20)2 (7)4 (13)0
Funding     
 Public45 (38)6 (20)15 (50)17 (57)7 (23)
 Manufacturer18 (15)8 (27)4 (13)06 (20)
 Public and manufacturer7 (6)2 (7)02 (7)3 (10)
 No funding8 (7)2 (7)006 (20)
 Not reported42 (35)12 (40)11 (37)11 (37)8 (27)
Sample size, median (IQR)100.0 (60–216)213.5 (85–431)66 (38–128)107 (77–140)95.5 (52–180)
Control group     
 Placebo intervention43 (36)18 (60)17 (57)6 (20)2 (7)
 Active treatment63 (52)12 (40)12 (40)11 (37)28 (93)
 Usual care14 (12)01 (3)13 (43)0
Internal validity: adequate     
 Generation of allocation sequences61 (51)19 (63)11 (37)18 (60)13 (43)
 Allocation concealment49 (41)16 (53)12 (40)14 (47)7 (23)
 Blinding of patients52 (43)26 (87)16 (53)2 (7)8 (27)
 Blinding of care providers38 (32)24 (80)11 (37)2 (7)1 (3)
 Blinding of outcome assessors71 (59)25 (83)22 (73)11 (37)13 (43)
 Intent-to-treat analyses38 (32)10 (33)10 (33)12 (40)6 (20)

The median sample size (interquartile range [IQR]) was 100 (IQR 60–216) and was twice as high for reports of pharmacologic trials as for nonpharmacologic trials.

The control group was described as receiving active treatment in 63 (52%) reports, a placebo intervention in 43 (36%), and usual care in 14 (12%). Pharmacologic treatments and nonimplantable devices were mainly compared with placebo or active treatments, whereas rehabilitation interventions were mainly compared with usual care or active treatments, and surgical procedures were compared with active treatment in most reports.

The generation of allocation sequences was adequate in 51% of the reports. The treatment allocation was adequately concealed in 49 (41%) reports. Blinding was reported and was adequate for patients in 43% of reports, for care providers in 32%, and for outcome assessors in 59%. An ITT analysis was described in only one-third of the reports.

External validity.

The results for assessing external validity are reported in Tables 2 and 3 and in Figure 1.

Table 2. Selected reports of trials of pharmacologic and nonpharmacologic treatments for hip and knee osteoarthritis that described items related to external validity*
Reporting ofAll treatment (n = 120)Pharmacologic treatment (n = 30)Devices (n = 30)Rehabilitation (n = 30)Surgery (n = 30)
  • *

    Values are the number (percentage) unless otherwise indicated. NSAIDs = nonsteroidal antiinflammatory drugs.

Recruitment     
 Method of recruitment43 (36)8 (27)13 (43)18 (60)4 (13)
 Specific method to enrich patient's recruitment23 (19)18 (60)5 (17)00
 Duration of recruitment (10 patients/month)56 (47)10 (33)11 (37)15 (50)20 (67)
Patients     
 Inclusion criteria118 (98)30 (100)30 (100)30 (100)28 (93)
 Exclusion criteria106 (88)30 (100)27 (90)28 (93)21 (70)
 Rate of exclusion criteria in each article, mean ± SD     
  Strongly justified75.5 ± 23.677.7 ± 20.779.0 ± 21.366.9 ± 26.179.3 ± 25.6
  Potentially justified1.5 ± 4.71.1 ± 3.41.5 ± 4.72.8 ± 6.80.5 ± 2.3
  Poorly justified22.9 ± 23.021.3 ± 20.519.5 ± 19.430.3 ± 26.320.2 ± 25.3
 Flow diagram48 (40)18 (60)11 (37)17 (57)2 (7)
 Number of eligible patients50 (42)12 (40)14 (47)21 (70)3 (10)
 Number of patients not meeting inclusion criteria39 (33)9 (30)8 (27)19 (63)3 (10)
 Number of patients refusing participation31 (26)6 (20)8 (27)14 (47)3 (10)
 Baseline characteristics of randomized patients109 (91)28 (93)28 (93)27 (90)26 (87)
  Age108 (90)28 (93)28 (93)27 (90)25 (83)
  Sex101 (84)27 (90)24 (80)25 (83)25 (83)
  Weight/body mass index74 (62)22 (73)18 (60)17 (57)17 (57)
  Ethnicity18 (15)8 (27)3 (10)5 (17)2 (7)
  Duration of disease47 (39)13 (43)20 (67)9 (30)5 (17)
  Measure of function status55 (46)17 (57)15 (50)16 (53)7 (23)
  Level of pain47 (39)14 (47)15 (50)15 (50)3 (10)
  Description of radiographic damage27 (23)5 (17)11 (37)4 (13)7 (23)
  NSAIDs/other drugs19 (16)6 (20)6 (20)6 (20)1 (3)
  Coexisting diseases14 (12)2 (7)1 (3)9 (30)2 (7)
Setting/center/care provider     
 Location of recruitment46 (38)9 (30)17 (57)17 (57)3 (10)
 Setting of recruitment40 (33)7 (23)16 (53)16 (53)1 (3)
 Country where trial took place25 (21)8 (27)6 (20)10 (33)1 (3)
 Number of centers54 (45)19 (63)16 (53)15 (50)4 (13)
 Details about centers24 (20)3 (10)10 (33)9 (30)2 (7)
 Number of patients recruited in each center00000
 Details of care provider35 (29)2 (7)6 (20)10 (33)17 (57)
 Number of care providers33 (28)2 (7)4 (13)7 (23)20 (67)
Table 3. Selected reports of trials of pharmacologic and nonpharmacologic treatments for hip and knee osteoarthritis that described the intervention (n = 30 for each category)
ReportsNo. (%)
Pharmacologic treatment 
 Mode of administration30 (100)
 Dose30 (100)
 Duration of treatment30 (100)
 Frequency of treatment30 (100)
 Compliance of patients10 (33)
Surgery 
 Type of anesthesia4 (13)
 Preoperative care2 (7)
 Postoperative care15 (50)
 Technical procedure30 (100)
 Compliance of care providers0
Rehabilitation 
 Number of sessions29 (97)
 Timing of sessions26 (87)
 Duration of each session24 (80)
 Content of each session28 (93)
 Mode of delivery27 (90)
 Supervision or not25 (83)
 Compliance of patients15 (50)
Devices 
 Manufacturer9 (30)
 Description of the device28 (93)
 Compliance of patients9 (30)
thumbnail image

Figure 1. Median percentage (interquartile range [IQR]) of items of components of external validity in selected reports of pharmacologic treatments (PT) and nonpharmacologic treatments for hip and knee osteoarthritis. Scores are based on the percentage of items of the components that were reported for A, baseline data, B, intervention, and C, context. Boxes represent median observations (horizontal rule), with 25th and 75th percentiles of observed data (top and bottom of box). In some instances the median observation coincided with the 25th and 75th percentiles. Error bars show 10th and 90th percentiles. The cross represents the mean. For detailed lists of the items in each category, see the Materials and Methods section.

Download figure to PowerPoint

The method of recruitment was described in 43 (36%) of the reports. When described, this method relied on referral in 29 (67%) reports and self-selection in 14 (33%) (Table 2). The duration of recruitment was described in 56 (47%) reports; reporting was better in articles about rehabilitation. The median (IQR) duration of recruitment for 10 patients per month described was 0.4 (IQR 0.2–0.8) months for pharmacologic trials, 0.8 (IQR 0.3–1.9) for device trials, 1.2 (IQR 0.9–2.7) for rehabilitation trials, and 2.5 (IQR 1.1–4.4) for surgical trials.

Participant inclusion criteria were described in almost all reports (118 [98%]) and exclusion criteria in 106 (88%) reports (Table 2). Exclusion criteria focused on age in 64 (53%) reports, medical comorbidities in 79 (66%), sex in 17 (14%), medication in 57 (48%), socioeconomic status in 3 (2%), and patients participating in another trial in 6 (5%). Twenty-three percent of reports poorly justified exclusion criteria. These rates did not differ by category of treatment.

A flow diagram of participants through the trial was given in 48 (40%) reports. Data related to the number of eligible participants and the number of participants not meeting inclusion criteria or those refusing participation were reported in less than 50% of the reports, but reporting was better for rehabilitation trials. When given, the mean rates of participants not meeting inclusion criteria or refusing to participate were 22.5 (30%) and 19.2 (16%), respectively.

The baseline data of excluded participants were given in only 1 report. The baseline clinical characteristics of randomized participants were described in 109 (91%) reports. Characteristics concerned age and sex in more than 80% of reports, weight or BMI in 62%, and severity of disease (i.e., duration of the disease, pain, function, radiographic evidence of damage) in less than half. Patients' comorbidities were provided in only 12% of reports.

The interventions were described according to the CONSORT recommendations in all reports of pharmacologic trials and in most reports of rehabilitation trials, but were missing in reports of devices and surgery trials (Table 3). In the reports of medical device trials, a description of the device was given in 28 (93%) reports, but the manufacturer was stated in only 9 (30%). In the reports of surgical intervention trials, the technical procedure was given in all reports, but the type of anesthesia was reported in only 4 (13%), preoperative care in 2 (7%), and postoperative care in 15 (50%). Control treatment was described in most reports (117 [98%]). Descriptions of cointerventions were lacking in 28 (23%) reports, mainly reports of pharmacologic trials.

The setting was described in 40 (33%) reports and the number of centers in 54 (45%) (Table 2). The country where the trial took place was clearly reported in only 25 (21%). Details of centers were given in 24 (20%) reports. Other details such as center sources, organization, and expertise were never reported. The number of participants recruited in each center was never reported. Details on the care providers were given in 35 (29%) reports.

Information related to external validity was provided in the abstract of reports as follows: 5 (4%) articles described the country where the trial took place, 18 (15%) the setting, 14 (12%) the number of centers, 2 (2%) the number of eligible patients, 110 (92%) the number of patients randomized, 6 (5%) the length of recruitment, 98 (82%) the length of followup, and 2 (2%) data on care providers. External validity was discussed in the discussion section of 11 (9%) articles.

The global assessment of each component of external validity by category of treatment is highlighted in Figure 1. Reporting of essential baseline characteristics items was lower in reports of surgical trials (median [IQR] of 30% [30–40] of the essential items reported) than in those of trials of pharmacologic treatments, nonimplantable devices, and rehabilitation (median [IQR] of 50% [40–60], 50% [30–60], and 45% [30–60] of the essential items reported, respectively; P = 0.006).

The reporting of the intervention was better in reports of trials of pharmacologic treatments and rehabilitation (median 80% [IQR 80–100] and 86% [IQR 71–100] of the essential items reported, respectively) than for those of trials of nonimplantable devices and surgery (median 33% [IQR 33–67] and 40% [IQR 20–40]) of the essential items reported, respectively; P < 0.001).

The items dedicated to the context of the trial were poorly reported for trials of all treatments, especially pharmacologic treatments and surgery (median 12% [IQR 12–25] and 25% [IQR 12–25]) of the essential items reported, respectively; P = 0.016).

DISCUSSION

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. AUTHOR CONTRIBUTIONS
  8. REFERENCES
  9. Supporting Information

This study assessed the reporting of external validity in a sample of 120 RCTs assessing pharmacologic and nonpharmacologic treatments for hip or knee OA during a 5-year period. Our results highlight the lack of data related to external validity in published reports of RCTs. Methods for recruiting patients were described in one-third of the reports; 22.9% of the exclusion criteria were poorly justified; important baseline data of patients were lacking; and setting, centers, and care providers were described in less than one-third of articles. Further, the reporting of external validity differed depending on the category of treatment. Reports of trials assessing rehabilitation provided more adequate data related to recruitment, participants, setting and centers, and intervention. Reports of trials assessing surgical procedures lacked such data, even though the reporting of some items, such as the setting, the number of centers, and center volume, is particularly important in this field. In reports of pharmacologic trials and trials assessing nonimplantable devices, the reporting was of varying quality. In reports of pharmacologic trials, the reporting of the method of recruitment and of data related to centers and care providers was poor, but the reporting of the intervention was good.

To our knowledge, this is the first study that has systematically appraised the reporting of data related to external validity from trials assessing pharmacologic and nonpharmacologic treatments. Most recent efforts of researchers and editors to improve the reporting of results of RCTs, such as the CONSORT initiative, have mainly focused on internal validity (1, 9). Nevertheless, external validity is also essential and needs to be emphasized (25, 26). The results of RCTs and systematic reviews cannot be relevant to all patients and all settings. Consequently, reporting the results of RCTs should allow clinicians to judge to whom and in which context these results could reasonably be applied.

The setting, care providers, and centers have obvious implications for external validity (5, 27). In fact, the applicability of results of trials performed in secondary or tertiary settings applied to primary settings is often a concern (5). Further, differences between health care systems can affect the applicability of results, especially regarding organization of care or reimbursement for the cost of care (5). These issues are crucial in trials assessing nonpharmacologic treatments such as surgery or technical interventions. In fact, hospital and care providers' volume and outcome are related (28–33). A surgical procedure might be found to be safe and effective in an RCT performed in high-volume centers by high-volume care providers, but applying these results to low-volume centers might result in very different results (27, 34, 35). Surprisingly, the reporting of data on care providers and centers was far less than optimal in our study, especially for trials assessing surgical procedures.

The representativeness of the patients included in an RCT is also a major issue for external validity. The inclusion and exclusion criteria are among the greatest challenges in achieving representativeness of participants. Highly selective eligibility criteria can considerably reduce the applicability of the trial results. Our results highlight the lack of reporting of exclusion criteria in 12% of the trial reports and 23% of reported exclusion criteria were poorly justified. These results are consistent with those of a systematic review of RCTs published in high impact factor journals between 1994 and 2006 (23). Exclusion criteria reported in our articles concerned mainly elderly patients, those with medical comorbidities, or those treated with specific categories of treatments. The exclusion of these specific categories of participants is problematic because it limits the representativeness of the patients.

The representativeness of the participants is also problematic because those agreeing to participate in RCTs often differ from those who do not participate (36–39). Consequently, the number of eligible nonrandomized patients, as well as the number of participants who were invited to participate but declined, is important for adequate appraisal of the external validity of a trial (22). However, these data were reported in only one-third and one-quarter of our reports, respectively, which is consistent with previous results (40).

Reporting the baseline clinical characteristics of participants included in RCTs should allow clinicians and others to assess external validity by comparison with their patients. Although baseline characteristics were described in almost all of our reports, some important data were missing: weight or BMI, while essential, was given in only 62% of the selected articles. Ethnicity, comorbidities, and severity and activity of the disease (pain, function, radiographic evidence of damage), which also predict response to and influence the generalizability of treatment, were also inadequately reported (41–44).

External validity could also be affected if trials have treatment protocols that differ from usual clinical practice, or have overly stringent limitations on the use of cointerventions. To be able to adequately apply the results of the trial in clinical practice, the treatments should be described in detail to allow for adequate reproducibility. Our results highlight the lack of descriptions of nontrial treatments in two-thirds of the reports of pharmacologic trials, and the lack of descriptions of all the components of nonpharmacologic trials, especially in reports of surgery (45). Finally, despite a specific item of the CONSORT statement dedicated to external validity, very few articles considered this issue in the discussion section.

Our study has several limitations. First, we focused on the reporting of the trial, not its conduct. Consequently, these results highlight the lack of adequate reporting of external validity criteria and do not provide information on the applicability of the results of the trial. Second, the results related to the rate of poorly justified exclusion criteria might be underestimated. Some researchers have highlighted the inadequate reporting of eligibility criteria when comparing the published article with the protocol (46); among an average of 31 eligibility criteria, only 63% were described in the main trial reports. Third, we focused on RCTs assessing hip and knee OA, and these results should be confirmed in other medical areas. However, we chose this disease because it is frequent and involves a wide range of pharmacologic and nonpharmacologic treatments. Further, the authors had some expertise in rheumatology and orthopedics and could therefore adequately evaluate the context of the trials.

In conclusion, this study highlights the lack of consideration of external validity in published reports of RCTs. Much attention is paid to the internal validity of clinical trials; however, even results of well-designed clinical trials are of limited use to clinicians if they have poor external validity and are not applicable to the patients for whom the intervention is designed. Recently, the CONSORT group developed an extension of the CONSORT statements for pragmatic trials. This extension increases the focus on data related to external validity. This initiative should help improve the consideration of external validity.

AUTHOR CONTRIBUTIONS

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. AUTHOR CONTRIBUTIONS
  8. REFERENCES
  9. Supporting Information

Dr. Boutron had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study design. Ahmad, Boutron, Moher, Ravaud.

Acquisition of data. Ahmad, Pitrou.

Analysis and interpretation of data. Ahmad, Boutron, Moher, Pitrou, Roy, Ravaud.

Manuscript preparation. Ahmad, Boutron, Moher, Ravaud.

Statistical analysis. Ahmad, Roy.

REFERENCES

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. AUTHOR CONTRIBUTIONS
  8. REFERENCES
  9. Supporting Information
  • 1
    Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, et al. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med 2001; 134: 66394.
  • 2
    Schulz KF, Grimes DA. Blinding in randomised trials: hiding who got what. Lancet 2002; 359: 696700.
  • 3
    Schulz KF, Grimes DA. Allocation concealment in randomised trials: defending against deciphering. Lancet 2002; 359: 6148.
  • 4
    Schulz KF, Grimes DA. Generation of allocation sequences in randomised trials: chance, not choice. Lancet 2002; 359: 5159.
  • 5
    Rothwell PM. External validity of randomised controlled trials: “to whom do the results of this trial apply?” Lancet 2005; 365: 8293.
  • 6
    Glasgow RE, Green LW, Klesges LM, Abrams DB, Fisher EB, Goldstein MG, et al. External validity: we need to do more. Ann Behav Med 2006; 31: 1058.
  • 7
    Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 1998; 352: 60913.
  • 8
    Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995; 273: 40812.
  • 9
    Plint AC, Moher D, Morrison A, Schulz K, Altman DG, Hill C, et al. Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. Med J Aust 2006; 185: 2637.
  • 10
    Boutron I, Moher D, Tugwell P, Giraudeau B, Poiraudeau S, Nizard R, et al. A checklist to evaluate a report of a nonpharmacological trial (CLEAR NPT) was developed using consensus. J Clin Epidemiol 2005; 58: 123340.
  • 11
    Higgins JP, Altman DG. Assessing risk of bias in included studies. In: HigginsJP, GreenS, editors. Cochrane handbook for systematic reviews of interventions: version 5.0.0 (updated February 2008). The Cochrane Collaboration. URL: http://www.cochrane-handbook.org.
  • 12
    Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials 1996; 17: 112.
  • 13
    Verhagen AP, de Vet HC, de Bie RA, Kessels AG, Boers M, Bouter LM, et al. The Delphi list: a criteria list for quality assessment of randomized clinical trials for conducting systematic reviews developed by Delphi consensus. J Clin Epidemiol 1998; 51: 123541.
  • 14
    Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of reporting of meta-analyses. Lancet 1999; 354: 1896900.
  • 15
    Bath FJ, Owen VE, Bath PM. Quality of full and final publications reporting acute stroke trials: a systematic review. Stroke 1998; 29: 220310.
  • 16
    Bhandari M, Richards RR, Sprague S, Schemitsch EH. The quality of reporting of randomized trials in the Journal of Bone and Joint Surgery from 1988 through 2000. J Bone Joint Surg Am 2002; 84A: 38896.
  • 17
    Dzewaltowski DA, Estabrooks PA, Klesges LM, Bull S, Glasgow RE. Behavior change intervention research in community settings: how generalizable are the results? Health Promot Int 2004; 19: 23545.
  • 18
    Boutron I, Moher D, Altman DG, Schulz K, Ravaud P, for the CONSORT group. Methods and processes of the CONSORT group: example of an extension for trials assessing nonpharmacologic treatments. Ann Intern Med 2008; 148: W607.
  • 19
    Boutron I, Moher D, Altman DG, Schulz K, Ravaud P, for the CONSORT group. Extending the CONSORT statement to randomized trials of nonpharmacologic treatment: explanation and elaboration. Ann Intern Med 2008; 148: 295309.
  • 20
    Zhang W, Doherty M, Arden N, Bannwarth B, Bijlsma J, Gunther KP, et al. EULAR evidence based recommendations for the management of hip osteoarthritis: report of a task force of the EULAR Standing Committee for International Clinical Studies Including Therapeutics (ESCISIT). Ann Rheum Dis 2005; 64: 66981.
  • 21
    Jordan KM, Arden NK, Doherty M, Bannwarth B, Bijlsma JW, Dieppe P, et al. EULAR recommendations 2003: an evidence based approach to the management of knee osteoarthritis. Report of a task force of the Standing Committee for International Clinical Studies Including Therapeutic Trials (ESCISIT). Ann Rheum Dis 2003; 62: 114555.
  • 22
    Boutron I, Tubach F, Giraudeau B, Ravaud P. Methodological differences in clinical trials evaluating nonpharmacological and pharmacological treatments of hip and knee osteoarthritis. JAMA 2003; 290: 106270.
  • 23
    Van Spall HG, Toren A, Kiss A, Fowler RA. Eligibility criteria of randomized controlled trials published in high-impact general medical journals: a systematic sampling review. JAMA 2007; 297: 123340.
  • 24
    Hewitt C, Hahn S, Torgerson DJ, Watson J, Bland JM. Adequacy and reporting of allocation concealment: review of recent trials published in four general medical journals. BMJ 2005; 330: 10578.
  • 25
    Bonell C, Oakley A, Hargreaves J, Strange V, Rees R. Assessment of generalisability in trials of health interventions: suggested framework and systematic review. BMJ 2006; 333: 3469.
  • 26
    Glasgow RE, Bull SS, Gillette C, Klesges LM, Dzewaltowski DA. Behavior change intervention research in healthcare settings: a review of recent reports with emphasis on external validity. Am J Prev Med 2002; 23: 629.
  • 27
    Moore WS, Young B, Baker WH, Robertson JT, Toole JF, Vescera CL, et al, and the ACAS Investigators. Surgical results: a justification of the surgeon selection process for the ACAS trial. J Vasc Surg 1996; 23: 3238.
  • 28
    Halm EA, Lee C, Chassin MR. Is volume related to outcome in health care? A systematic review and methodologic critique of the literature. Ann Intern Med 2002; 137: 51120.
  • 29
    Hodgson DC, Zhang W, Zaslavsky AM, Fuchs CS, Wright WE, Ayanian JZ. Relation of hospital volume to colostomy rates and survival for patients with rectal cancer. J Natl Cancer Inst 2003; 95: 70816.
  • 30
    Khuri SF, Daley J, Henderson W, Hur K, Hossain M, Soybel D, et al. Relation of surgical volume to outcome in eight common operations: results from the VA National Surgical Quality Improvement Program. Ann Surg 1999; 230: 41432.
  • 31
    Lavernia CJ, Guzman JF. Relationship of surgical volume to short-term mortality, morbidity, and hospital charges in arthroplasty. J Arthroplasty 1995; 10: 13340.
  • 32
    McGrath PD, Wennberg DE, Dickens JD Jr, Siewers AE, Lucas FL, Malenka DJ, et al. Relation between operator and hospital volume and outcomes following percutaneous coronary interventions in the era of the coronary stent. JAMA 2000; 284: 313944.
  • 33
    Urbach DR, Baxter NN. Does it matter what a hospital is “high volume” for? Specificity of hospital volume-outcome associations for surgical procedures: analysis of administrative data. Qual Saf Health Care 2004; 13: 37983.
  • 34
    Executive committee for the Asymptomatic Carotid Atherosclerosis Study. Endarterectomy for asymptomatic carotid artery stenosis. JAMA 1995; 273: 14218.
  • 35
    Bond R, Rerkasem K, Rothwell PM. Routine or selective carotid artery shunting for carotid endarterectomy (and different methods of monitoring in selective shunting) [review]. Stroke 2003; 34: 8245.
  • 36
    Steg PG, Lopez-Sendon J, Lopez de Sa E, Goodman SG, Gore JM, Anderson FA Jr, et al. External validity of clinical trials in acute myocardial infarction. Arch Intern Med 2007; 167: 6873.
  • 37
    Fortin M, Dionne J, Pinho G, Gignac J, Almirall J, Lapointe L. Randomized controlled trials: do they have external validity for patients with multiple comorbidities? Ann Fam Med 2006; 4: 1048.
  • 38
    Coca SG, Krumholz HM, Garg AX, Parikh CR. Underrepresentation of renal disease in randomized controlled trials of cardiovascular disease. JAMA 2006; 296: 137784.
  • 39
    Petersen MK, Andersen KV, Andersen NT, Soballe K. “To whom do the results of this trial apply?” External validity of a randomized controlled trial involving 130 patients scheduled for primary total hip replacement. Acta Orthop 2007; 78: 128.
  • 40
    Gross CP, Mallory R, Heiat A, Krumholz HM. Reporting the recruitment process in clinical trials: who are these patients and how did they get there? Ann Intern Med 2002; 137: 106.
  • 41
    Ettinger WH, Davis MA, Neuhaus JM, Mallon KP. Long-term physical functioning in persons with knee osteoarthritis from NHANES I: effects of comorbid medical conditions. J Clin Epidemiol 1994; 47: 80915.
  • 42
    Imamura K, Black N. Does comorbidity affect the outcome of surgery? Total hip replacement in the UK and Japan. Int J Qual Health Care 1998; 10: 11323.
  • 43
    Kadam UT, Jordan K, Croft PR. Clinical comorbidity was specific to disease pathology, psychologic distress, and somatic symptom amplification. J Clin Epidemiol 2005; 58: 90917.
  • 44
    Kadam UT, Croft PR. Clinical comorbidity in osteoarthritis: association with physical function in older patients in familly practice. J Rheumatol 2007; 34: 1899904.
  • 45
    Jacquier I, Boutron I, Moher D, Roy C, Ravaud P. The reporting of randomized clinical trials using a surgical intervention is in need of immediate improvement: a systematic review. Ann Surg 2006; 244: 67783.
  • 46
    Shapiro SH, Weijer C, Freedman B. Reporting the study populations of clinical trials: clear transmission or static on the line? J Clin Epidemiol 2000; 53: 9739.

Supporting Information

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. AUTHOR CONTRIBUTIONS
  8. REFERENCES
  9. Supporting Information

Additional Supporting Information may be found in the online version of this article.

FilenameFormatSizeDescription
ART_24279_sm_AppendixAB.doc65KAppendix A. External validity components; Appendix B. Flow diagram of the selected articles

Please note: Wiley Blackwell is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.