To evaluate the association of adequate allocation concealment and patient blinding with estimates of treatment benefits in osteoarthritis trials.
To evaluate the association of adequate allocation concealment and patient blinding with estimates of treatment benefits in osteoarthritis trials.
We performed a meta-epidemiologic study of 16 meta-analyses with 175 trials that compared therapeutic interventions with placebo or nonintervention control in patients with hip or knee osteoarthritis. We calculated effect sizes from the differences in means of pain intensity between groups at the end of followup divided by the pooled SD and compared effect sizes between trials with and trials without adequate methodology.
Effect sizes tended to be less beneficial in 46 trials with adequate allocation concealment compared with 112 trials with inadequate or unclear concealment of allocation (difference −0.15; 95% confidence interval [95% CI] −0.31, 0.02). Selection bias associated with inadequate or unclear concealment of allocation was most pronounced in meta-analyses with large estimated treatment benefits (P for interaction < 0.001), meta-analyses with high between-trial heterogeneity (P = 0.009), and meta-analyses of complementary medicine (P = 0.019). Effect sizes tended to be less beneficial in 64 trials with adequate blinding of patients compared with 58 trials without (difference −0.15; 95% CI −0.39, 0.09), but differences were less consistent and disappeared after accounting for allocation concealment. Detection bias associated with a lack of adequate patient blinding was most pronounced for nonpharmacologic interventions (P for interaction < 0.001).
Results of osteoarthritis trials may be affected by selection and detection bias. Adequate concealment of allocation and attempts to blind patients will minimize these biases.
Inadequate methodology may flaw the results of randomized osteoarthritis trials (1). Meta-epidemiologic studies examine the association of specific trial characteristics, such as concealment of allocation or patient blinding, with estimated treatment effects in a collection of meta-analyses and their component trials (2–5). In meta-analyses of these meta-epidemiologic studies, inadequate concealment of allocation and a lack of double blinding were associated with exaggerated estimates of treatment benefits (1, 6). In a combined analysis of data from 3 meta-epidemiologic studies of binary outcomes across different medical fields, we recently found that overestimations of treatment benefits were more pronounced for subjective outcomes as compared with objective outcomes such as overall mortality (4). Subjective outcomes, such as patient-reported pain intensity measured on visual analog, numeric rating, or Likert scales, are frequently used in osteoarthritis trials, whereas objective binary outcomes, such as mortality, are rarely addressed (7–9).
We performed a meta-epidemiologic study in the field of clinical osteoarthritis research to determine whether components of methodologic quality are associated with overestimates of treatment effects. We previously reported that the exclusion of randomized patients from the analysis was associated with likely overestimates of treatment benefits in osteoarthritis trials, but the extent and direction of this attrition bias resulting from the biased exclusion of patients after entry into the trial remained unpredictable in a specific situation (5). Bias may also occur at earlier stages of a trial: selection bias through the biased allocation of patients to comparison groups at trial entry if the allocation of patients is not adequately concealed, and detection and performance bias if blinding of patients is inadequate, which may result in biased assessment of self-reported outcomes, differential placebo or nocebo effects across comparison groups, and the unequal intake of analgesic cointerventions apart from the treatment under evaluation (1). Here we report on the association of estimates of treatment benefits with the adequacy of concealment of allocation and patient blinding in clinical osteoarthritis research.
We searched The Cochrane Library, Medline, EMBase, and CINAHL using a combination of keywords and text words related to osteoarthritis, which were combined with validated filters for controlled clinical trials and meta-analyses. Details of the search strategy are described elsewhere (5). The last update was performed on November 20, 2007.
Meta-analyses of randomized or quasi-randomized trials in patients with osteoarthritis of the knee or hip were eligible if they evaluated patient-reported pain in patients allocated to any intervention compared with patients allocated to placebo, a sham intervention, or a nonintervention control group. If one topic was covered by several reports, the most recent report was included. Two reviewers independently evaluated the reports for eligibility and disagreements were resolved by discussion or by involvement of a third reviewer. Reports of all component trials were obtained and no language restrictions were applied.
Two reviewers independently extracted data from individual trials regarding interventions, funding, publication year, design characteristics, study size, and results on a standardized form. The primary outcome was pain intensity. If different pain-related outcomes were reported, we referred to a previously described hierarchy of outcomes (9, 10) and extracted the outcome that was highest on this list. Global pain took precedence over pain on walking and the Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC) pain subscores, for example. If a trial report provided data on global pain scores and WOMAC pain subscores, we only recorded data on global pain scores. If more than one time point was reported, we extracted the outcome at 3 months after the end of treatment for potentially structure-modifying agents, such as chondroitin, and at 12 months after the end of treatment for behavior-changing interventions, such as education. For all other interventions, we extracted the outcome at the end of treatment. When necessary, means and measures of dispersion were approximated from figures. For crossover trials, we extracted data from the first period only (11). Disagreements were discussed with a third reviewer and subsequent consensus was reached.
Concealment of allocation was considered adequate if the investigators responsible for patient selection and inclusion were unable to know before allocation which treatment was next, e.g., central randomization; the use of sequentially numbered, sealed, and opaque assignment envelopes; or coded drug packs. Concealment of allocation of trials, which lacked a specific statement, was classified as unclear. Patient blinding was considered adequate if a placebo or sham control intervention was used and experimental and control interventions were described as indistinguishable or the use of a double-dummy technique was reported. Analyses were considered to be performed adequately according to the intent-to-treat principle if all of the randomized patients were included in the analysis (5). The definitions of different types of bias in randomized trials and measures to minimize them are provided in Table 1.
|Type of bias||Definition||Measure to minimize bias|
|Selection bias||Biased allocation to comparison groups||Concealment of allocation: Procedures that prevent personnel assigning patients to intervention groups from foreseeing allocation. Adequate if the investigators responsible for patient selection and inclusion were unable to know before allocation which treatment was next, e.g., central randomization; the use of sequentially numbered, sealed, and opaque assignment envelopes; or coded drug packs.|
|Performance bias||Unequal provision of care apart from intervention under evaluation||Blinding: Procedures that prevent therapists and patients from knowing which intervention was received. Adequate if allocated interventions were indistinguishable.|
|Detection bias||Biased assessment of outcome(s)||Blinding: Procedures that prevent outcome assessors from knowing which intervention was received. Adequate if independent, blind outcome assessors evaluated outcomes or, in the case of patient-reported outcomes, if allocated interventions were indistinguishable.|
|Attrition bias||Biased occurrence and handling of deviations from protocol and losses to followup||Procedures that prevent exclusion of randomized patients from the analyses and minimize protocol deviations and losses to followup. Adequate if all randomized patients were included in the analysis in the group they were originally allocated to, regardless of their adherence to the study protocol (intent-to-treat analysis).|
Treatment effects were expressed as effect sizes (ES), dividing the difference in mean values at the end of the trial by the pooled SD (10). A negative ES indicates a beneficial effect of the experimental intervention. If some required data were unavailable, we used approximations as previously described (10). We used standard random-effects meta-analyses to combine ES across trials and calculated the DerSimonian and Laird estimate of the variance τ2 to determine heterogeneity between trials (12, 13).
Within each meta-analysis, we estimated the ES of trials with and without adequate allocation concealment separately using a random-effects meta-analysis. For each meta-analysis, we derived the difference between pooled estimates from trials with adequate allocation concealment and trials without adequate allocation concealment. Then we combined these differences using a random-effects meta-analysis fully allowing for heterogeneity between meta-analyses (3), and measured the variability in bias estimates between meta-analyses using τ2 as a measure of heterogeneity (13). Formal tests of interaction between concealment of allocation and estimated treatment benefits were performed separately for each meta-analysis based on Z scores for the estimated difference in ES between trials with and without adequate concealment of allocation and the corresponding SE. In sensitivity analyses, we additionally stratified by patient blinding and intent-to-treat analysis to account for potential confounding by these factors. The same procedure was followed for trials with and without adequate blinding of patients. A negative difference in ES indicates that trials with adequate allocation concealment or adequate patient blinding show a less beneficial treatment effect. Then we performed stratified analyses accompanied by interaction tests based on Z scores according to the following prespecified characteristics (5): treatment benefit in overall meta-analysis (small [ES greater than −0.5] versus large [ES less than or equal to −0.5]), between-trial heterogeneity in overall meta-analysis (low [τ2 <0.06] versus high [τ2 ≥0.06]), and type of intervention assessed (pharmacologic versus nonpharmacologic interventions; conventional versus complementary medicine). The prespecified cutoff of τ2 = 0.06 corresponds to a difference between the smallest and largest ES of approximately 1 ES.
Finally, we compared pooled ES, between-trial heterogeneity, precision defined as 1/SE, and P values for pooled ES between random-effects meta-analyses including all trials and restricted to meta-analyses including trials with adequate concealment of allocation or adequate patient blinding only using Wilcoxon's rank tests for paired observations. All P values were 2-sided. All analyses were performed with Stata statistical software, version 10.1 (StataCorp, College Station, TX).
We identified 151 reports of meta-analyses of osteoarthritis trials. A total of 134 reports were excluded because they either included no continuous pain outcome (n = 43), covered duplicate topics (n = 83), or used only active control interventions (n = 8) (5). One report described 4 meta-analyses and one report described 2 meta-analyses. Therefore, 21 meta-analyses described in 17 reports were eligible (5), and 16 meta-analyses of 175 trials and 41,142 patients (10, 14–24) contributed to the analyses. Characteristics of the included meta-analyses are shown in Supplementary Appendix A (available in the online version of this article at http://www3.interscience.wiley.com/journal/77005015/home). The median treatment benefit in the 16 included meta-analyses was an ES of −0.43 (range −0.88 to −0.07) with a median between-trial heterogeneity variance of 0.04 (range 0.00–0.52). Four meta-analyses showed a large treatment effect (10, 15, 21) and 7 showed a high degree of between-trial heterogeneity (10, 15–17, 21). Seven meta-analyses addressed pharmacologic interventions (10, 14, 15, 17, 22, 24) and 9 addressed nonpharmacologic treatments (16, 18–21, 23). Nine were related to conventional interventions (14, 17–20, 22–25) and 7 were related to complementary medicine (10, 15, 16, 21).
Fourteen meta-analyses with 158 trials and 40,437 patients included both trials with and without adequate concealment of allocation and contributed to the analysis. Table 2 shows a comparison of the characteristics of these trials. Forty-six trials (29%) reported adequate allocation concealment and 111 trials (70%) were unclear about concealment of allocation. One trial (1%) was quasi-randomized using alternation, and allocation concealment was considered inadequate. Of trials with adequate concealment, 26 (56%) used coded drug packs or devices; 15 (33%) used central randomization; 4 (9%) used sequentially numbered, sealed, and opaque assignment envelopes; and 1 (2%) used an onsite computer system with allocations kept in a locked unreadable computer file that could be accessed only after the characteristics of an enrolled participant were entered into the database. Four trials that reported the use of assignment envelopes were not deemed to have adequate concealment of allocation because they did not specify that the envelopes were sequentially numbered, sealed, and opaque. Trials with adequate allocation concealment were more likely to report adequate blinding of patients (P = 0.008) and to perform intent-to-treat analyses (P = 0.07), were larger (P = 0.021), and were published more recently (P < 0.001) than trials with inadequate or unclear concealment of allocation.
|Adequate (n = 46), no. (%)||Inadequate or unclear (n = 112), no. (%)||P*|
|Adequate patient blinding||0.008|
|Yes||33 (72)||42 (37)|
|No/unclear||13 (28)||70 (63)|
|Yes||15 (33)||17 (15)|
|No/unclear||31 (67)||95 (85)|
|Number of allocated patients||0.021|
|>200||27 (59)||40 (36)|
|≤200||19 (41)||72 (64)|
|Yes||30 (65)||79 (71)|
|No||16 (35)||33 (29)|
|Yes||15 (33)||35 (31)|
|No||31 (67)||77 (69)|
|Year of publication||< 0.001|
|1980–1999||8 (17)||61 (54)|
|2000–2007||38 (83)||51 (46)|
Figure 1A shows the forest plot of differences in ES between trials with and trials without adequate concealment. Trials with adequate allocation concealment tended to show smaller treatment benefits than trials with inadequate or unclear concealment, with a difference in ES of −0.15 (95% confidence interval [95% CI] −0.31, 0.02; P = 0.08). Differences in ES between trials with and trials without adequate concealment ranged from −1.07 to 0.46, and the variability in bias estimates between meta-analyses was moderate, with a τ2 estimate of 0.06. Tests of interaction between allocation concealment and ES were positive in 3 of 14 meta-analyses at the conventional level of P = 0.05. The results of stratified analyses are shown in Figure 2. Differences in ES between trials with and without adequate concealment were larger in meta-analyses with a large treatment benefit as compared with meta-analyses with a small benefit (P for interaction < 0.001), meta-analyses with a high degree of between-trial heterogeneity (P for interaction = 0.009), and meta-analyses of complementary medicine as compared with conventional medicine (P for interaction = 0.019).
Figure 3A shows the comparisons of overall meta-analyses, including all trials with meta-analyses restricted to trials with adequate concealment of allocation. Estimates of treatment benefits became smaller in 9 and larger in 5 meta-analyses (P = 0.11). Between-trial heterogeneity decreased in 12 meta-analyses and increased in 2 (P = 0.003), and P values of pooled effects increased in 11 meta-analyses and decreased in 3 (P = 0.026). Statistical precision decreased in 9 meta-analyses and increased in 5 (P = 0.36).
Ten meta-analyses in 122 trials and 27,452 patients included both trials with and trials without adequate blinding of patients and contributed to the analysis. The characteristics of these trials are shown in Table 3. In 64 trials (52%) patients were adequately blinded, in 51 trials (42%) a placebo or sham intervention was used but adequacy of patient blinding remained unclear, and in 7 trials (6%) no placebo or sham intervention was used. Of all of the trials with adequate patient blinding, 55 (86%) reported indistinguishable interventions and 9 (14%) reported the use of double-dummy techniques. Trials with adequate patient blinding were more likely to adequately conceal treatment allocation (P = 0.006) and to evaluate complementary medical interventions (P = 0.023).
|Adequate (n = 64), no. (%)||Inadequate or unclear (n = 58), no. (%)||P*|
|Adequate allocation concealment||0.006|
|Yes||23 (36)||3 (5)|
|No/unclear||41 (64)||55 (95)|
|Yes||18 (28)||13 (22)|
|No/unclear||46 (72)||45 (78)|
|Number of allocated patients||0.13|
|>200||27 (42)||20 (34)|
|≤200||37 (58)||38 (66)|
|Yes||49 (77)||47 (81)|
|No||15 (23)||11 (19)|
|Yes||38 (59)||23 (40)|
|No||26 (41)||35 (60)|
|Year of publication||0.74|
|1980–1999||31 (48)||30 (52)|
|2000–2007||33 (52)||28 (48)|
Figure 1B shows the forest plot of differences in ES between trials with and trials without adequate blinding. Again, estimated treatment effects in trials with adequate patient blinding tended to be smaller compared with treatment effects in trials with inadequate or unclear patient blinding, with a difference in ES of −0.15, but the corresponding CI was wide (95% CI −0.39, 0.09; P = 0.22). In 2 of 10 meta-analyses, tests of interaction between patient blinding and ES were positive. The variability in bias estimates between meta-analyses was high, with a τ2 estimate of 0.07, and differences in ES ranged from −1.01 to 0.26 between individual meta-analyses. Results of stratified analyses are shown in Figure 4. Differences in ES between trials with and without adequate patient blinding were similar in meta-analyses with small and large treatment benefits (P for interaction = 0.75) and with high and low between-trial heterogeneity (P for interaction = 0.19), but differences were more pronounced in meta-analyses of nonpharmacologic interventions as compared with meta-analyses of pharmacologic interventions (P for interaction < 0.001). and in meta-analyses of complementary medicine compared with conventional medicine (P for interaction = 0.07).
Figure 3B shows the comparisons of overall meta-analyses including all trials with meta-analyses restricted to trials with adequate patient blinding. Estimates of treatment benefits decreased in 6 meta-analyses and increased in 4 (P = 0.28). Heterogeneity between trials decreased in 5 meta-analyses and increased in 5 (P = 0.44), and P values increased in 10 meta-analyses and decreased in none (P = 0.005). Statistical precision decreased in 6 meta-analyses and increased in 4 (P = 0.11).
The effects of allocation concealment became more robust after accounting for the presence or absence of adequate patient blinding (difference in ES −0.24; 95% CI −0.41, −0.07), and more precise but attenuated after accounting for intent-to-treat analyses (difference in ES −0.08; 95% CI −0.21, 0.04). The variability in bias estimates in these analyses was similar after accounting for patient blinding (τ2 = 0.07, P < 0.001), but decreased after accounting for intent-to-treat analyses (τ2 = 0.04, P = 0.002). The effects of patient blinding entirely disappeared when accounting for allocation concealment (difference in ES 0.01; 95% CI −0.18, 0.18) and were attenuated when accounting for intent-to-treat analyses (difference in ES −0.06; 95% CI −0.20, 0.09). The variability in bias estimates decreased after accounting for these characteristics: τ2 estimates were 0.03 in both analyses (P = 0.08 and 0.05 for heterogeneity, respectively).
In this meta-epidemiologic study of osteoarthritis trials, we found that trials with inadequate or unclear concealment of allocation showed larger estimates of treatment benefits than trials with adequate concealment. Evidence of bias was mainly seen in meta-analyses with large treatment effects, meta-analyses with high between-trial heterogeneity, and in meta-analyses of complementary medicine, with a pattern and magnitude of effects similar to what we found previously for bias associated with failure to perform an intent-to-treat analysis (5). The average bias associated with a lack of concealment of allocation corresponds to one-fourth to one-half of a typical treatment effect found for interventions in osteoarthritis (9). Evidence of bias associated with a lack of adequate patient blinding was found less consistently. Patients are difficult to blind if allocation is not adequately concealed: if patients and investigators enrolling patients are able to decipher the allocation schedule, subsequent blinding will be impossible. Unsurprisingly, the effects of blinding entirely disappeared after accounting for concealment of allocation in the overall analysis. However, stratified analyses suggested that adequate blinding of patients may be important for nonpharmacologic interventions. The average bias found for this group of interventions was an ES of −0.67, which is larger than the typical treatment effect found for most interventions used for osteoarthritis (9). This effect was robust to the adjustment for concealment of allocation in a post hoc analysis (difference in ES after accounting for concealment −0.62; 95% CI −1.09, −0.16).
The assessment of the methodologic quality of a trial is intertwined with the quality of reporting: the extent to which a report provides information about the design, conduct, and analysis of the trial (1). Unfortunately, reports often omit important methodologic details (26), including who was actually blinded and whether blinding was successful at the time of patient-reported assessments of pain intensity (25, 27–29). A widely used approach to this problem is to assume that the quality is inadequate unless the information to the contrary is provided. This is often justified because faulty reporting generally reflects faulty methods (1, 2). A well-conducted but badly reported trial will, however, be misclassified. Misclassification may have been particularly prominent in the assessment of the adequacy of patient blinding in drug trials. Some of these trials could have adequately blinded patients using matching placebos without describing it. The resulting misclassification would explain the apparent lack of bias associated with patient blinding in pharmacologic trials.
The current study differs in 3 important aspects from previously published meta-epidemiologic studies that addressed the impact of allocation concealment and blinding on estimated treatment benefits (2, 6, 30–34). First, we specifically estimated the extent of bias in trials using patient-reported pain intensity as a subjective outcome. Subjective outcomes are likely to be more prone to bias due to unclear allocation concealment and inadequate blinding than objective outcomes such as mortality (4). Second, almost all of the previous meta-epidemiologic studies have considered binary outcomes. To our knowledge, only one pilot study including 35 trials addressed the association between treatment benefits and allocation concealment or blinding in continuous outcomes, but was underpowered to obtain conclusive results (35). Third, we provide a comprehensive assessment of the extent of unclear concealment of allocation and the lack of patient blinding in randomized osteoarthritis trials and the resulting biases. Less than one-third of the trials reported adequate concealment of allocation. On average, these trials suggested less beneficial treatment effects than the remaining trials. Random allocation of patients can be adequately concealed in any trial, irrespective of the types of interventions compared. Admittedly, patient blinding is not possible for some interventions, such as exercise or self-management. However, even in trials that evaluated interventions that were amenable to blinding, only approximately half reported adequate attempts to blind patients.
Selection bias at trial entry might be the underlying mechanism of an overestimation in trials with inadequate or unclear allocation concealment, whereas selection bias after entry is the likely mechanism resulting in overestimates of treatment benefits in trials that exclude randomized patients from the analysis (1, 5). Lack of adequate patient blinding might result in exaggerated treatment effects due to detection bias in patient-reported outcomes and performance bias introduced by the unequal intake of analgesic cointerventions apart from the treatment under evaluation (1). Differential placebo or nocebo effects may also be important: patients who know that they receive active treatment may perceive less pain than patients in the inactive control group. In our study, these possible sources of bias introduced by the behavior and perception of patients appeared less important than the selection biases discussed above, which are mainly introduced by investigators (1).
Only a combination of adequate allocation concealment and adequate analysis according to the intent-to-treat principle will avoid selection biases and render trial results valid and credible. Special caution should be taken when interpreting the results of meta-analyses indicating large benefits of experimental interventions, a high degree of between-trial heterogeneity, or in meta-analyses of complementary medicine. Here, stratified analyses according to the presence or absence of adequate concealment of allocation and intent-to-treat analysis should be considered mandatory (5). In case of discrepancies, trials that avoided selection biases should be given precedence.
Trialists should always ensure adequate concealment of allocation and take measures to minimize dropout rates and maximize compliance with the trial protocol to allow an analysis according to the intent-to-treat principle. Blinding of patients is desirable and should be attempted. Authors of reports of osteoarthritis trials should painstakingly follow the Consolidated Standards of Reporting Trials statement (36, 37) to ensure fully transparent reporting of methods and results.
All authors were involved in drafting the article or revising it critically for important intellectual content, and all authors approved the final version to be submitted for publication. Dr. Jüni had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study conception and design. Jüni.
Acquisition of data. Nüesch, Reichenbach, Trelle, Rutjes, Liewald, Sterchi.
Analysis and interpretation of data. Nüesch, Reichenbach, Trelle, Rutjes, Altman, Jüni.
Statistical analysis. Nüesch, Jüni.
We thank Sacha Blank, Elizabeth Bürgi, Liz King, Linda Nartey, Martin Scherer, and Beatrice Tschannen for contributing to data extraction. We are grateful to Malcolm Sturdy for the development and maintenance of the database.