Observational studies—just telling us what we want to hear or telling us where we need to look?


  • Ian R Reid,

    Corresponding author
    1. Department of Medicine, Faculty of Medical and Health Sciences, University of Auckland, Auckland, New Zealand
    • Faculty of Medical and Health Sciences, University of Auckland, Private Bag 92019, Auckland, New Zealand.
    Search for more papers by this author
  • Mark J Bolland

    1. Department of Medicine, Faculty of Medical and Health Sciences, University of Auckland, Auckland, New Zealand
    Search for more papers by this author

  • This is a Commentary on Wolfe et al. (J Bone Miner Res. 2013;28:984–991. DOI: 10.1002/jbmr.1792).

The association of osteoporosis with cardiovascular disease and increased mortality has been recognized for many years.1 Over the last decade, increased mortality after specific fractures has also been documented, though whether this represents a causal link or just reflects the frailty and comorbidities of those who fracture, is unresolved. The osteoporosis–mortality connection took on a much greater significance after a randomized controlled trial of zoledronate in patients with recent hip fracture demonstrated a 28% reduction in all-cause mortality and a 35% reduction in numbers of clinical fractures associated with randomization to zoledronate.2 Because not everybody who fractures dies, the reduction in fracture incidence could not account for the substantial reduction in mortality,3 opening up the possibility that zoledronate had some direct effect on mortality. A direct effect of bisphosphonates on cardiovascular mortality is an attractive explanation for the rapid onset of this effect and its magnitude, so there has been a significant effort to explore possible mechanisms for a cardioprotective effect of bisphosphonates, and seek further evidence for it.

In the elderly, calcification of arteries is very common, and bisphosphonates bind avidly to calcium. Thus, they are targeted to the vasculature,4 where they are postulated to influence T cell and macrophage function, and to reduce levels of inflammatory cytokines.5 The inhibition of farnesyl pyrophosphate synthase (in the mevalonate pathway) by bisphosphonates, leads to a reduction in prenylated RhoA,6 which in turn will increase tissue levels of nitric oxide. This would be expected to decrease monocyte adhesion, platelet aggregation, and vascular smooth muscle cell proliferation, and to cause vasodilation. Thus, a cardioprotective of bisphosphonates is biologically plausible.

This possibility is further explored in the carefully conducted observational study of Wolfe and colleagues5 in this issue of the Journal of Bone and Mineral Research. They accessed data on 19,281 adults with rheumatoid arthritis enrolled in a longitudinal study recruited from rheumatology practices in the United States. Patients completed 6-monthly questionnaires relating to medication use and health status. Myocardial infarctions (MIs) were confirmed by a central adjudicator. Among the 5689 patients who were treated with bisphosphonates at some time during the study period, the risk of MI while on bisphosphonate compared to when not on bisphosphonate was 0.56 (95% CI 0.37, 0.86, p < 0.01), after adjustment for multiple confounders. This approach, of focusing the analysis on the bisphosphonate-treated patients, and comparing event rates in their periods on and off treatment, is an innovative way to overcome the many significant differences between the bisphosphonate-treated subjects and the others in the cohort. Although this overcomes the problem of residual confounding between subjects, it might create a separate set of difficulties. Are patients with increasing angina or other acute medical problems more likely to discontinue their oral bisphosphonates, either because osteoporosis management seems a lesser priority at that time, or because of the possibility that the bisphosphonates may have been contributing to the chest pain? The long residence time of bisphosphonates in calcified tissue (skeletal or vascular) makes this approach less suitable with these drugs. Reassuringly, similar results emerge from this within-patient analysis to those found when users and nonusers are compared with the broader cohort.

These findings appear to agree with the findings of the Lyles' trial2 in suggesting that bisphosphonates are cardioprotective. In the presence of a reasonable biological explanation for these findings, should we be embracing bisphosphonates with even greater enthusiasm in the elderly at risk of both osteoporosis and cardiovascular disease? Possibly not, since the full picture is not quite so straightforward. Following the Lyles article,2 the same authors published very detailed analyses of the mortality effects in that study and could not demonstrate that the mortality benefit was attributable to an effect on any single organ system. The incidence of common conditions was not different between the treatment groups, but beneficial trends in mortality were seen for cardiac, respiratory, and neoplastic diseases.3 Thus, we have the puzzling situation of zoledronate-treated patients appearing to be more robust in the face of whatever pathology afflicts them, and therefore having lower all-cause mortality. Subsequently, the FREEDOM trial of denosumab showed similar beneficial trends in mortality.7 These findings motivated us to formally meta-analyze mortality data for all effective osteoporosis treatments.8 These analyses showed a 10% reduction in mortality that was not specific to any agent or class of agents, but was seen across all the effective osteoporosis therapies. The effect was more marked in trials with high mortality. Interestingly, the other major trials of zoledronate in osteoporosis did not show benefit in mortality or cardiac events,9, 10 though they were carried out in populations with much lower mortality than the Lyles study.

The situation of having limited, sometimes conflicting, data from randomized trials but a large amount of observational data to address any given question, is increasingly common in clinical medicine. Is it appropriate to attempt to resolve uncertainties from trials using observational databases? We suggest that the answer to this depends on the frequency of the particular event being considered, and needs to bear in mind the substantial risk of confounding in any observational database. Thus, in the Wolfe analysis,5 the beneficial associations of bisphosphonate use are substantial. The database contains other risk factors for cardiovascular disease, so there is a temptation to be reassured that differences between the patient groups have been corrected through careful adjustment for these other factors, as Wolfe and colleagues5 have done. However, as they note, there is potential for residual confounding. Furthermore, it has been repeatedly demonstrated in the past that the users of a variety of classes of medicines (eg, estrogen therapy,11 calcium supplements12) are quite different from nonusers, and that adjustment for these baseline differences in observational studies does not necessarily produce outcomes that accord with those from randomized controlled trials.12, 13 It was only through an adequately powered clinical trial that the cardiovascular effects of estrogen therapy were identified, notwithstanding the mass of observational data that already existed. (Subsequent reanalyses of these observational datasets have resolved some of the apparent discrepancies with the Women's Health Initiative,14 but these reanalyses were only conducted because of the randomized controlled trial findings and the fact that these were considered to be persuasive). Having been misled by observational studies in the past, we should now be much more cautious in resorting to them to resolve our uncertainties. This is particularly so in the case of cardiovascular disease and bisphosphonate use, where we have clinical trials involving tens of thousands of people. To determine whether MI is less frequent in bisphosphonate users, we should collate existing data from these trials. This will address the question much more authoritatively than is possible with even the most carefully maintained observational database. The Wolfe paper should give impetus to such further studies.

There are parallels between this dilemma and the controversy surrounding the cardiovascular effects of calcium supplements. These supplements increase cardiovascular risk in meta-analyses of randomized, controlled trials,12, 15 and yet now we have a plethora of mutually contradictory observational studies.16–21 One particular problem associated with such analyses of existing databases is that they are so easy to undertake. Thus, when a question in clinical medicine arises, many people around the world have large databases with which they can address the benefits or risks of a particular factor. This can lead to a problem of selective publication. Individuals who find a result that is significant are probably more likely to publish that result, than are those who find a nonsignificant or surprising finding. Thus, when interpreting findings from observational studies, it is important to consider how many other studies could have asked this question, and why have they not reported their findings.22 In contrast, secondary analyses of RCTs are only possible when the relevant endpoints have already been collected, and these analyses are much less subject to bias. However, individual RCTs are likely to be underpowered for secondary endpoints, requiring meta-analysis of several RCTs to address hypotheses, and such meta-analyses have their own weaknesses. Additionally, secondary endpoints may not have received the same scrutiny as the primary endpoint (for example being self-reported rather than independently adjudicated), and the restrictive entry criteria to trials result in patient cohorts that are not necessarily representative of all the patients using that drug in clinical practice. With these caveats, we need to remain mindful that the value of observational analyses is extremely limited when there are already substantial randomized trial data available.

Do observational studies then, have any value in understanding medication benefits and risks? The answer is that they do, when the clinical trial database is inadequate to address a particular question, and guidelines for such analyses have been proposed.23 For bisphosphonates, this is true for osteonecrosis of the jaw, atypical subtrochanteric fractures, and some cancers, all of which are too infrequent for significant effects to be detected in a clinical trial. It is only through the careful collation and analysis of large observational databases that the existence of these potential problems or benefits will be confirmed. In the face of case reports of osteonecrosis of the jaw in osteoporosis patients taking bisphosphonates, it is extremely valuable to have large observational databases that can compare the incidence of this problem in users and nonusers, matched for other clinical characteristics.24, 25 The same has proven to be true with concerns regarding esophageal and colon cancer.26, 27 This is brought most particularly into focus with atypical subtrochanteric fractures. These were first identified as a potential problem from case reports, but clinical trial databases were unable to adequately address this issue, partly because the necessary radiology had not been systematically collected and retained for analysis, but mainly because the events were so rare. Large observational studies have now provided a much better understanding of the effects of bisphosphonate use on the incidence of this problem.28, 29

Clinical medicine is moving into a potentially dangerous situation of having a huge number of observational databases that can be analyzed to address any clinical question in a matter of minutes. It is very important that we are not seduced by the ease of these analyses into believing that they adequately address the important clinical questions confronting us. There is a hierarchy of medical evidence that has been established for sound reasons. Observational databases are always subject to confounding and this confounding is frequently not able to be corrected for by the various adjustments that are possible. When considering effects of interventions, analyses of observational databases have value for hypothesis generation, but we should be extremely cautious before attributing causality. Analyses of observational data should only be accepted when trial databases are inadequate to address the question at hand. If we do not observe this caution, we will condemn the field to oscillations between completely incompatible positions on the benefits or risks of particular interventions, confusing ourselves, the regulators, and our patients in the process. The ideal scenario, of course, is to achieve agreement between the trial and observational databases, as was eventually possible for estrogen treatment, but we remain some way from that for bisphosphonates and cardiovascular disease.

Do bisphosphonates prevent cardiovascular disease? There is little evidence to suggest this from the trial data available, but these could be subjected to more rigorous analysis. Openness from manufacturers to allowing their databases to be pooled for such analyses is a critical part of resolving this and other related questions. What seems much more likely is that the treatment of osteoporosis itself is associated with reduced mortality and that this is not specific to a single drug class nor to a specific cause of death. Understanding the mechanism of this mortality reduction is critical to our understanding of the pathogenesis of osteoporosis. The immediate clinical significance of the mortality reduction is that it is large enough to impact on the cost-effectiveness of osteoporosis treatment, particularly in elderly subjects. As new therapies for osteoporosis become available, a consideration of their mortality effects is now clearly on the agenda of items to be considered.


IRR has received research grants or speaking/consulting fees from Merck, Amgen, Sanofi, Lilly, and Novartis. MJB has no conflicts of interest.


This work was supported by grants from the Health Research Council of New Zealand.

Authors' roles: IRR wrote the first draft of this article which was then critically revised and embellished by MJB.