SEARCH

SEARCH BY CITATION

Keywords:

  • epidemiology;
  • randomised;
  • health services;
  • evaluation;
  • routine practice

Abstract

  1. Top of page
  2. Abstract
  3. Similarities Between RHS and RCT
  4. Differences Between RHS and RCT
  5. Examples of RHS
  6. A Register for RHS
  7. Conclusions
  8. References

The randomised controlled (or clinical) trial (RCT) is recognized as the most valid among the study designs. The use of RCT in research is widespread and well formalised. In contrast, implementations of new methods and policies in routine health care are commonly lacking a formalised design, impairing the ability to evaluate and improve health care. Use of experimental designs in health care is possible at the implementation phase of clinical or preventive action or more broad process-of-care. We propose the terminology randomised health services studies (RHS) to denote the use of a randomised design with observations in routine health care, regardless of whether randomisation is done at individual, population or process level. In contrast to RCT, the RHS should be based on the same regulative actions, funding mechanisms and ethical framework as routine health care itself. This commentary discusses the different basis, practicalities, and formalities that distinguish the RHS from the RCT. Development of a formalised framework for RHS, including distinct registration, could contribute to an increased use of valid methods in effectiveness research, thus gaining better and more direct evidence on routine medical practice.

Research on outcomes in routine health services is imperative for optimising the service and maximising health benefits. A randomised controlled trial (RCT) is considered to be the most valid method to increase knowledge because the design minimises bias and the potential of false conclusions. However, research on health service effectiveness is usually done using observational studies with a comparison group. After full coverage, the healthcare interventions are monitored only for time trends by before-after analyses without a concomitant comparison group or with a heavily selected one. The bias inherent in observational evaluations may easily obscure any potential effect, especially a small one, and the results remain inconclusive. In addition, inferiority of practices used may escape detection because bias in terms of expectations or beliefs can affect evaluation.

Research on humans is performed in a systematic way that is regulated by legislation and guidelines such as the Helsinki Declaration,1 the EU Clinical Trials directive2 and ethical approvals from Institutional Review Boards (IRBs). For health services research, there is confusion regarding the ethical rules, funding mechanisms and research practices that apply. The purpose of this communication is to emphasize the need of a randomised design in health services research or in the policy and service interventions3 and to identify the essential elements that are similar to RCTs and different from RCTs.

The experimental approach with randomisation is often the only way to obtain unbiased results. It is equally important to run a randomised study with health outcome when a totally new service is offered as when a new technology replaces an old one. A randomised design should be applied also if the advocacy of the new method is not based on an improvement in health or the evidence is only indirect. We propose the term randomised health services studies (RHS) for randomised studies in the routine health care (where the main purpose is to improve health in all its dimensions), in contrast to the traditional RCT (where the main purpose is to increase scientific knowledge).

A fundamental difference between scientific medical research and research on health services is the origin of observation or data. In a scientific study, the observation is the result of the research activity itself. Without running of the study, the observation would never have existed. The optimal design is an RCT. In health services research the practice of medicine is in place irrespectively of research and the observations are by-products of routine activity. In such a case, the optimal design is an RHS.

The terminology on health services studies is variable. In the UK, the terms of pragmatic trials or complex interventions (CXI) are used.4, 5 Here, we use the term pragmatic trial as one referring to an RCT in the original meaning of Schwartz and Lellouch.6 In the US, the term comparative effectiveness research (CER) is used in studies on the effectiveness of different health care policies and it is increasingly prioritised.7 Although there are proposals on how to evaluate policy and service intervention,3 current discussions on methods of health care evaluation focus more on statistical methods and observational design alternatives than on the randomised design.8 In our article, we propose that the routine health interventions (i.e., clinical or population actions or processes of care) can (and should) be randomised. In other words, the randomised study design can and should be applied in CER or in CXI.

A classical article of Schwartz and Lellouch6 distinguished the pragmatic and explanatory trial. Their classification emphasized the dependence of the design of an RCT on the objective (pragmatic or explanatory). Both types of trials are RCTs in our terminology, even if the pragmatic trial and the RHS have the effectiveness as their objective, that is, applicability in the real life setting. In a pragmatic trial, the observation had never existed without the trial whereas in the RHS it is a by- product of routine practice or service. Therefore, we define the pragmatic trial differently from the extension of the CONSORT statement.4

The Population Health Sciences Research Network of the Medical Research Council (MRC)5 recommended consideration of alternatives to randomised trials in evaluation of complex interventions. In addition, characteristics of the patient, population, or process drive the real life medical decisions and they may have an effect on the outcome. Only randomisation provides a solution on this problem of confounding by indication. Therefore, our proposal to separate pragmatic trials and other RCTs from RHS and to randomise the services may also contribute to the MRC guidance on CXI.

It is common to consider the evaluation of a very rare problem as impractical. One may consider not to establish a service that cannot be shown to make any health difference due to rarity of the condition. This is a deep ethical problem on which we do not have any ready made answer. Our opinion favours the latter option with a demand to evaluate the service if established.

Similarities Between RHS and RCT

  1. Top of page
  2. Abstract
  3. Similarities Between RHS and RCT
  4. Differences Between RHS and RCT
  5. Examples of RHS
  6. A Register for RHS
  7. Conclusions
  8. References

Randomisation in routine healthcare is sometimes regarded as unnecessary or even unethical. This is true especially if experimental evidence from randomised clinical trials exists. However, extending such evidence to routine practice is invariably problematic and the effectiveness may be much smaller than the effect in a randomised clinical trial. Therefore, effectiveness in routine practice should always be evaluated. The stepped wedge design5 with randomisation uses the time window of the implementation phase. Launching any new routine service always requires some time; implementation has to be gradual while capacity is built. During this phase randomisation is ethical, given that the available resources are used to their full extent and that the allocation of groups is based on the available resources. Randomisation maintains equity as—a priori—every individual in the target population has the same probability of gaining benefit and of experiencing harm. This is true independently of the outcome, whether the new activity turns out to be effective or not (or better or worse than the old one).

Randomisation is possible during the narrow time window when the new activity or technology is being implemented, but before it has become completely established. After the new health service has become fully implemented, that is, all the members in the population are covered, evaluation using randomisation may be considered impractical or unethical because part of the population cannot be denied the service. Therefore, randomisation in routine practice should start from the first application, tha tis, immediately when the activity has been approved in the public domain. This principle is similar in the RHS and in the RCT.

Differences Between RHS and RCT

  1. Top of page
  2. Abstract
  3. Similarities Between RHS and RCT
  4. Differences Between RHS and RCT
  5. Examples of RHS
  6. A Register for RHS
  7. Conclusions
  8. References

The scientific study (RCT) is carefully controlled and may result in data with very high within-study validity, but generalizing the results to routine medicine may be difficult. Also, several competing interventions may be approved for clinical use after evaluation for safety and efficacy. Research is needed to evaluate which of the policies are most (cost-) effective in the real life setting. The RHS results in data that reflect the actual real-life setting. Application to any other routine setting should, however, go via another RHS.

Scientific research and routine health services are as a rule funded from different sources. This will vary by country and health care setting. For example, the costs of an RCT can be covered from public scientific funding often operating under the ministry of education or from private funds or foundations. The data from an RHS are the result of public or private health services. Therefore, the funding can originate from the routine health budget of the health care provider (e.g., ministry of health or via various public insurance systems), or even by the subjects themselves. Typically, only a marginal amount of funding in RHS stems from research sources, sometimes nothing. As an example, in the UK the service support and treatment costs are covered by the National Health Service (NHS) while the research costs are from the Medical Research Council (MRC)5 (http://www.dh.gov.uk/en/Aboutus/Researchanddevelopment/AtoZ/DH_4016456).

The approval of interventions in the RCT is based on scientific regulation including ethical review boards. Publication of an RCT usually requires registration to some existing clinical trial registry (e.g., http://www.who.int/ictrp/en/). Routine use of drugs or diagnostics, or other interventions require approval by the health authorities, like the Food and Drug Administration (FDA) in the USA. If the intervention had already been approved in the routine practice (e.g., by the FDA) it does not seem logical, for research purposes, to ask for approval again from the ethical board. The control of subjects is considered to be better in research than in routine use. Therefore, the safety of and the effect in subjects taking part in RHS is probably better and the harm is smaller than in other medical routine. Also, the stopping rules in study protocols are designed with RCT as the model. However, these rules should not be identical for RCT and RHS. The stopping rules of medical research are designed to prevent a new invention if harmful and they cannot be extended to starting or stopping the routine medical practice; making a medical product or procedure available in the market or not follows a different logic and is decided by different authorities.

There are important differences also in the actual running of the RHS and RCT. An individual informed consent is considered a prerequisite in medical research involving an intervention. Usually the type of the study, an RCT or an RHS, has not been taken into consideration. However, in RHS the interventions (including no intervention, if this is an alternative) that are studied are both generally approved and in routine medical use. Therefore, the consent procedures used in an RHS should be identical with the consent procedures used when there is no evaluation or other research aspect of the health services. In many cases, the consent is implicitly included in the basic medical activity when the patient is actively seeking or accepting medical care. Therefore, it should be possible to approve an RHS without a formal individual consent. In fact, this is the standard practice in some countries already.9, 10

Double blinding is the golden standard in an RCT. In the RHS, the principles of the clinical practice actually in use should be followed. In clinical interventions, it is usually assumed that the patient is informed on the medical activity and she participates in medical decision-making. The clinician always knows the treatment specifics of a patient. Any placebo effect of the intervention is accepted as part of the health services effect. The arms, therefore, are open and not blinded in the RHS. However, specific diagnostic methods and therapeutic technologies in the normal clinical practice are not introduced with options of choice to the patient. The practice used in an RHS should be the same.

Thus, there are major differences between the RCT and RHS and only two obvious similarities: the randomised design itself and the fact that the experimental design is possible to use at a time when there is uncertainty about which practise is best.

Examples of RHS

  1. Top of page
  2. Abstract
  3. Similarities Between RHS and RCT
  4. Differences Between RHS and RCT
  5. Examples of RHS
  6. A Register for RHS
  7. Conclusions
  8. References

There are many examples of successful RHS, especially in the Nordic countries. In Sweden, a cluster randomised RHS was implemented in cervical screening for evaluation of routine use of HPV-based triaging in borderline or low-grade cervical abnormalities.11 There was no research funding for the RHS, instead the participating hospitals were randomised either to change their policy (seven hospitals) or not to change (eight hospitals) their policy. The coordination and evaluation office was at the regional cancer screening unit, funded by the routine health budget. As RHS was unheard of, the initial position of the IRB was that either it is scientific research and an individual written consent is required or it is routine health service and, if so, no consent or ethical permission is required. Following discussion, ethical approval was awarded with an opt-out type of consent that included printing and mailing of 180,000 information leaflets. Information for the invitees on the available testing options (HPV triaging or not) was thus systematically provided in the RHS but not in the rest of Sweden where no RHS was run. The TOMBOLA trial (Trial of Management of Borderline or Low grade Abnormalities) in the UK also studied HPV triaging, but included components of both routine health care and a scientific study as well.12 The costs were covered both by the NHS (because of routine activity components) and by the MRC (because of research components). The regulatory rules of RCT were applied also in this study, which contributed to the low participation rates.12, 13 By comparison, the Swedish study had complete participation as none of the women used the possibility to opt-out.11

Another study of screening for cervical cancer with primary HPV-test14 has been proposed in the UK, also with a budget of several million pounds. It is essentially identical to a Finnish RHS on the same subject that included over 100,000 women in 2003–2005,9 but operated mainly within the routine health budgets of the authorities. Ultimately, in the UK the pilot implementation of the HPV-test was done with an observational design of before and after evaluation.15

In Finland, the Public Health Act in 1987 gave the legislative background to run the nationwide programme of breast cancer screening as an RHS. The allocation with cluster randomisation of the target population into invitees or controls was based on birth year.16 Such cluster randomisation results in a simple outcome evaluation because the evidence is provided by routine (death) statistics.

The RHS can also be applied to routine clinical treatment17 or more broadly to the process-of-care, clinical health policy. Many of these activities are still not regulated at all if they are socially accepted or believed to result in health or in reduction of cost. Cluster randomisation may be the only option in comparing patterns of care. However, we prefer individual randomisation as long as possible as demonstrated by the examples on patterns of care in the treatment and follow-up policies18, 19 and in the implementation of colorectal cancer screening.20

Altogether, three RCTs were run on hormonal therapy (HT).21–23 These trials serve as an example on the difference of stopping rules in an RCT and in healthcare, because HT was (and still is) in routine use, approved by health authorities years before the trials. All the three randomised trials were stopped before the planned completion, because the harms were so severe that the ethical rules for scientific research prevented continuation.21–23 In medical practice, however, the use of HT still continues. The decision to close the RCTs but not the on-going medical practice had a fundamental effect on research ethics: the effects of HT on health cannot be studied anymore by the valid method of randomisation. The decision also relates to patient ethics, since use of HT in the research setting would be both safer and more controlled than in routine use. There is the dilemma that research on unsafe practises will be stopped but not the routine use of these practices. This should be addressed by stopping rules in RHS with the power to stop the routine if the study is stopped.

Although most of the examples provided are from organised cancer control programs the RHS design is generally applicable and both could and should be applied in any clinical or preventive action or process-of-care.

A Register for RHS

  1. Top of page
  2. Abstract
  3. Similarities Between RHS and RCT
  4. Differences Between RHS and RCT
  5. Examples of RHS
  6. A Register for RHS
  7. Conclusions
  8. References

The registration practices for studies evaluating health care policies vary, both because many changes in health care are implemented without prior consideration of implementation design and evaluation and because the format of registration for RCTs is not suitable for an RHS. The Estonian hormonal therapy (HT) trial was registered as an RCT21 as was the Finnish RHS on HPV screening for cervical cancer9 but the Finnish colorectal cancer RHS was not.20

The WHO has defined a minimal registration data set with 20 items for RCTs which are only partly applicable for RHS (http://www.who.int/ictrp/en/). To register an RHS in registers for randomised trials is problematic because it may further blur the distinction between an RCT (research) and an RHS (routine). In Table 1, we have modified the WHO list of data items to be registered for an RCT to fit the needs of an RHS. To avoid confusion and to improve the recognition of RHS as an entity in CER or CI distinct from RCT, the Finnish Cancer Registry hereby announces the establishment of a global RHS register (www.cancer.fi/rhs).

Table 1. A minimal data set for registration of randomised health services studies (based on the WHO recommendations for RCT) www.cancer.fi/rhs
inline image

Conclusions

  1. Top of page
  2. Abstract
  3. Similarities Between RHS and RCT
  4. Differences Between RHS and RCT
  5. Examples of RHS
  6. A Register for RHS
  7. Conclusions
  8. References

We propose that more of the routine health interventions should be evaluated using randomisation. While the principle itself is well accepted5 the application is relatively rare. The approach has mostly been used in public service environments like in the Nordic countries or in the UK. Individual randomisation may not be feasible in evaluation of population actions or processes of care, but cluster randomisation can also be used.24 The randomised application should be started early, at the implementation phase, immediately after the method has been approved for clinical use. Large scale success is based on high competence of the health services administration. Consensus statements, modelling, indirect observations, routine monitoring, or observational studies may provide helpful information, but should not replace the RHS because of the inherent bias in these alternatives.

We define randomised health services studies (RHS) as a distinct category of research in CER or in CXI that should not be confused with RCT whether pragmatic or explanatory. The difference between RCT and RHS stems from whether the observations, data, were specifically created by an intervention for research or whether they appeared as a by-product from routine use of health services. The experimental design with randomisation is, however, the same in studies in the scientific research setting (RCT) and in RHS.

RHS should be considered an integral part of evidence-based medicine. It will generate valid, large-scale data on the effectiveness and safety of medical practices in routine health services. An RHS should be allowed to use the same ethical framework and the same funding mechanisms as routine medical activities. Randomisation and timing should be similar between the RCT and RHS, whereas stopping rules, the need for an informed consent, and blinding can be different. A formal registration of an RHS should be separated from that of an RCT.

References

  1. Top of page
  2. Abstract
  3. Similarities Between RHS and RCT
  4. Differences Between RHS and RCT
  5. Examples of RHS
  6. A Register for RHS
  7. Conclusions
  8. References
  • 1
    Ethical principles for medical research involving human subjects. 59th World Medical Association (WMA) General Assembly 2008, Seoul, Korea, October 2008.
  • 2
    Directive 2001/20/EC of the European parliament and of the council of 4 April 2001. Official J Eur Union 2001; L121: 3444.
  • 3
    Lilford RJ, Chilton PJ, Hemming K, Girling AJ, Taylor CA, Barach P. Evaluating policy and service interventions: framework to guide selection and interpretation of study end points. BMJ 2010; 341: c4413.
  • 4
    Zwarenstein M, Treweek S, Gagnier JJ, Altman DG, Tunis S, Haynes B, Oxman AD, Moher D. Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ 2008; 337: a2390.
  • 5
    Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ 2008; 337: a1655.
  • 6
    Schwartz D, Lellouch J. Explanatory and pragmatic attitudes in therapeutical trials. J Chronic Dis 1967; 20: 63748.
  • 7
    Iglehart JK. Prioritizing comparative-effectiveness research-IOM recommendations. N Engl J Med 2009; 361: 3258.
  • 8
    Sox HC, Greenfield S. Comparative effectiveness research: a report from the Institute of Medicine. Ann Intern Med 2009; 151: 2035.
  • 9
    Leinonen M, Nieminen P, Kotaniemi-Talonen L, Malila N, Tarkkanen J, Laurila P, Anttila A. Age-specific evaluation of primary human papillomavirus screening vs conventional cytology in a randomized setting. J Natl Cancer Inst 2009; 101: 161223.
  • 10
    Malila N, Oivanen T, Malminiemi O, Hakama M. Test, episode, and programme sensitivities of screening for colorectal cancer as a public health policy in Finland: experimental design. BMJ 2008; 337: a2261.
  • 11
    Dillner L, Kemetli L, Elfgren K, Bogdanovic G, Andersson P, Carlsten-Thor A, Andersson S, Persson E, Rylander E, Grillner L, Dillner J, Tornberg S. Randomized healthservices study of human papillomavirus-based management of low-grade cytological abnormalities. Int J Cancer 2011; 129: 1519.
  • 12
    Cotton SC, Sharp L, Little J, Duncan I, Alexander L, Cruickshank ME, Gray NM, Jenkins D, Philips Z, Robertson A, Seth R. Trial of management of borderline and other low-grade abnormal smears (TOMBOLA): trial design. Contemp Clin Trials 2006; 27: 44971.
  • 13
    Cytological surveillance compared with immediate referral for colposcopy in management of women with low grade cervical abnormalities: multicentre randomised controlled trial. BMJ 2009; 339: b2546.
  • 14
    Cuzick J, Szarewski A, Cubie H, Hulman G, Kitchener H, Luesley D, McGoogan E, Menon U, Terry G, Edwards R, Brooks C, Desai M, et al. Management of women who test positive for high-risk types of human papillomavirus: the HART study. Lancet 2003; 362: 18716.
  • 15
    Moss S, Gray A, Legood R, Vessey M, Patnick J, Kitchener H. Effect of testing for human papillomavirus as a triage during screening for cervical cancer: observational before and after study. BMJ 2006; 332: 835.
  • 16
    Hakama M, Pukkala E, Heikkila M, Kallio M. Effectiveness of the public health policy for breast cancer screening in Finland: population based cohort study. BMJ 1997; 314: 8647.
  • 17
    van Marrewijk CJ, Mujakovic S, Fransen GA, Numans ME, de Wit NJ, Muris JW, van Oijen MG, Jansen JB, Grobbee DE, Knottnerus JA, Laheij RJ. Effect and cost-effectiveness of step-up versus step-down treatment with antacids, H2-receptor antagonists, and proton pump inhibitors in patients with new onset dyspepsia (DIAMOND study): a primary-care-based randomised controlled trial. Lancet 2009; 373: 21525.
  • 18
    Jarvenpaa R, Holli K, Hakama M. Resource savings in the single reading of plain radiographs by oncologist only in cancer patient follow-up: a randomized study. Acta Oncol 2005; 44: 14954.
  • 19
    Auvinen A, Hakama M, Ala-Opas M, Vornanen T, Leppilahti M, Salminen P, Tammela TL. A randomized trial of choice of treatment in prostate cancer: the effect of intervention on the treatment chosen. BJU Int 2004; 93: 526; discussion 6.
  • 20
    Malila N, Anttila A, Hakama M. Colorectal cancer screening in Finland: details of the national screening programme implemented in Autumn 2004. J Med Screen 2005; 12: 2832.
  • 21
    Veerus P, Hovi SL, Fischer K, Rahu M, Hakama M, Hemminki E. Results from the Estonian postmenopausal hormone therapy trial [ISRCTN35338757]. Maturitas 2006; 55: 16273.
  • 22
    Rossouw JE, Anderson GL, Prentice RL, LaCroix AZ, Kooperberg C, Stefanick ML, Jackson RD, Beresford SA, Howard BV, Johnson KC, Kotchen JM, Ockene J. Risks and benefits of estrogen plus progestin in healthy postmenopausal women: principal results from the Women's Health Initiative randomized controlled trial. JAMA 2002; 288: 32133.
  • 23
    Vickers M, Meade T, Darbyshire J. WISDOM: history and early demise-was it inevitable? Climacteric 2002; 5: 31725.
  • 24
    Duffy SW, Tabar L, Vitak B, Yen MF, Warwick J, Smith RA, Chen HH. The Swedish Two-County Trial of mammographic screening: cluster randomisation and end point evaluation. Ann Oncol 2003; 14: 11968.