Setting priorities for comparative effectiveness research in inflammatory bowel disease: Results of an international provider survey, expert rand panel, and patient focus groups


  • Unrestricted funding for the in-person Scottsdale RAND Appropriateness Panel meeting was generously provided by Warner-Chilcott, Abbott, and Shire. Unrestricted funding for the BRIDGe group is provided by Abbott, Salix, Shire, UCB, and Warner-Chilcott



Comparative effectiveness research (CER) is an emerging field that compares the relative effectiveness of alternative strategies to prevent, diagnose, or treat patients who are typical of day-to-day practice. We developed a priority list of CER topics for inflammatory bowel disease (IBD).


Following the Institute of Medicine's approach, we developed and administered a survey to gastroenterologists asking for important CER topics in IBD. Two patient focus groups were convened to solicit additional CER studies. CER topics were presented to the expert panel using the RAND/UCLA methodology. Following initial ratings, the panel met to discuss and re-rate priorities. The top 10 CER topics were identified using a point-allocation system.


Responses were collated into 234 CER topics across 21 categories, of which 87 were prioritized for discussion and re-rated. Disagreement regarding priorities was observed in 5 of 87 studies. We utilized a point-allocation system to prioritize the top-10 CER topics. These related to comparing the effectiveness of: biomarkers in IBD; withdrawal of anti-tumor necrosis factor (TNF) or immunomodulators for Crohn's disease in remission; mucosal healing as an endpoint of treatment; infliximab levels versus standard infliximab dosing; anti-TNF monotherapy versus combination therapy in patients failing thiopurines; safety of long-term treatment options; anti-TNF versus thiopurines for prevention of postoperative recurrence; and treatment options for steroid-refractory UC.


We systematically developed a list of high-priority IBD topics for CER based on a survey of gastroenterologists, expert review, and patient input. This list may guide IBD research toward the most important CER studies. (Inflamm Bowel Dis 2012;)

The U.S Institute of Medicine (IOM) defines comparative effectiveness research (CER) as “the generation and synthesis of evidence that compares the benefits and harms of alternative methods to prevent, diagnose, treat, and monitor a clinical condition, or to improve the delivery of care.”1 The purpose of CER is to “assist consumers, clinicians, purchasers, and policy makers to make informed decisions that will improve health care at both the individual and population levels.”1 Two key elements are 1) the direct comparison of effective interventions, and 2) the application of these interventions in patients who are typical of day-to-day clinical practice.2

Although there has been a recent push toward adopting practical clinical trials that evaluate different medications or treatment algorithms in the context of everyday clinical practice,3 most of the medical literature remains focused on placebo-controlled trials, as they are commonly performed to obtain Food and Drug Administration (FDA) approval of a new treatment or intervention. They rarely address scenarios that are typical of day-to-day clinical practice, a key component of CER research. A systematic methodology for identifying and prioritizing CER research topics was recently published by the IOM. The process includes obtaining input from a broad array of “stakeholders,” not just physicians, to determine which study topics should be given the highest priority for funding. This set of top 100 national priorities of CER was published in 2009 and broadly spans all areas of medicine.1

The key research topic in the area of inflammatory bowel disease (IBD) from the IOM report was to compare the effectiveness of competing biologic agents in inflammatory diseases. Yet there are additional key questions that are important to study in the day-to-day care of persons with IBD not represented in the IOM report. To date, the IOM method has not been used to develop CER priorities relevant to the day-to-day clinical practice of a specific disease or specialty. The feasibility of this process outside of a large government-funded effort has also not been assessed. To this end, we adapted the methods developed by the IOM to determine the feasibility and establish a list of high-priority CER topics specifically related to the care of patients with IBD.


Sample and Survey Administration

Following the IOM's approach for developing CER priorities,1 we developed and administered a survey to practicing gastroenterologists asking them to suggest important CER topics in IBD (Supporting Table 1). The survey was electronically distributed to an international sample of gastroenterologists from the United States, Canada, the United Kingdom, and Australia using the membership lists of the American College of Gastroenterology (ACG), Canadian Association of Gastroenterology (n = 453), the Australian IBD Association (n = 128), and the British Society of Gastroenterology (n = 1470). To reach members of the ACG, an email was sent to approximately a 15% sample (n = 1650) of randomly selected physicians after protocol review and approval from the ACG. The other groups received the invitation to participate in the study as part of an emailed newsletter sent to members of the organization on the email distribution lists.

The gastroenterologists were requested to submit up to three free text response IBD CER topics that they felt were of the highest priority. For each proposed CER study, responders were asked to comment on 1) why each topic should be a national priority, 2) the primary and secondary area to be studied, and 3) the study population to be evaluated. We collected demographic information about the individual respondents, including age, number of years in practice, number of patients seen each month with IBD, percent of patients treated with IBD, and practice type. As an incentive to complete the survey, the gastroenterologists were notified they would be entered into a lottery for one of three Lenovo Netbooks to be awarded to randomly selected survey participants who completed the survey by the deadline.

To further bolster the proposed CER list, members of the Building Research in Inflammatory Bowel Disease Globally (BRIDGe) group independently proposed CER studies. This international group of 13 IBD specialists was formed to engage in IBD research ( and draws members from the United States, Canada, Australia, and the United Kingdom, and from both university and private practice settings. To ensure adequate representation of CER topics from various facets of IBD management, each BRIDGe member was assigned one or two categories of IBD care (Supporting Table 2) from which they were asked to submit a minimum of three CER topics. Including the required CER topics, each individual BRIDGe member could propose up to 20 individual CER topics from any category.

To invite input from other stakeholder groups, we convened two patient focus groups to understand the patients' perspective of the most important IBD research topics. The patient focus groups were held at Dartmouth-Hitchcock Medical Center (six patients) and in Los Angeles through the IBD Support Foundation (eight patients) and moderated by two members of the BRIDGe group (C.A.S. and G.Y.M.) following a scripted outline. We designed the interviews to be flexible so that patients felt free to offer their opinions. The script was prepared by one of the coauthors (G.Y.M.). After the introduction about CER and goals of the meeting, the focus groups began with an open-ended probe using the “think-aloud” technique of cognitive interviewing.4 The patients were instructed to suggest ideas for CER in IBD which could be related to prevention, diagnosis, treatment, monitoring, or improving care. The interviewers then focused the respondents to formulate their ideas into appropriate CER phrasing. A list of CER topics was produced by each of the two focus groups.

Prioritization of Proposed CER Topics and Analysis

All candidate CER studies were compiled into 21 topic categories (Supporting Table 2). Duplicate CER topics were combined and proposed topics that did not meet key elements of the IOM definition of CER were excluded. A two-person team (A.S.C. and C.A.S.) worked together to bin and winnow CER topics within each of our set of 21 CER domains.5 The two reviewers evaluated each CER topic generated from BRIDGe members, surveyed physicians and patients, and determined how best to categorize each CER within our conceptual framework of CER priorities. We established key words for each bin and assigned each CER topic a bin label. We then deleted CER topics (winnowed) from the CER library for 1) lack of face validity, or 2) redundancy with other CER topics. The purpose of winnowing was to eliminate unworkable CER topics rather than to arbitrarily select the best CER topics. CER topics were rejected due to face validity concerns for reasons including inconsistency with the IOM definition, vaguely written CER topics, and confusing CER topics. Following the review of all items, we updated the CER library to reflect whether the CER topic was selected for subsequent review in our RAND panel.

The candidate CER studies were then presented to the BRIDGe group using the validated RAND/UCLA methodology.6 This approach employs a modified Delphi panel to confidentially and iteratively rate the priority of the proposed CER studies using a standard 1–9 point scale (1 = lowest priority; 9 = highest; ≥7 = high priority). The first round of rating was performed individually and confidentially with deidentified responses via the Internet.

Following the initial ratings the panel convened at a moderated, in-person, meeting to anonymously re-rate priorities after detailed group discussion of the proposed CER topics. After the second round of rating the RAND/UCLA disagreement index was calculated to quantify the level of disagreement.6 A disagreement index greater than 1.0 indicates extreme variation, whereas disagreement index values less than 1.0 reflect increasing agreement.

As a final step to better define the 10 most important CER topics, each BRIDGe member was then given 55 “points” to allocate to any of the 15 highest-priority CERs, with up to 10 points allocated to any single CER. The total points of 55 was chosen to allow for up to 10 points to be given to a single CER and sequentially lower points to less important CER topics to create a rank order from the first to tenth priority. However, the points could be distributed as each BRIDGe member saw fit, with up to a maximum of 10 points per topic. This final voting process mirrored that of the IOM where members were given 300 points and could allocate up to 10% to any single topic. This final stage was carried out after the in-person meeting via the Internet.

The Institutional Review Board at Dartmouth-Hitchcock Medical Center issued a waiver of required review for this study.


Table 1 displays the characteristics of the survey respondents. In all, 245 respondents returned their surveys (6.3% response rate). Approximately half of the respondents were from the United States (51%) and 66% had been in practice for at least 10 years. Eighty-two percent of the physicians were male. Over 50% of the respondents saw, on average, more than 20 IBD patients a month and 60% estimated that 25% or less of their practice was IBD. Forty-three percent of physicians described themselves as being in private practice.

Table 1. Demographics of Complete Respondents (N = 139)
Location of PracticeN (%)
Canada10 (7.2%)
U.K.29 (20.9%)
Australia14 (10.1%)
U.S. – Northeast30 (21.6%)
U.S. – Southeast13 (9.4%)
U.S. – Midwest21 (15.1%)
U.S. – Southwest5 (3.6%)
U.S. – West14 (10.1%)
Other3 (2.2%)
Years in practice
 < 5 years31 (22.3%)
 5–10 years17 (12.2%)
 10–20 years40 (28.8%)
 > 20 years51 (36.7%)
 < 3513 (9.4%)
 36–4556 (40.3%)
 46–5530 (21.6%)
 56–6531 (22.3%)
 > 659 (6.5%)
 Male114 (82.0%)
 Female25 (18.0%)
Number of IBD patients seen in a 4-week period
 None2 (1.4%)
 1–513 (9.4%)
 6–1019 (13.7%)
 11–2031 (22.3%)
 > 2074 (53.2%)
Percentage of patients with IBD 
 < 10%40 (28.8%)
 11–25%43 (30.9%)
 26–50%18 (12.9%)
 > 50%38 (27.3%)
Practice setting 
 Private individual practice12 (8.6%)
 Private group practice48 (34.5%)
 University practice38 (27.3%)
 Managed care practice7 (5.0%)
 VA Hospital practice11 (7.9%)
 Academic: Clinical research22 (15.8%)
 Academic: Basic science research1 (0.7%)

Respondents generated 341 candidate CER topics: 111 from the sample of U.S. gastroenterologists, 70 from the group of international gastroenterologists, 140 from the BRIDGe group, and 20 from the patient focus groups. The justification for listing a particular CER as a priority, the research areas of interest, and the specific patient populations proposed by the gastroenterologists are reported in Tables 2, 3, and 4.

Table 2. Justification for Listing a Proposed CER as a Priority
Reason for PriorityN (%)
  1. *N (Respondents) = 318.

  2. *N (Responses) = 1090.

Variability in care110 (10.1%)
Cost130 (11.9%)
Information gap162 (14.9%)
Controversy107 (9.8%)
Public interest48 (4.4%)
Disproportionate impact by subpopulation26 (2.4%)
Answer will likely influence practice251 (23.0%)
Answer important for clinical decision making235 (21.6%)
Other21 (1.9%)
Table 3. Research Area Proposed CER Is to Address
Area CER Will StudyN (%)
  1. *N (Respondents) = 139.

  2. *N (Responses) = 278.

Biologics48 (17.1%)
Immunomodulators34 (12.2%)
5-ASA therapy10 (3.6%)
Antibiotics5 (1.8%)
CAM therapy4 (1.4%)
Surgery7 (2.5%)
Risks of therapy23 (8.3%)
Postoperative management9 (3.2%)
Pharmacogenomics2 (0.7%)
Biomarkers7 (2.5%)
Imaging modalities10 (3.6%)
Preventing colon cancer18 (6.5%)
Pregnancy1 (0.4%)
Pediatrics4 (1.4%)
Extraintestinal manifestations of IBD2 (0.7%)
Etiology/risk factors of IBD6 (2.2%)
Osteoporosis3 (1.1%)
Shared decision-making7 (2.5%)
Patient reported outcomes16 (5.8%)
Patient decision support tools11 (4.0%)
Cost-effectiveness40 (14.4%)
Other11 (4.0%)
Table 4. Study Populations for Proposed CER Topics
Study PopulationN (%)
  1. *N (Respondents) = 323.

  2. *N (Responses) = 1202.

Crohn's disease224 (18.6%)
Ulcerative colitis166 (13.8%)
Adults210 (17.5%)
Children/adolescents145 (12.1%)
Women146 (12.1%)
Men143 (11.9%)
Elderly67 (5.6%)
Ethnic subpopulation39 (3.2%)
Pregnant women29 (2.4%)
Pouch patients23 (1.9%)
Other10 (0.8%)

By combining duplicate answers (n = 98) and eliminating proposals that could not be considered CER (n = 9), the 341 responses were collated into 234 unique CER questions (Fig. 1). In the first round, each of these 234 CER studies were rated for priority by each BRIDGe member on a scale of 1–9 using the RAND method described above. Based on this initial round of ranking, group discussion, and clinical relevance, the BRIDGe group (RAND panel) selected 87 CER questions to re-rate, of which 40 were rated as high priority (mean score of 7 or greater). Disagreement, based on the RAND/UCLA criteria,6 was observed in only 5 of 87 CER questions. Finally, using the point allocation system, the 15 highest priority CER topics (scores >7.5) were distilled to the final top 10 CER studies (Table 5; Supporting Table 3).

Figure 1.

Study design. CER, comparative effectiveness research; ACG, American College of Gastroenterology; BRIDGe, Building Research in Inflammatory Bowel Disease Globally; IBD, inflammatory bowel disease.

Table 5. Top 10 Comparative Effectiveness Research Studies in IBD
Final RankCompare the Effectiveness of…Points Allocated in Final Round of RankingPrevious Rank from Top 15 List
1Using biomarkers (genetics, immunology) versus standard of care in predicting response and toxicity to therapy of IBD.832
2Withdrawal of anti-TNF agent versus withdrawal of immunomodulator therapy versus continued combination therapy in Crohn's disease patients who achieved remission on combination therapy.801
3Tailoring therapy to mucosal healing versus standard clinical factors on long-term outcomes in IBD.719
3Using biomarkers versus standard clinical data to predict disease prognosis in new onset IBD.7115
5Standard anti-TNF agent dosing versus trough-level anti-TNF agent dosing in IBD.615
6Anti-TNF agent alone versus anti-TNF agent + thiopurine in patients with moderate to severe Crohn's disease failing thiopurine.548
7Efficacy and safety of long term combination anti-TNF agent + immunomodulator therapy beyond 1 year versus mono-therapy with anti-TNF agent versus monotherapy with immunomodulator5211
8Thiopurine versus anti-TNF agent in preventing clinical and endoscopic post-op recurrence of Crohn's disease after resection.373
8Efficacy and safety of infliximab versus cyclosporine versus surgery in hospitalized steroid- refractory ulcerative colitis.376
8Efficacy and safety of long-term immunomodulation versus anti-TNF therapy in Crohn's disease and UC3710

None of the CER topics proposed by the patient focus groups made the top 10 and the only CER study proposed by the patients in the top 15 was comparing the effectiveness of competing dietary interventions.

The most frequent responses for why the CER topics should be a national priority were that the answers would likely influence practice, they were important for clinical decision-making, and that there was a knowledge gap between what is known and what clinicians want to know to guide decisions. The most common areas selected for study were biologics (17.3%), immunomodulators (12.2%), and cost-effectiveness (14.4%).


We used a systematic approach mirroring the methods applied by the U.S. IOM to develop a priority list of CER studies for IBD. Two of the top 10 CER topics (including the top-rated CER study) focused on the use of biomarkers versus standard of care to predict prognosis, outcomes, and response to therapy. Optimizing safety and efficacy of biologics in IBD was exceedingly important to stakeholders and was the primary focus of 5 of the top 10 CER. Not surprisingly, respondents believed that the CER topics they submitted would likely influence practice, were important for clinical decision-making, and would help to close an information gap between what is known and what clinicians would like to know to guide clinical practice.

CER is not a new concept in IBD. There are many examples of excellent CER studies in the field of IBD; these share the goal of providing practical answers to questions that affect the day-to-day care of patients. One of the best examples is the recently published SONIC trial, which compared the use of azathioprine versus infliximab versus the combination of the two drugs in patients with moderate-severe Crohn's disease who were naïve to immunomodulators and biologics.7 Another recent CER study in IBD is the so-called “top-down, step-up” trial in Crohn's disease, which compared the competing treatment strategies of early aggressive therapy with combination infliximab and azathioprine induction with azathioprine maintenance to “conventional” therapy with multiple courses of corticosteroids followed by the sequential addition of azathioprine and then infliximab only if necessary.8 However, according to a recent review, most studies performed in the field of IBD do not meet CER criteria.9

Our study has limitations. First, only 8% surveyed U.S. gastroenterologists returned our survey. The response rate (5.4%) was even lower from international physicians, likely due to the fact that the request to participate in the study was part of a larger email bulletin and they had less time to respond. However, our response rate was similar to the experience of the Institute of Medicine study, which surveyed over 20,000 individuals and received 1758 responses (less than 8.8%) and 1268 unique CER topics.1 Additionally, it is important to distinguish this survey from a traditional survey that aims to draw generalizable conclusions about knowledge, attitudes, beliefs, or practice patterns. In this instance, we cast a wide net to draw in ideas for comparative effectiveness research, not to draw representative conclusions about community practice patterns. The survey was also subject to a responder bias, as the respondents likely represented individuals with a vested interest and expert knowledge in IBD, including the BRIDGe group. In this survey 27% of responders have practices with >50% IBD, which is considerably higher than a general community practice. Another limitation is that we did not specifically ask gastroenterologists or patients to take cost or the feasibility of the study into consideration. Although the practicalities and cost of the studies are a critical consideration if they are eventually undertaken, our main goal was to prioritize the most important, clinically relevant topics that require further research irrespective of cost or feasibility.

To our knowledge, this is the first attempt by a group, other than the Institute of Medicine, to identify the most important CER topics within their specialty field. We demonstrated that this process is feasible outside of a government-sponsored setting. We acknowledge that the prioritized list of top 10 CER topics is clearly the opinion of those who participated in the process and there are many other important day-to-day clinical questions that are relevant to the care of persons with IBD. Yet we propose that this systematic process yielded a list of CER topics that may be useful for researchers and funding agencies as a “roadmap” for prioritizing future directions of research. Since the IBD knowledge base is constantly changing, like the IOM, we propose that identifying and prioritizing IBD CER topics should be an ongoing process of continuous reevaluation and updating as conditions change and the impact of previous CER become evident. The methodology we used is feasible and could also be repeated as part of an ongoing reevaluation process and utilized by other groups interested in developing a priority list of CER in their particular specialties.


We systematically developed a list of high-priority CER topics for IBD based on a survey of gastroenterologists, expert review, and patient input that may be useful for guiding IBD research priorities that would impact day-to-day clinical practice. We demonstrated feasibility of the process outside of a government-sponsored setting, which may be helpful for other subspecialty groups interested in developing a list of leading CER topics for their specialties.


The authors thank Dr. Harold Sox for contributions in consulting on study design and following IOM methodology and for article review. Disclosures: A.S.C.: Advisory Board (Warner Chilcott, Abbott, UCB); G.Y.M.: Consultant and/or Research Support (Centocor, Setpoint, Amgen, Celgene), Speaker Bureau (Abbott, Prometheus), Advisory Board (UCB); C.A.S.: Advisory Board (Abbott, UCB, P&G), Consultant and Research Support (Elan, P&G), CME activities (Abbott, UCB); B.M.S.: Research (Shire Pharmaceuticals, Ironwood Pharmaceuticals, Takeda Pharma, Rose Pharma A/G), Advisory Board (Prometheus, Salix, Boehringer Inngelheim, Ironwood, AstraZeneca, Pfizer); B.B.: Advisory Board (Abbott, Schering-Plough Canada, Shire, Ferring, Merck, Warner Chilcott, Genentech), Speaker Bureau (Janssen Ortho, Abbott, Merck, Shire, Takeda, Ferring, Warner Chilcott) Research Support (Abbott, Merck, Amgen, GlaxoSmithKline, Millenium Pharmaceuticals, Genentech, Bristol-Myers Squibb); J.J.: Consulting, Speaker's Bureau and Educational Funding (Schering-Plough Canada, UCB Pharma, Shire, P&G and Abbott); M.S.: Research Support (Ferring Pharmaceuticals, Abbott Pharmaceuticals), Advisory Board (Schering-Plough Pharmaceuticals); F.V.: Research Support (P&G); P.M.I.: Advisory Board (Abbott, MSD, Shire, Falk), Speaker's Bureau (Shire, Abbott, MSD, Merck); S.M.D.: Speaker (Schering-Plough, Abbott, Merck, Shire), Advisory Board (Abbott, UCB, Schering-Plough, Shire, Merck); G.G.K.: Advisory Board (Schering-Plough, Abbott, UCB, Shire); L.H.: Consultant (Abbott, Salix), Research support (Glycominds); L.B.: Advisory Board (UCB), Speaker's Bureau (Centocor, Salix); P.K.: None. Contributions: A.S. Cheifetz: study concept and design, drafting of the article, acquisition of data, analysis and interpretation of data, critical revision of the article for important intellectual content of the article; G.Y. Melmed, B. Spiegel, C.A. Siegel: study concept and design, acquisition of data, analysis and interpretation of data, critical revision of the article for important intellectual content of the article; J. Talley: acquisition of data, analysis and interpretation of data, critical revision of the article for important intellectual content of the article; S.M. Devlin, L. Harrell, P.M. Irving, J. Jones, G.G. Kaplan, P. Kozuch, M. Sparrow, F. Velayos, L. Baidoo, B. Bressler: acquisition of data, interpretation of data, critical revision of the article for important intellectual content.