The impact of panel composition and topic on stakeholder perspectives: Generating hypotheses from online maternal and child health modified‐Delphi panels

Abstract Introduction Multistakeholder engagement is crucial for conducting health services research. Delphi‐based methodologies combining iterative rounds of questions with feedback on and discussion of group results are a well‐documented approach to multistakeholder engagement. This study develops hypotheses about the impact of panel composition and topic on the propensity and meaningfulness of response changes in multistakeholder modified‐Delphi panels. Methods We conducted three online modified‐Delphi (OMD) multistakeholder panels using the same protocol. We assigned 60 maternal and child health professionals to a homogeneous (professionals only) panel, 60 pregnant or postpartum women (patients) to a homogeneous panel, and 30 professionals and 30 patients to a mixed panel. In Round 1, participants rated the seriousness of 11 maternal and child health outcomes using a 0–100 scale and explained their ratings. In Round 2, participants saw their own and their panel's Round 1 results and discussed them using asynchronous, anonymous discussion boards moderated by the study investigators. In Round 3, participants revised their original ratings. Our outcome measures included binary indicators of response changes to ratings of the low, medium and high severity maternal and child health outcomes and their meaningfulness, measured by a change of 10 or more points. Results Participants changed 818 of 1491 (55%) of responses; the majority of response changes were meaningful. Patterns of response changes were different for patients and professionals and for different levels of outcome seriousness. Using study results and the literature, we developed three hypotheses. First, OMD participants, regardless of their stakeholder group, are more likely to change their responses on preference‐sensitive topics where there is a range of viable alternatives or perspectives. Second, patients are more likely to change their responses and to do so meaningfully in mixed panels, whereas professionals are more likely to do so in homogeneous panels. Third, the association between panel composition and response change varies according to the topic (e.g., the level of outcome seriousness). Conclusions Results of our work not only helped generate empirically derived hypotheses to be tested in future research but also offer practical recommendations for designing multistakeholder OMD panels. Patient or Public Contribution Pregnant or postpartum women were involved in this study.


| INTRODUCTION
Multistakeholder engagement is crucial for conducting health services research; it helps ensure that key stakeholder perspectives inform the research process and its outcomes. 1 Patients, caregivers, clinicians, researchers, payers, purchasers and policy-makers are key stakeholders 2 whose engagement can positively impact all stages of the research process. 3,4 Nonetheless, multistakeholder engagement is challenging due to logistical difficulties, power imbalances and stakeholders' capacity to participate meaningfully. 5 One way to conduct multistakeholder engagement is to convene a Delphi panel. [6][7][8] Delphi-based methodologies that combine iterative rounds of questions with feedback on intermediary panel results were designed to more objectively develop group consensus. 9,10 The Delphi method is based on the idea that exposure to alternative perspectives improves the quality of the final responses, which are used to determine the existence of consensus. Delphi-based methodologies provide a useful approach for measuring whether and how participants' perspectives change. 11,12 Modified-Delphi methodologies that start with a survey, proceed with feedback on and an in-person, telephone or virtual discussion of initial survey results, and end with participants revising their original survey responses offer stakeholders an opportunity to directly engage with each other, which is absent in traditional Delphi panels. 8,[13][14][15][16] Online modified-Delphi (OMD) approaches are particularly useful engagement techniques because they allow for largescale (50+ participants) anonymous engagement, which is not possible in modified-Delphi panels that meet in-person. The requirement of in-person discussion limits the panel size to 9-20 participants. 13,14 While the online method has clear benefits, little is known about the contextual factors such as panel composition or topic that might affect the outcomes of multistakeholder engagement.
Research suggests that stakeholder perspectives vary, with patients and clinicians, for example, having different perceptions of research priorities, treatment preferences and harm-benefit tradeoffs. 17,18 Although patients' voices may be dominated by clinicians', 17 true consensus in multistakeholder initiatives may not be achieved without directly exposing stakeholders to the perspectives of other groups. While patients may be more comfortable sharing their perspectives with peers and, therefore, could be more satisfied with engagement in homogeneous panels, participants in mixed panels could change their positions after being exposed to the alternative perspectives, which is key for developing true consensus in multistakeholder panels. 19 Although it is possible to imagine how the outcomes of a multistakeholder engagement might vary depending on its topic, we are not aware of previous studies that directly addressed this question in the context of modified-Delphi panels. This paper advances methods for conducting online multistakeholder panels using a modified-Delphi approach by exploring the impact of panel composition and topic on stakeholder judgments and uses the results of this analysis to generate empirically grounded hypotheses for future research. We use the propensity and meaningfulness of response changes after stakeholders receive statistical feedback and discuss their original responses with others as a measure of panel impact on individual stakeholder judgments. To reach the study goals, we use the data from three OMD panels that engaged patients and professionals around the severity of maternal and child health outcomes linked to gestational weight gain. 20 panel of 30 professionals and 30 patients. This panel size is consistent with the recommendations for the optimal number of participants in each OMD panel. 24 The study team members used their professional networks and social media, including Twitter and Facebook, to recruit 90 maternal and child health professionals who have worked in the field for at least five years and 90 patients-women who were either pregnant or gave birth in the past 2 years. Interested individuals residing in the United States were asked to complete a study registration form. We used stratified randomisation to assign participants to either homogeneous or mixed panels and ensured the desired composition of each panel.
Participants were informed about the number of rounds at the recruitment stage. In Round 1, participants rated the seriousness of 11 Additional details about study design 20 and its findings 21 were published elsewhere.

| Sample
Our analysis focuses on response changes to the same question between Rounds 1 and 3. We only include a participant's response to a question if it was provided in both rating rounds. Our final sample includes 143 participants and 1491 response changes.

| Variables
The main outcome variables in this study include binary indicators of response change and its meaningfulness (Yes/No). We considered a change of 10 or more points to be meaningful because it moves a response from one decile to another on the 100-point scale.
Our main predictor variable is the composition of the panel a participant was randomized into (patients in a homogeneous panel Our control variables include three measures of stakeholders' participation experiences, such as overall satisfaction and perceptions of the impact of two key features of the OMD panels most relevant to the goals of this study-statistical feedback as presented in charts and perceived ability of online discussions to change participant responses. Participants used 7-point Likert scales, where 1 = strongly disagree, 2 = disagree, 3 = slightly disagree, 4 = neutral, 5 = slightly agree, 6 = agree, 7 = strongly agree, to rate their agreement with the following statements: 1. Participation in this study was satisfying.

The charts helped me understand how my responses compared to
those of other participants.
3. Round 2 discussion changed my perspective on the study topics.
As in previous studies, we dichotomized responses and considered those scoring an item as 5, 6 or 7 as having positive participation experiences. 19,24,25 Other control variables include participants' race (White vs. other) and age.

| Statistical analysis
We used mixed-effect logistic regression to estimate the panel composition effects on the presence and meaningfulness of response changes. All models were clustered at the individual level to address within-participant correlations, and robust standard errors were produced. We first ran all the models using the seriousness ratings of all pregnancy outcomes combined (n = 1491 response changes). We then stratified all analyses by health outcome severity levels, which we considered as different panel topics. High severity outcomes included infant death, stillbirth, preterm birth and pre-eclampsia (n = 537 response changes). Medium severity outcomes included obesity in women, childhood obesity, gestational diabetes and metabolic syndrome in women (n = 542 response changes). Finally, low severity outcomes included small-for-gestational-age (SGA) birth, large-for-gestational-age birth and unplanned caesarean delivery (n = 412 response changes). Additional details on outcome severity can be found elsewhere. 21 We conducted all the analyses using STATA SE 14.

| Participation experiences
Participants were generally satisfied with their study experiences  Table 1). There were no major differences in participation experiences across panel types. Among professionals, those in the mixed panel, on average, had slightly lower scores on the questions about charts and discussions, but slightly higher scores on the overall satisfaction. Patients had slightly higher scores on all three measures of subjective participation experiences than professionals, with patients in the mixed panel being slightly more satisfied than patients in the homogeneous panel.

| Response changes
Almost all of our 143 participants changed at least one response (n = 139, 97%, data not shown). Of the 1491 questions that participants answered twice, responses to 55% (n = 818) of all questions changed in Round 3 ( Table 2). Of the 1491 responses provided twice, 563 (38%) were changed by 10 or more points (mean value of response change = 7.14, SD = 9.98; median = 5). Although the pattern of changes was similar between professionals and patients when panel type was not considered, it varied once panel type was ac-  Table 3

(22)
Previous pregnancy (20 weeks     In contrast to previous research that suggested that personal characteristics of OMD panellists were not associated with response changes, 12  | 741 statistically significant differences between patient/caregiver and clinician/research experiences with OMD panels or their willingness to use OMD in the future, 19 this study suggests that mixed panels may promote mutual learning in multistakeholder panels on certain topics.
Our study has important limitations. Our analysis was limited to three OMDs that engaged pregnant and postpartum women and health professionals on the topic of maternal and child health outcomes. We note that the women who participated in our study were highly educated. Therefore, the patterns of findings may be different in panels that engage different stakeholders and/or focus on other topics. Moreover, not all study participants answered the same questions twice or provided responses to satisfaction questions, which limited our sample size.
Nonetheless, attrition is common in OMD panels, and our participation rates were higher than in other panels. 21 Finally, this paper relies solely on the rating data and has not looked at how the content of online discussion comments affects response changes. Future research should test our hypotheses in OMD panels conducted with different types of participants and on different topics and analyse the impact of the discussion content on response changes in different panel types.

| CONCLUSIONS
We recognize that our study cannot provide conclusive answers to our research questions. That is why we consider our study results, and the empirical data it relies on, as the necessary basis for formulating evidence-informed hypotheses about panel composition and topics that should be tested in future research. Nonetheless, we believe that our results offer a number of practical recommendations, presented in Figure 3, which can help panel designers assess possible threats to achieving valid, reliable panel conclusions and encourage them to consider how panel design considerations may affect their conclusions.