Using theory of change frameworks to develop evaluation strategies for research engagement: results of a pre‐pilot study

Abstract Introduction Inadequate community and stakeholder engagement can lead to accusations that research is unethical and can delay or slow research or translation of results to practice. Such experiences have led major funders as well as regulatory and advisory bodies to establish minimal requirements for community and stakeholder engagement in HIV and other clinical research. However, systematic efforts to formally evaluate the contributions and impact of particular practices are lacking. Methods A theory of change framework aligned with Good Participatory Practice for TB clinical trials was used to develop a set of measures for use in a minimally burdensome survey of trial implementing sites. The survey was pre‐piloted with three TB trial sites in North America, South America and Asia to assess the feasibility of surveying global research sites in a systematic way, and to see if the measures captured informative variation in the use of engagement strategies and desired outcomes. Surveys were conducted at baseline and six months. In‐depth interviews were conducted with site staff prior to the baseline survey to understand how sites conceptualized the concepts underlying the framework and the extent to which they viewed their work as aligned with the framework. Results Survey measures captured considerable variability in the intensity and variety of engagement strategies, both across sites and within sites over time, and moderate variability in outcomes. Interviews indicated that underlying concepts were often unfamiliar to staff at baseline, but the goals of engagement aligned well with existing values. Conclusions Brief, targeted surveys of trial sites to characterize use of broad strategies, specific practices and some outcomes are a feasible option for evaluating good participatory practice. Additional testing is warranted to assess and enhance validity, reliability and predictive value of indicators. Options for collecting outcome measures through additional objective means should be explored.

. Theory of Change framework for evaluating Good Participatory Practices for TB Clinical Trials. A set of powerful strategies, each comprised of a range of potential practices, are hypothesized to lead to short-term, intermediate, and long-term outcomes that cumulatively result in achieving the elements outlined in the GPP-TB ethical goal. To qualify as powerful, a convincing argument or causal hypothesis had to be made for how a proposed strategy would lead to outcomes that in turn would lead to achieving the ethical goal.
For additional information contact Kate MacQueen kmacqueen@fhi360.org or Natalie Eley neley@fhi360.org  (CAG) or similar mechanism in place for TB-related research? By active we mean that the board or group has met formally with research staff at least once in the past 12 months.
• If (1) selected then value=1 • If (2) selected then value=0 (1) Yes (2) No NOTE: If YES to CAB presence question above then complete following questions; else skip to . [Non-indicator question] Q2.3 What do you call your CAB-type mechanism? CAB research diversity • Rationale: higher value indicates greater diversity of stakeholder perspectives related to TB in CAB • Range: 1-3 Q2.4 Does your CAB-type mechanism provide support only for TB-related research or for other purposes as well?
• If (1) or (4) selected then value=1 • If (2) or (5) selected then value=2 • If (3) or (6) selected then value=3 • If (7) then value=1 (1) TB-related research only (2) TB and HIV-related research only (3) A range of research activities, not limited to TB or HIV (4) TB-related research and programs (5) TB and HIV-related research and programs (6) A range of research and program activities, not limited to TB or HIV (7) Other, please describe CAB age • Rationale: higher score indicates greater sustainability of CAB mechanism Q2.5 How long has your CAB-type mechanism been in existence?
• If (1) then value=1 • If (2) then value=2 • If (3) then value=3 (1) One year or less (2) 2-3 years (3) 4-5 years (4) More than 5 years Q2.13 There are many ways that a research site may engage communities and stakeholders, in addition to using a CAB-type mechanism. In the following list please check each mechanism that has been used by your site at least once in the last 12 months.
• If (18) checked then value=0; Else value=sum of checked items • One-to-one meetings with community members/stakeholders • Routine/scheduled meeting with a community group or organization other than CAB-type mechanism • Unscheduled or emergency meeting with a community group or organization other than CAB-type mechanism • Health education event ( .15 Thinking about all the different community members and stakeholders that your site has interacted with in the last 12 months, how often was your site able to provide updates to them in their preferred language and format (e.g., face-to-face meetings, newsletters, email)?
(1) Our site has not provided any updates to community members and stakeholder in the last 12 months (2) We are rarely able to provide updates in the preferred language and format (3) We are usually able to provide updates in the preferred language and format (4) We always provide updates in the preferred language and format Engagement log • Rationale: systematic documentation indicates greater transparency and accountability • Range: 0-1 Q2.16 Does your site maintain an up to date log of communications and interactions with community members and stakeholders? This could be a spreadsheet, data entry system or paper file that records when the communication or interaction happened, who (1) Yes (2) No

Response items within measures
was involved, what was discussed and whether any follow up was required.
• If (1)  (1) None (2) One or two (3) Some but fewer than half (4) More than half but not all (5) (5) then value=4 (1) None (2) One or two (3) Some but fewer than half (4) More than half but not all (5) All Learn needs • Rationale: Use of a needs assessment will help identify opportunities to improve shared learning • Range: 0-1 Q4.6 In the last 12 months has your site undertaken a needs assessment of community stakeholder and research staff to determine potential capacity-building needs with regard to levels of knowledge and expertise needed for effective engagement with each other?
• If (1)  (1) Yes, and the opportunity led to a successful resolution (2) Yes, but we were not able to reach agreement on a resolution (3) No, but we were able to resolve the issue through other means (4) No, and we have ongoing problems or concerns about it (5) No, and the problem seems to have faded away on its own (1) A structured opportunity was not used (2) Explicit norms or rules for discussion were established, e.g., stay on topic, not interrupt, give everyone a chance to speak, turn-taking (3) A neutral facilitator was used throughout the process (4) A process was used where authority was shared equally by all stakeholders (5) A decision-making mechanism was used where all stakeholders participated in the decision (e.g., a vote was taken) (6) A neutral party arbitrated the decision after hearing arguments from all sides (e.g., an ethics committee) (7) Recommendations were presented to your site's leadership, the sponsor or the funder for a final decision (8) None of the above

Response items within measures
Deliberation as a future strategy • Rationale: The more explicitly the strategy includes plans for input beyond the research team, then the greater support there is likely to be for decision-making to be equitably shared by researchers and concerned stakeholders • Range: 0-5 Q6.5 Imagine that a conflict or tension were to arise tomorrow between research principles and/or principles of importance to other stakeholders in the local context. Which of the following statements best describes how your site would first respond? (6) then value=2 (1) Our research site has written procedures in place for this kind of event and we would consult them (2) The principle investigator for the research in question would determine the appropriate steps to take (3) We would consider which stakeholders were involved and then decide how to respond (4) We would conduct a rapid assessment to map out the issues and who would be affected by a final decision (5) We have a CAB-type mechanism that handles these kinds of things (6)  (1) Yes, and the opportunity led to a successful resolution (2) Yes, but we were not able to reach agreement on a resolution (3) No, but we were able to resolve the issue through other means (4) No, and we have ongoing problems or concerns about it (5) No, and the problem seems to have faded away on its own Trials implemented • Rationale/Hypothesis: Use of the powerful strategies will generate the awareness, political will, and/or resources needed to implement clinical trials.
• Range: open Q7.4 Has your site EVER implemented a TB clinical trial? By implement we mean your site was activated and enrolled participants.
• If (1) then go to next question • If (2) then value=0 Q7.5 How many TB clinical trials have your site implemented? Include current ongoing trials in the total.
• Value=number reported (1) Yes (2) No Trial outcomes • Rationale/Hypothesis: Use of the powerful strategies will, over the long term, lead to more positive outcomes reflective of the GPP-TB goal (access, social value, acceptability). Conversely, negative outcomes indicate potential long-term harm associated with use of the powerful strategies.
• Range: o Effective products available 0 to 1 o Access -5 to 6 o Social value -1 to 4 o Acceptability -3 to 2 Q7.6 Thinking about the most recent clinical trial that your site implemented, which of the following are true for that trial? Please check all that apply.
Intermediate & long-term outcome: • Effective products available o If any of (9) through (12) then value= 1, else 0 Ethical Goal outcomes: • If (1)  (1) Our site was not able to recruit the target number of participants (2) Our site was not able to retain and follow up with the target number of participants (3) Our site was closed early due to recruitment or retention problems but the trial continued at other sites (4) The trial was closed early (5) The trial was successfully completed (6) The trial demonstrated efficacy in one or more treatment arms (7) The trial ultimately led to new TB treatment or prevention guidelines (8) The experimental drug tested in the trial is not suitable for use in the local context (9) The experimental drug tested in the trial is now available and affordable in our local community (10) The experimental drug tested in the trial is available but many providers refuse to use it (11) The experimental drug tested in the trial is available but many patients refuse or fail to use it as directed (12) The experimental drug tested in the trial is available and successfully used by providers and patients (13) The trial results have contributed to lower TB morbidity and mortality in our country

Response items within measures
o Acceptability= +1 Sum response values above for • Access • Social value • Acceptability