CJCheck Stage 1: development and testing of a checklist for reporting community juries – Delphi process and analysis of studies published in 1996–2015

Abstract Background Opportunities for community members to actively participate in policy development are increasing. Community/citizen's juries (CJs) are a deliberative democratic process aimed to illicit informed community perspectives on difficult topics. But how comprehensive these processes are reported in peer‐reviewed literature is unknown. Adequate reporting of methodology enables others to judge process quality, compare outcomes, facilitate critical reflection and potentially repeat a process. We aimed to identify important elements for reporting CJs, to develop an initial checklist and to review published health and health policy CJs to examine reporting standards. Design Using the literature and expertise from CJ researchers and policy advisors, a list of important CJ reporting items was suggested and further refined. We then reviewed published CJs within the health literature and used the checklist to assess the comprehensiveness of reporting. Results CJCheck was developed and examined reporting of CJ planning, juror information, procedures and scheduling. We screened 1711 studies and extracted data from 38. No studies fully reported the checklist items. The item most consistently reported was juror numbers (92%, 35/38), while least reported was the availability of expert presentations (5%, 2/38). Recruitment strategies were described in 66% of studies (25/38); however, the frequency and timing of deliberations was inadequately described (29%, 11/38). Conclusions Currently CJ publications in health and health policy literature are inadequately reported, hampering their use in policy making. We propose broadening the CJCheck by creating a reporting standards template in collaboration with international CJ researchers, policy advisors and consumer representatives to ensure standardized, systematic and transparent reporting.


| BACKGROUND
It is incumbent on researchers to provide adequate descriptions of their research methodology and methods when reporting their findings in the peer-reviewed literature. Where research methods are well described, we better understand the process by which the findings have been generated and have more confidence in appraising the credibility of these outcomes. Reporting standards also create an incentive for researchers to be meticulous in designing studies. Reporting guidelines have been developed for a range of methods and disciplines for these reasons (e.g. CONSORT, PRISMA, STARD, 1 CHEERS 2 ) and some journals now require these checklists to be completed prior to publication. More recently, templates have been developed to provide guidance for describing the interventions tested in research (TIDieR 3 ). When used, these checklists allow comparisons between studies, enhance transparency, support trust in the process and provide robust foundations for future research.
In Western liberal democracies, there has been an increased openness in government public policy processes to include public voices in policy debate and formulation. [4][5][6] Public engagement processes are gaining popularity, but their methods vary. 4,7 Three broad methods of public engagement include communicative (e.g. public meetings, website information), consultative (e.g. opinion poll, focus groups) and participatory (deliberative poll, citizen jury). 8 This article is focused on a group of participatory, deliberative democratic processes with methodologies similar to those described by the Jefferson Center. 9 Some have been called citizens' panels, citizens' fora and community/ citizens' jury. We refer to these, collectively, as community juries (CJs).
As a participatory form of public engagement, CJs aim to illicit an informed community perspective on controversial or difficult topics where the values and preferences of community members are sought. For this reason, CJ participants are recruited from the general population and deliberate on questions that require an ethically sensitive or value-based decision.
Common CJ elements include extensive provision of information from expert presentations, questioning opportunities, substantial time for deliberation and formulation of a consensus or majority "verdict" by CJ participants. 9 CJs have been used to explore community perspectives on several health areas including screening, 10-12 resource priority setting, 13,14 informed consent processes for screening decisions 15 and pandemic planning. 16 These particular examples were conducted for research purposes and none formally linked CJ outcomes to policy decisions.
But in Australia, the extent to which CJs are embedded in policy processes is changing. Recently, CJs have been used by local and provincial governments to garner public input to policy making. [17][18][19] However, despite the increased use in policy decision making, some hesitancy exists around the trustworthiness of CJ processes. 20 If the process methods of CJs were reported comprehensively, consistently and transparently, their use in shaping health policy may increase.
Over a decade has passed since Abelson et al. 4 suggested design and evaluation principles for public deliberations. Four key components were identified: representation of citizenry, documentation of deliberative procedures, descriptions of information provided to community members and reporting of outcomes. However, inconsistencies in reporting of these components still occur. A recent systematic review explored the adaptations in CJ methodologies and reported wide variations in how CJs were conducted and reported. 21 Also, in a review of public deliberation in health policy, Abelson et al. 5 continued to find inconsistencies and ambiguity in reporting method descriptions. Different CJs could legitimately have different outcomes even if they followed the same methodology (random recruitment, same expert information, etc.), but poor and/or inconsistent reporting means it is difficult to compare CJ findings or consider potential reasons for differing results, and it is not yet empirically clear how methodological variation may alter outcomes.
We cannot assess quality until we are clear about methodology.
Implementers of CJs know little about what might enhance or detract from the legitimacy of claims about the representation of different types of public; quality of the processes of information provision and public deliberation; or the authenticity of the outcomes. We sought to determine the adequacy of process reporting in CJs published in the peer-reviewed health and health policy literature. We used two processes. First, we conducted a Delphi survey with a group of published CJ researchers to generate an agreed checklist of CJ characteristics and research methods that are important to report in publications. Second, using this checklist, we then reviewed published CJs to establish the utility of the checklist and determine the most frequently reported and missing descriptions of the CJ methods.

| Stage 1: initial CJCheck development
Fourteen published Australian CJ researchers and selected government policy advisors attended a CJ Research Symposium at Bond University, Australia, in July 2015. Participants were invited if they were active Australian CJ researchers (13 invited, 10 attended); active policy advisors/government workers (four invited, all attended or were represented); and/or consumer representatives (two invited and neither able to attend). Symposium goals were broad with an aim to discuss several key research and policy questions. Outcomes included the identification of several research agenda and collaborative links.
One research question arising from the symposium was to identify potentially important CJ characteristics and processes and determine whether these were reported in published CJ studies. Methodological elements of CJs were brainstormed and iteratively developed by all symposium participants in small group, round robin exercises.
A smaller self-selected working party (from the larger symposium group) participated in a two-stage Delphi method 22 to refine these CJ elements.
The first Delphi round was conducted in September 2015. The survey consisted of the 17 reportable methods suggested by the larger CJ symposium group. These criteria were divided into four groups (planning the CJ, characteristics of the jurors, CJ procedures and CJ schedule).
Respondents were asked to rate each criterion on a five-point Likert scale (1="important" to 5="unimportant"). Changes to item wording and new items could be suggested for ranking in the second round. Items that were scored by six or more respondents as "important" or "somewhat important" were retained. Items that had fewer than five respondents scoring in these top two categories were re-ranked in the second round.
The second round was conducted in October 2015. Questions were again ranked on a five-point Likert scale. As before, there was one optional, free response question, for further suggestions. On this occasion, respondents were also asked whether the item was important but not necessary to report.
The final checklist (CJCheck) was used in data extraction. Seven items were rated as either "yes" or "no," while the remainder as "yes," "no" or "unclear." Specific checklist criteria are reported in Table S1.

| Stage 2: selection of CJ studies and data extraction
We used two strategies to search for included studies. First, included articles were drawn from two recent systematic reviews pertaining to deliberative democracy. 21,23 These systematic reviews included a broad range of deliberative democratic techniques; therefore, we only included articles from these reviews that nominated the deliberative democratic technique most resembling CJs (e.g. random or representative selection of jurors, provision of balanced information to jurors, sufficient time allocated to deliberation). 8 Included studies from these two reviews described the deliberative process as "citizens' panels," "community or citizens' juries," "citizens' council" or "citizens' fora").
In addition to including relevant studies from the two previous reviews, we replicated the search strategy of Degeling et al. 23  Remaining consistent with others, 9,21,23 we included studies that described the deliberative process as having characteristics similar to CJs.
Our inclusion criteria required studies to describe any or some characteristics resembling those outlined by the Jefferson Centre 9 regardless of whether they were reported in detail. For example, studies were required to define themselves as conducting a citizen jury, community jury, citizen panel or fora, providing information to participants from different "experts" and having a deliberative component. 9 Protocols were excluded.
All studies were screened independently against eligibility criteria by two authors (R.T. and R.S.). Conflicts were checked periodically and resolved through discussion. Criteria definitions were clarified where necessary.

| Data analysis
Data extraction was conducted by the same two authors using the checklist developed in the Delphi rounds. Data were analysed descriptively using Excel.

| Stage 1: initial CJCheck development
In the first Delphi round, 13 CJ methodology reporting criteria were identified as either important or somewhat important and retained. In the second Delphi round, nine items were presented: four from round one that required further ranking and five new items. Following the second round, a further three questions were indicated to be important in the reporting of CJs and were retained, two were combined with other retained items, one item was reworded and retained and three were excluded. The Delphi round outcomes are reported in Table 1. No items were consistently ranked as "somewhat unimportant" or "unimportant." Six checklist items were rated higher in importance than others. To be designated as an important item, it had to be rated either important or somewhat important by a clear majority of respondents (8/8 respondents in the first round and 8/9 respondents in the second round).
The final checklist contained 17 items grouped into four areas: planning of the CJ; juror information and characteristics; procedural information; and the scheduling of CJs.

| Stage 2: adequacy of CJ method reporting
From the two systematic reviews, 21,23 45 studies were excluded because they were not described as a CJ and 22 studies were reported in both reviews, leaving a total of 46 studies for full-text review. In addition to these, our focused literature search modified from Degeling et al. 23 found 1598 de-duplicated articles for further analysis, of which only 18 met the inclusion criteria and were included in full-text review and data extraction. We excluded 26 full-text articles from data extraction, leaving 38 studies in our final analyses (see Fig. 1).
Included studies and their characteristics are listed in Table 2, with data extraction details of included studies available in Table S2 and reason for study exclusion from full-text review provided in Table S3.
Studies varied widely in their reporting of the methodological processes necessary for reliable CJ reproduction. No study reported all 17 checklist items. On average, only 53% of checklist items (nine items from a possible 17) were reported in any given article (SD=19.5%) and this proportion ranged from 12% (2/17 items 13 ) to 88% (15/17 items 24 ). Overall, the checklist item reported most consistently was "Was the number of jurors reported" (92%, 35/38 studies). In contrast, the least reported item was "Are the expert presentations available," which was only reported in 5% (2/38 studies) of articles. Figure 2 shows the proportion of articles reporting checklist items.

| Planning the CJ
When reporting about planning a CJ, 21% (8/38) of studies described whether they had a stakeholder group or committee and if so, the role of this group in CJ planning. Less than half (45%, 17/38) stated how and why the experts were chosen; however, more (58%, 22/38) defined the roles of the experts and most (89%, 34/38) gave a clear description of the jury "charge."

| Information about the jurors
The majority of studies reported some type of juror information. The recruitment strategy was described in 66% (25/38) of studies and What was the influence of the jury outcome on policy?
What was the framing/nature of jury deliberations?
What was the influence of the commissioning body on the jurors?
Re-ranked from Round 1 Was the role and experience of the facilitator described (e.g. impartial, informed, member of research team, independent)?  What was the framing/nature of jury deliberations?
What was the influence of the commissioning body on the jurors?

| Procedural information
There was a wide variation in what procedural information was pro-

| CJ scheduling information
Overall, CJ scheduling information was poorly reported. The frequency and timing of deliberations was reported in only 29% (11/38) of examined studies. The number of presenters and their topics was reported less than half the time (45%, 17/38) and the availability of expert presentations was reported in only 5% of studies (2/38).
Was the selection of experts (who was chosen and why) adequately described?
* Were the experts roles clearly defined?
* Was the Jury "charge" or instruction clearly described?
* Was the study recruitment strategy clearly described?
* Were inclusion/exclusion criteria reported?
Was the type of participant/juror described (unaffected public, affected public, advocate)? Were the demographics of the jurors reported (age, gender, education, attainment)?
* Was the number of jurors reported?
Was the role and experience of the facilitator described (e.g. impartial, informed, member of research team, independent)? Were materials provided to the jurors adequately described and accessible?
Was the expert cross-examination opportunities described?
Was the jury outcome reported?
Was the framing/nature of jury deliberations described?
* Was the schedule of events (meeting frequency, timing of deliberations) described?
Was the number of presenters and their topics described? Our study has numerous strengths and some limitations. We conducted a focused review of published CJ studies within the health sector and mapped descriptions of their methods to criteria developed by CJ researchers. The criteria checklist had two rounds of iterations by published CJ authors to ensure that important methodology and processes were captured. Studies were then independently rated using these criteria for completeness of reporting. However, the criteria may be somewhat limited as they were developed by a group of Australian CJ researchers and policy advisors and may not reflect the wider view of other CJ organizers. We consider this an important future development. Additionally, we rated the adequacy of reporting on the basis of the written publication and did not contact the authors for further information. This was a deliberate choice, as our purpose was to assess the adequacy of reporting of published CJs.
Other CJ research has reported the variations in methodology. 5,21,23 This was the first study to quantify those variations by mapping researcher-identified methodological reporting criteria.
Our process was similar to the initial development of the TIDieR checklist, 25 the uptake of which seems likely to strengthen the utility of intervention research. We intend to expand our consortium of CJ researchers, policy advisors and consumer representatives to develop an internationally agreed CJ reporting template. We expect that the legitimization of CJs' input into public health policies will be assisted by the use of standards for reporting of methodology and methods.

| CONCLUSION
To fully report processes within a CJ, it is important that the planning of the event, information about the jurors and their recruitment, and procedural and technical information be available and clearly documented. Our findings suggest that many current CJ publications in health and health policy literature are inadequately reported.
Therefore, methods are not transparent and readers cannot compare CJ processes and outcomes. This may not be solely a result of inadequate reporting, but also due to the absence of specific reporting standards for CJs.
Street et al. 21 suggested that strict adherence to CJ methodology as described by the originators might limit the generation of new knowledge. We agree. We advocate that CJ organizers and authors be committed to rigour and openness, but also innovation. It is through testing and adapting methodologies that new ideas are developed and understanding expands. As researchers who utilize CJ methodology, it is not our intention to restrict methodological development, but to enhance understanding. In this study, we present an empirically developed and trialled, proposed checklist for reporting CJ methodology and methods. We advocate broadening CJCheck by creating a reporting standards template developed by a consortium of international CJ researchers, health policy advisors and consumer representatives to use when designing and reporting CJs.