Developing and applying a deductive coding framework to assess the goals of Citizen/Community Jury deliberations

Abstract Background Public participation in health policy decision making is thought to improve the quality of the decisions and enhance their legitimacy. Citizen/Community Juries (CJs) are a form of public participation that aims to elicit an informed community perspective on controversial topics. Reporting standards for CJ processes have already been proposed. However, less clarity exists about the standards for what constitutes a good quality CJ deliberation—we aim to begin to address this gap here. Methods We identified the goals that underlie CJs and searched the literature to identify existing frameworks assessing the quality of CJ deliberations. We then mapped the items constituting these frameworks onto the CJ goals; where none of the frameworks addressed one of the CJ goals, we generated additional items that did map onto the goal. Results This yielded a single operationalized deductive coding framework, consisting of four deliberation elements and four recommendation elements. The deliberation elements focus on the following: jurors’ preferences and values, engagement with each other, referencing expert information and enrichment of the deliberation. The recommendation elements focus on the following: reaching a clear and identifiable recommendation, whether the recommendation directly addresses the CJ question, justification for the recommendation and adoption of societal (rather than individual) perspective. To explore the alignment between this framework and the goals underlying CJs, we mapped the operationalized framework onto the transcripts of a CJ. Conclusion Results suggest that framework items map well onto what transpires in an actual CJ deliberation. Further testing of the validity, generalizability and reliability of the framework is planned.


| INTRODUC TI ON
Public participation in health policy processes is believed to improve the quality of decision making, enhance the legitimacy of decisions and build capacity among both decision makers and publics. 1 Citizen/Community Juries (CJs) are a deliberative democratic process and a form of public participation that aims to elicit an informed community perspective on topics that are viewed as controversial or divisive, crowded with voices of those with competing interests or have scientific uncertainty around the balance between benefits and harms. 2,3 Citizen/Community Jury methods rest on the assumption that an informed public can be brought together to engage in high-quality dialogue and craft "thoughtful well-informed solutions" 4 to difficult problems. In a published essay resulting from a 2-day symposium of 25 deliberative researchers, Blacksher and colleagues 5 identified three core elements of public deliberation processes: provision of balanced information; inclusion of diverse perspectives; and reflection and discussion opportunities. If done well, they further suggested three normative goals could be achieved: an informed citizenry; reciprocity and mutual respect; and public-spirited/"common good" recommendations. Similarly, in their handbook, the Jefferson Centre 4 proposes that CJs deliver in-depth learning to participants, respectful and focussed discussions, "common ground solutions," but also expression of values and concerns, thoughtful and informed input and reasons for their recommendations. Similar elements are noted elsewhere in the literature, including the emphasis on the importance of expression and consideration of participants' values, [6][7][8] interactions between participants and exchange of diverse viewpoints, 6-8 the importance of decisions that are better informed 7,8 and recommendations that address the "common good." 8 Reporting standards for CJs have been proposed to increase the transparency and trustworthiness of CJ processes. 9 Although this may help to facilitate the quality of the organization processes (by better reporting standards that aid critical reflection), it does not inform us about whether the goals of the CJ are upheld. That is, it does not inform us about the content of the CJ deliberation-about "what happens as people deliberate". 10 Some researchers who use deliberative methods have provided tools to assess aspects or elements of the deliberation. This includes, for example, discourse quality of deliberative processes (eg Discourse Quality Index 11,12 ) that analyses the speech (or discourse) of the participants. Others still have suggested frameworks that consider the structure of deliberative events, their process and outcomes. 10,13,14 Despite the growth in scholarship on deliberative methods and an increasing interest among policymakers in the publicly generated evidence that these processes can provide, there are no benchmarks for good quality CJ content, and our understanding of the goals assumed to underpin CJs-such as respect, justified reasoning and turn-taking-remains limited.
To begin to address these gaps, we developed a deductive qualitative coding framework that operationalized the theoretical goals underlying CJ methodology and deliberations. To explore the utility of this framework, we apply it to transcripts of a recent CJ on dementia. Briefly, the Community Jury on case-finding for dementia recruited 50-to 70-year-olds with no previous self-reported diagnosis of dementia or Alzheimer's disease or mild cognitive impairment. Jurors were randomly selected from the local community using landline-based sample; quotas were utilized to ensure balance of gender and education. The Jury consisted of 10 participants (five male and five female), whose average age was 62, and educational level was mixed, ranging from some secondary education to university postgraduate. Full details of that CJ have been reported elsewhere. 15

| ME THODS
First, we compiled the "goals" of the community jury deliberation processes from two primary sources: the Citizens Jury Handbook 4 and an article presented and further developed in a two-day symposium on public deliberation attended by 25 deliberative researchers. 5 We chose these sources particularly because one is the published manual which articulates the assumptions/goals of the CJ process and the other represents the deliberation of deliberative researchers. Nevertheless, these "goals" are widely accepted in the literature as conditions that create effective deliberation. [6][7][8] We then conducted a literature search to identify existing quality frameworks or tools (focused on process and/or content) used in CJs specifically and other deliberative processes more generally.
The literature search was not meant to be exhaustive, but rather, to provide illustrative frameworks or instruments or tools (hereafter "frameworks") from which we could build a deductive qualitative coding framework to analyse CJ deliberations.
Finally, we mapped the frameworks identified in the literature against the goals underlying CJ deliberative methods, to identify whether any of these goals are currently unaddressed by the existing frameworks. Where this was the case, we added an itemthis yielded the operationalized deductive coding framework (Table 1). Finally, we mapped the operationalized framework onto the transcripts of deliberations a CJ on dementia to explore the alignment between the framework and the underlying goals of a CJ (Table 2).

| Stage 1: Developing a deductive qualitative coding framework using existing content assessment frameworks for CJs
We searched PubMed and ProQuest on 8 January 2018 for quality assessment frameworks referencing Community/Citizen's Juries, deliberative democracy and public deliberation. We did not restrict the search by language or date (Appendix 1). As it was a focussed literature search, we chose to search on the term "quality" rather than the broader terms "evaluation" or "assessment," as "quality" was the term typically used by researchers known to focus on the assessment of public deliberation exercises. 10 Two authors independently examined the title and abstract of each reference resulting from the literature searches to identify those articles which discussed quality assessment frameworks applicable to CJs (Figure 1). Where an article's title or abstract suggested that the article did not discuss this, we excluded it from further consideration. Where an article's title or abstract suggested that this article did do so, we read it in full. We then excluded those articles which, upon reading in full, were found not to discuss a framework and included those that did do so. The latter comprises the set of "studies included in qualitative synthesis" (Figure 1).
The institutional websites of all the authors whose articles discussed a framework were also examined to identify any further work on this topic (as indicated by, eg the lists of publications on their websites or a CV). We additionally examined the reference lists of all of the articles that discussed the quality frameworks (backward citation) and searched for articles that subsequently cited our included studies (forward citation). The latter was done on the assumption that any subsequent quality assessment framework on this topic would cite these earlier references. Finally, we also hand searched the contents of the Journal of Public Deliberation. All of the steps were conducted independently by two authors, with discrepancies in decisions resolved by consensus.
Items from articles that potentially assessed content quality were extracted to form a preliminary list of relevant quality assessment items based on the literature. Finally, we reviewed this list of potential quality assessment frameworks, compared them with the goals of CJs, and operationalized these to form a deductive qualitative coding framework.

| Stage 2: Analysing CJ deliberations using the deductive qualitative coding framework
Using the coding framework developed from stage 1, we piloted the application of the framework on transcripts of the jurors' private deliberations during a recent community jury on case-finding for dementia 15 that was conducted in March 2017. We asked participants of this CJ "Should the health system encourage General Practitioners to practice 'case-finding' of dementia in people older than 50?". Two authors independently examined transcripts of CJ discussions from day 2 using the deductive qualitative coding framework to identify whether text that supported the presence of the framework's elements can be identified.

| RE SULTS
The results of the literature searches were amalgamated, and duplicate references removed, leaving 768 references to examine. On examination of those references' titles and abstracts, 737 references were excluded from further consideration as they did not discuss a quality assessment framework. We read 31 references in full text, excluding 25 of them for failing to discuss a quality assessment framework and including six that did do so (see Figure 1 for search results). We mapped each of the five frameworks (Discourse Quality Index was used in two articles) onto the goals of the community jury deliberative process, as identified in the literature. [4][5][6][7][8] Table 1 maps the quality frameworks described in the included papers to key goals of public deliberation and CJs. Table 2 explores the utility of the proposed framework.

| Goal 1: Express values and preferences of participants
The

| Goal 2: Reciprocal interactions and consideration of alternative views
Respectful and reciprocal discussions between the jurors are frequently recognized as a key goal of the community jury process. 4 Blacksher further argues that in a public deliberation, jurors should be able to have an opportunity to engage with each other to "articulate and justify their positions and weigh alternate views." 5 That is, a CJ should not only offer an opportunity for jurors to voice their own opinions-but also to consider and learn from the opinions of others, whether they agree or disagree with those opinions.
This goal is frequently reflected in the included frameworks. Jurors also helped each other to understand the information provided by the experts: (Juror 5) "Didn't the doctor say when we asked him about the education, did he say that they had found that people who have a lesser education were inclined to get this, or did he…" (Juror 7) "There was a higher incidence, yes". (p100)

| Goal 4: Produce thoughtful, wellinformed solutions
Community juries' goal is to be an effective means of developing a solution to a problem or an issue that is thoughtful and well informed. 4 Two key dimensions constitute this goal: the thoughtful and well-informed aspect, and the solution aspect.
First, the thoughtful and well-informed dimension suggests that community jurors will go beyond exposure to new knowledge and its repetition to actively engaging with the new knowledge. Several of the frameworks include this element, highlighting the "understanding and application of information," 10 that additional (new) information be considered in the community jury process 17 and that jurors carefully weigh both the advantages and disadvantages of proposals being considered. 11,12 To guide our coding, we operationalized this as, "Does the information provided by the experts enrich the deliberation?". The notion of enriching is deployed here to signal going beyond mere reiteration of experts' points to active engagement with them-for example by challenging, affirming, negating and clarifying. In addition to being well informed, CJs aim to produce a recommendation (potential solution) that actually addresses the charge or question issued to the community jury. We did not identify items from other frameworks that specifically address this goal. We operationalized this using two coding questions: "Has the CJ reached a clear and identifiable recommendation?" and "Does the CJ recommendation directly address the charge that the CJ was given?" The first asks whether the jury reached any identifiable recommendation that can be discerned from its transcript, as opposed to failing to come to a recommendation. The second queries whether the jurors' recommendation specifically addresses the specific charge or challenge that was issued to the jury.
Applying the framework: The jury recommendation was clear and identifiable in the deliberation transcript, and it directly addressed the charge that the jury. The jurors unanimously voted against the jury charge: "Should the health system encourage GPs to practice case-finding of dementia in people older than 50?" clearly and identifiably offering a recommendation (against the practice of case-finding in dementia), and directly addressing the question. In addition to this, the jury also recognized the practice was currently endorsed and so also made some recommended amendments to the current guidelines.

| Goal 5: Provide reasons for recommendations
Community juries need to not only produce a recommendation-but one that is backed by reasons or justifications. 4 (what we as a community would like to do collectively). 4,5 This goal is commonly reflected in the frameworks. For example, De Vries 10 notes the "adoption of societal perspective," and similarly, Han 11 and Himmelroos 12 emphasize the "common good orientation." Anderson and Hansen 16 approach the issue from the opposite side, noting the importance of "minimizing the use of arguments referring to narrow self-interest." We operationalized this goal by querying: "Does the CJ deliberation reflect a societal (rather than individual) perspective?" Is there evidence in the transcripts that the jurors differentiated between the decision they might make for themselves personally and the decision they would make for the community as a whole.
Applying the framework: This goal is exemplified in the following juror remarks:

| D ISCUSS I ON
To our knowledge, this is the first attempt at generating a deductive coding framework and mapping it against the key goals of Citizen/ Community Jury by analysing the transcripts of CJ deliberations. As CJs are becoming more popular for addressing complex policy questions in a wide range of areas, 18 it is becoming increasingly important to ensure the CJs meet the goals of a deliberative democratic process, such as participant engagement and reciprocity, expression of preference and values, and well-informed recommendations.
We identified six key goals of public deliberations and Citizen/ Community Juries more explicitly. 4,5 The proposed coding framework addresses these, by using eight questions which are directly mapped to quality frameworks identified from the literature. The proposed framework brings together these goals and quality frameworks and operationalizes them by developing questions to help guide analyses of deliberative transcripts. The coding framework has the potential to improve the use of CJs by demonstrating their capacity to uphold the goals of deliberative processes to produce considered and informed recommendations for the society as a whole. Our coding framework can be used retrospectively and prospectively. Retrospectively, it can assess whether the goals of deliberative processes were met, while prospectively, it can help to guide the facilitator to structure deliberation to meet the goals of CJs.
Lack of uptake of the recommendation generated from past CJs indicates that policymakers may lack trust in CJ processes-or that CJ sponsors (researchers, policymakers, etc) are not building their translation processes into CJ design. 19 Researchers have conducted multiple CJs, experimented with methods (eg recruitment, presentations of experts, dissemination of materials and quantitative analyses of knowledge) 18,20 and written reporting templates in an attempt to provide evidence of robustness and stability of decision making. 9,21 The approach proposed here is an attempt to explore whether CJ deliberations uphold the goals of deliberative process and thus provide another reason for decision makers to trust the outcome of CJ processes.
It is a potential limitation that the six goals considered here have been derived from two key documents, 4,5 as this leaves open the possibility that additional goals, considered elsewhere in the literature, may have been missed. The comprehensiveness of these six goals will therefore have to be formally corroborated. However, it is reassuring that most of the six goals considered here are echoed elsewhere in the literature on the theoretical goals that underpin community jury and deliberative democracy approaches more generally. [6][7][8] Likewise, it is a potential limitation that in developing this coding framework, we have also explored its use in one CJ. We welcome other CJ researchers to use this framework to assess its validity, generalizability and reliability. We have planned a second pilot evaluation to compare the researchers' qualitative assessment of each of the framework's items from the CJ deliberative transcript, with the self-reported views of the jurors using survey items which were designed to align with the framework.
To improve the trustworthiness of CJs in the minds of policymakers, we must demonstrate that the constructs of robust deliberative democratic techniques are upheld. This coding framework has the potential to assess CJ deliberations at least as they pertain to the key goals of Citizen/Community Juries and deliberative democratic processes. Used together with the CJCheck reporting template 9 to describe CJ processes, we can progress towards routinely using deliberative democratic techniques like CJs for difficult and controversial health policy decision.