Evidencing the impacts of health research: Insights from trials reported in the 2018 Australian Engagement and Impact Assessment

Issue Addressed: While definitions of impact may vary, they often refer to the wider benefits of research evidenced beyond academia. We evaluated case studies featuring randomised trials from the 2018 Engagement and Impact Assessment to better understand how the impacts of health research are evidenced and assessed within Australia. Methods: We collated and evaluated ‘ high ’ scoring case studies submitted by higher education institutions with a focus on randomised trials across all areas of health research. A qualitative coding system was used for manual content analysis to assess the key characteristics of trials reported, subsequent impacts and the methods used to evidence impacts. Results: A total of 14 case studies were identified citing 35 clinical trials. The majority of interventions were behavioural with a focus on mental, behavioural or neurodevelopmental disorders. Most trials were phase III, focused on the treatment of the indication and were funded by industry. Contribution to clinical guidelines was the highest cited research impact. While there was evidence of researchers seeking to maximise trial impact, case studies lacked details on the role of trial participants and other beneficiaries in generating impact. Conclusions: The impacts of health research can be improved through a better understanding of the priorities and agendas of funders, providing evidence of tangible impact rather than information that is contextual or predictive, and through the early development of impact strategies involving both researchers and beneficiaries. So What?: Large-scale impact exercises intended for a broad range of disciplines may not be reflective of the depth and scope of health sciences research including trials

applied measures; and a staff census as evaluated by distinct Research Evaluation Committees. 2 The goal of ERA is to assess the quality of research, defined by the ARC as: 'the creation of new knowledge and/or the use of existing knowledge in a new and creative way so as to generate new concepts, methodologies, inventions and understandings'. 3Results from ERA are used to inform government policy, assist higher education in the strategic planning of research initiatives and provide discipline-specific analyses used to promote the Australian research landscape both locally and globally. 1,2th growing interest in the need to demonstrate the value of research beyond academia, the Engagement and Impact Assessment (EI) was introduced as a companion exercise and first implemented alongside ERA in 2018. 4The inaugural EI was reflective of broader economic, social and environmental outcomes discussed within national agendas that sought to further capitalise on the publicly funded production of knowledge. 57][8] While it was acknowledged that research outcomes could be easily 'gamed' by publishing in quantity versus quality, the high cost of implementing an assessment exercise and the lack of distinct methods for impact evaluation initially deterred systemic change. 8,9While ERA and the EI do not determine the allocation of public funds for research, they are mandatory reporting exercises as aligned with government mandates. 1,2[12] Impacts from health research do not always follow a linear path and assessment narratives may not be able to account for indirect or long-term impacts of research.The study that follows aims to evaluate key characteristics of clinical trials and subsequent impacts as reported in EI 2018 case studies.This supplemental analysis will contribute to recommended best practices for impact planning within the lifecycle of a trial, in addition to global input and perspective as to a body of evidence that can be used to further support the recognition of trials research.
The key objectives were similar to other qualitative analyses of impact case studies 13 : 1. Identify, quantify and explore the characteristics of trials and subsequent impact claims as submitted by institutions in EI 2018 case studies.
2. Reflect on the types of evidence and methods used to substantiate claims of impact.

Critically appraise case study narratives for examples of
researchers or research users actively enhancing trial impact(s).

| Data collection
Impact case studies from EI 2018 that received a rating of 'high' are publicly available and indexed on the ARC Data Portal. 14The ARC Data Portal allows for simple search queries with limited use of additional syntax.The query 'clinical trial' was selected and further optimised by the portal search engine.Of the 276 case studies available, 27 were returned and reviewed in full for a readily identifiable randomised trial cited in the references of the associated research.Given the limited number of case studies available on the portal, no restrictions were observed on the subject-based Unit of Assessment.Case studies not featuring human trials research, citing non-randomised trials, or where trials were not a focus of the narrative were excluded.
Case studies for potential inclusion were discussed among the review group (SP, ST and KK) for additional appraisal and consensus.

| Data analysis
Content analysis was performed using a coding manual originally developed to further explore the impacts of trials submitted for assessment in the United Kingdom as part of the Research Excellence Framework (REF).The original authors, Hanna et al., incorporated methods of impact assessment for trials as described in the literature and further sought to pre-define categories of impact for coding. 13e EI 2018 case studies follow a similar narrative presentation as the REF to report the impacts of trials, and only minor amendments were needed within the coding manual for the Australian context.The full coding manual with amendments noted for the EI 2018 can be found in Appendix S1.
The coding process was tested by three reviewers (SP, ST and CH) on a sample of two case studies using NVivo 12.0 and yielded an average 98.10%rate of agreement with a Kappa coefficient of .84 for impacts evidenced.
The coding manual contains the following sections and amendments from Hanna et al. 13 :

Characteristics of trials reported
Standard data capture, such as the submitting institution, Unit of Assessment, name or acronym of trial, phase of trial, and so on, remain unaltered per the original manual.Amendments to section 1 include capture of trial registry details if available.The indication as described by the trial authors in the corresponding publication was also noted and aligned with the most appropriate ICD-11 chapter and code.The type of trial was updated to include prevention alongside treatment, diagnostic, screening and other.The previous data capture of primary endpoints, and whether they were met, was amended to reflect study outcomes as described by the trial author(s) and author interpretation as either in favour or not in favour of the intervention.

All categories of impact described
The category of 'dissemination and knowledge transfer' was incorporated to the master list in section 2 for ease of coding as the exemplars provided are reflective of unique impacts.During the test coding process, it was identified by the reviewers that some case study references are contextual and do not cite novel impact.The category 'contextual information' was added to capture these details as separate from coded impacts.
No formal sources to corroborate impact are required for EI 2018 case studies.In lieu, a singular code was selected for each unique impact statement presented within a case study narrative.The EI case study template also contains a section titled 'additional impact indicator information' reserved for quantifiable variables not reported within the narrative; if applicable, this information was coded as appropriate.All other categories of impact remain unaltered from the original manual.

Clinical guidelines cited
A pre-defined list of clinical guidelines was not identified for section 3 given the varying scope of non-cancer indications.Clinical guidelines were captured ad-hoc per case study from the 'Sources to Corroborate Impact' list and compiled using NVivo 12.0.

Methods used by institutions to evidence impact
It was agreed after testing that section 4 of the coding process would be reflective of the impacts described in section 2. The manual was modified to reflect the methods noted during the test coding process and updated to include 'other qualitative methods' to capture any additional data.

Examples of researchers or research users enhancing trial impact(s)
The language used for section 5 was modified from the original coding manual citing 'producer push' and 'user pull'.The terminology was simplified to better reflect potential examples of 'researchers or research users enhancing trial impacts'. 13These examples may be contextual to the case study narrative and were coded in the 'Summary of the Impact' section.
The following details were additionally recorded as part of the case study selection process for further analysis and discussion:

| Impact case studies
Of the 276 publicly available EI 2018 impact case studies, 27 were returned by the search.Of these results, 14 case studies met the eligibility criteria.Figure 1  Submitting institutions were also given the opportunity to 'identify impact studies where the impact, associated research and/or approach to impact relates to Aboriginal and Torres Strait Island peoples, nations, communities, language, place, culture and knowledges and/or is undertaken with Aboriginal and Torres Strait Island peoples, nations and/or communities'. 15Only one case study for inclusion submitted by La Trobe University identified with a narrative relevant to Aboriginal and Torres Strait Island content. 15I G U R E 1 Case study selection.

| Characteristics of trials identified
A total of 35 randomised trials were identified across the 14 included case studies.The number of trials cited per case study ranged from 1 to 7. All trial citations were unique, as there was no overlap in reported trials between case studies.The key characteristics of the trials identified are summarised in Table 3.The majority of trials  (Continues)

As shown in
Case studies (n = 14) Number Percentage a (%) Griffith University 1 7 La Trobe University 2 14 Monash University 1 7 Murdoch University 1 7 The Australian National University 1 7 The University of Melbourne 1 7 The University of New South Wales 1 7 The University of Queensland 1 7 The University of Sydney 1 7 University of Tasmania

| Categories of trial impact
The categories of impact as noted in the coding manual and the frequency with which they appear are summarised in  The Lancet 3 12 The New England Journal of Medicine 5 19 a May not equal 100% due to rounding.
14 case studies.There was no overlap in clinical guidelines cited by case studies with 19 unique publications observed.
The section 'additional impact indicator information' was provided for only one case study. 16This additional contextual information is to be supplied for quantifiable indicators not mentioned within the main study narrative.Exemplars listed on the EI case study template include return on investment, jobs created and improvement in quality-of-life years.For this singular study, 71 newspaper/magazine articles mentioned the trial and its key intervention between 2011 and 2016. 16This detail was coded under both the impact and method of alternative metrics.

| Examples of researchers or research users enhancing trial impact(s)
A total of 17 instances of researchers enhancing trial impact were observed with the most frequent being active links with industry as relevant to the trial (8/17).The development and delivery of training packages to encourage implementation of trial results were further observed in 5/17 instances.There were no instances recorded of research users (i.e., patients) enhancing trial impact.

| DISCUSSION
While the EI template advises submitting institutions to signpost appropriate evidence directly within the narrative 'such as cost-benefit-analysis, quantity of those affected, reported benefits, etc.' no formal citations are required. 15This text is vague at best and creates a good faith system that impact claims within EI case studies that have been realised in practice versus aspiration.The EI 2018 impact rating scale may offer some further insight to the quality of impacts.
Those rated 'high' were deemed by a panel of experts to have made a 'highly significant contribution beyond academia' where a clear link between the research and the impact was demonstrated. 17A total of 42 Australian institutions participated in ERA and the EI in 2018. 2 Only 13 institutions were observed for this analysis as scoring 'high' in the production of impact related to one or more randomised trials.Further public transparency and accessibility to 'medium' and 'low' rated case studies may additionally bolster the comparative quality of impact claims.
As with any narrative exercise, the structure and flow of the 14 case studies selected for inclusion varied in their written accessibility and subsequent ease of coding.For example, a case study submitted by the University of Sydney focused on the prevention and treatment of liver disease employed clear sub-headings to denote the impacts of trials research within a larger workplan of other impact initiatives. 18In contrast, a case study submitted by the University of New South Wales describing advances in the treatment of Hepatitis C utilised unstructured block text that proved challenging to clearly delineate impact claims. 19Best practices for the coding of qualitative data suggest pragmatism as defined by the study objectives and methodological constraints. 20,21The detailed examples provided by the coding manual still proved effective regardless of narrative presentation.
The EI impact case studies also provide additional data for consid-  23 The priority area of 'health' was identified for 13 out of 14 case studies.The further challenge area of 'better models of health care and services that improve outcomes, reduce disparities for disadvantaged and vulnerable groups, increase efficiency and provide great value for a given expenditure' was cited in a cumulative 12 instances.
The inclusion of these additional data points was intended to reflect the return on public investment as it relates to 'research that addresses the most immediate problems facing Australia'. 24ly one study identified impacts associated with Aboriginal and Torres Strait Island content. 15It accepts all manner of trials for both prospective and retrospective registration.3][34] However, the use of ANZCTR is voluntary and a 2018 study of the platform found that compliance with prospective registration was suboptimal unless required by stakeholder ethical approvals. 35A previous review of the cancer trial landscape in Australia suggested that registries could also be used to guide future research by identifying gaps in current trial activity compared to the known burden of disease. 36pacts of trials research featured in case studies were largely reported as global in scale.Only 3 of 14 case studies reflected impact solely in Australia (excluding New Zealand) with the remaining narratives spanning five additional continents and a veritable mix of countries.[39] In addition to reducing disease burden, global collaboration can further impact agendas of 'sustainable and culturally appropriate research environments'. 30As the COVID-19 pandemic has illustrated, the capacity to better represent under-served populations requires unified global efforts within trials research including agreed upon best practice methodologies and equitable allocation of resources. 40,41ere was a lack of transparency in funding details provided within trial publications.Of the 35 trials represented, 7 (20%) did not explicitly state where trial funding had been sourced.For example, one trial reported by the University of Melbourne featured fortified milk as a dietary intervention for the treatment of enamel subsurface lesions. 42While no funding disclosures were stated on the trial publication, it was observed that 7 of the 10 authors were from the Centre for Oral Health Science at the University of Melbourne, and the additional 3 authors were from the Meiji Dairies Corporation of Japan. 42is raises questions as to the role of a dairy company in a milk trial, and the potential financial gain to be realised if the trial proved favourable.4][45] This in turn can 'distort the medical literature and undermine clinical trial research by obscuring information relevant to patients and physicians'. 44e EI case study narratives overall did not refer to or make blanket statements regarding the public, as opposed to the health service or a specific patient population.Section 8. Social and cultural was the 1% outlier with two impact claims related to health knowledge, attitudes, and behaviours of the public.One case study coded to this heading details the public uptake of MoodGYM, an interactive selfhelp program that provides cognitive behaviour therapy training to support mental well-being. 16After publication of the MoodGYM trial efficacy results, the platform was funded by the Australian Department of Health for widespread delivery to the national public. 16As a result, the case study claims both an increased awareness of mental well-being among the general population and the potential for reduction in health-risk behaviours associated with depression and anxiety regardless of formal diagnoses. 16tably absent from the coding exercise were instances of research users (i.e., patients, carers and service users) enhancing trial impact.The success of a trial is not exclusively tied to the researcher, as the role of users in generating impact has been previously explored and is reflected within the coding manual.Examples include user/research collaboration to further develop trial research, lobbying for treatment access and discussion of trial research in public forums (see Appendix S1).Engagement with research users can be prohibited 'by lack of access to academic journals, lack of time to read long complex research papers and lack of opportunities to interact directly with the researchers'. 46Advances in widespread communication, such as social media, may improve impact outcomes in future as new means of connection and community building are observed.The overarching engagement narrative required of each institutional Unit of Assessment may also contribute to the lack of identifiable instances of either researchers or users enhancing trial impacts.
The Australian EI presents unique considerations for trialists, health researchers, academic units and funders alike: Evaluating the impact of trials • Trial stakeholders (trialists, academic units, funders, etc.) should seek to educate themselves as to the importance of inclusion of under-served populations within trials research and how impact can and should be routinely evidenced and adopted within diverse cultural traditions.
• A clear system of impact record-keeping established at the outset of a trial process can provide robust evidence of impact that is measurable and not aspirational.
Communicating the impact of trials • The use of clinical trial registries for both transparency in practice and to avoid publication bias can be further enhanced when communication of trial outputs is required as a formal matter of ethical due diligence.

Maximising the impacts of trials research
• The global trial landscape is rapidly evolving, and collaboration to further enhance trial impacts should be considered through international clinical trial networks and other data-sharing initiatives.
• The role of research users and the methods by which engagement is observed between researcher and user should be further explored to further maximise novel research impacts that can only be understood through lived experience.
The analysis of EI case studies was not without limitations.The pool of publicly available 'high' scoring case studies is small in comparison to jurisdictions such as the United Kingdom, where all impact narratives have been made freely available.As a result, only a narrow array of indications and disease areas have been presented.Full transparency through the public release of both 'low' and 'medium' scoring studies would provide a more in-depth analysis of Australian-based trial characteristics, impacts claimed, and methods used to evidence such impacts.

| CONCLUSIONS
The coding manual and methodology proved robust and encompassing when applied to trials research within EI 2018 case studies.The data analysis was further improved with the capture of additional information unique to the EI exercise, such as global impact context and relevancy to indigenous populations.The findings of this study further support and contribute to additional discourse as to the need for transparency within stakeholder processes, considerations of impact at the outset of a trial, and the adoption of impact strategies that reflect tangible evidence for wider dissemination.The impact landscape in Australia is not yet clearly defined, and such recommendations will be integral to the development of assessments that are fit for purpose within the health sciences and wider trials research.

1 |
INTRODUCTION Excellence in Research for Australia (ERA) is a research management initiative of the Australian Government and developed by the Australian Research Council (ARC).ERA was first conducted in 2010, with subsequent data collections in 2012, 2015 and 2018. 1 This longitudinal exercise is comprised research outputs; research income;

1 .
Optional Field of Research, Socio-economic Objective and Australian and New Zealand Standard Industrial Classification codes.2. Science and Research Priorities and associated Practical Research Challenges. 3. Geography of impact (summarised by continent).4. Indication of Aboriginal and Torres Strait Island content.

( 23 /
35, 66%) focused on treatment.Thirteen citations (37%) clearly noted a trial registry identifier such as the Australian New Zealand Clinical Trials Registry (ANZCTR; Aus/NZ); an International Standard Randomised Controlled Trial Number (ISRCTN; UK); or National Clinical Trial number (NCT; USA).Of these, seven citations identified the phase of trial, with the majority (6/7, 86%) reporting phase III development.The trial indication, as described by the authors, was mapped to the best fit ICD-11 chapter and code.Mental, behavioural, or neurodevelopmental disorders accounted for 34% (12/35) of all trials observed and were reflective of indications such as depression and anxiety, abusive drug-related behaviours, and other mental wellbeing and socially acceptable conducts.This aligns with behavioural interventions as the primary intervention type (15/35, 43%), followed by lifestyle interventions (7/35, 20%).There was little duplication in source of publication, with only four journals exhibiting multiple citations.

T A B L E 3
Characteristics of trials identified.Trial citations in references (n = 35Journals of trial publication (number of citations, >1) (n = 26) Methods used by institutions to evidence impact were observed as per the coding manual in a total of 91 instances.The top three methods used by institutions to evidence trial impacts included: policy citations (clinical guidelines, 18 instances), other qualitative methods (16 instances) and alternative metrics (14 instances).Examples of other qualitative methods included surveys, interviews and additional thematic analyses.Alternative metrics were reflective of indicators such as engagement with web pages and apps, digital training resources, and other forms of media production including video and podcasts.Some impacts and methods are not mutually exclusive within a case study such as the use of clinical guidelines, economic strategies (including modelling) and traditional bibliometrics of publications and subsequent citations.

1 8 3. 2 1 5 6. 2 1 4 7. 2 4 7. 3 3 7. 4 0 7. 5 0 7. 6
Clinical trial has contributed to the development (or intentional ceasing of the development) of further research, clinical trials and researchers 13 Clinical trial has led to collaboration and/ or Clinical trial has contributed to improved health for patients (other than those in the trial) and actual health changes have been evaluated 8 Clinical trial has contributed to improved health for patients (other than those in the trial) and health changes have been estimated Clinical trial has led to direct cost savings for the health service 6 Clinical trial has shown that a diagnostic or management strategy is cost effective 6 Clinical trial has led to measured or estimated benefits for the macro economy 4 Clinical trial has led to measured or estimated benefits to the macro economy from a healthy workforce 0 Measure of the intrinsic value to society of health gain from implementation of the research 0 Any economic evaluation that considers the cost of conducting the trial in the evaluation (cost/benefit analysis) 100% due to rounding.outcomes,including reduced need for caesarean sections and number of babies requiring admission to special care.Building on this landmark study, the project team were able to secure an additional 1.5 million AUD to implement midwifery support for Aboriginal and Torres Strait Islander women across four maternity services in the state of Victoria.15In 2016, 3.3% of the total Australian population (approximately 798 400) were estimated to identify as Aboriginal and Torres Strait Island people.25 Differing Indigenous cultures are under-served across global trials research including those in Australia, Canada, New Zealand, and the United States.
details the search results, including reasons for exclusion, in a PRISMA-style diagram.The case study Unit of Assessment and additional Field of Research, Socio-economic Objective and Australian and New Zealand Standard Industrial Classification codes are reported in Table 1 alongside submitting institution and geographical impact details.Impact case studies are also reflective of the Science and Research Priorities and associated Practical Research Challenges as defined by the Australian Government.One or more Practical Research Challenge may be identified per case study if applicable, as presented in Table 2.

Table 4
Key characteristics of included case studies.
, trial funding was not disclosed in 20% (7/35) of publications.Industry as a standalone funder was reported in 15% (5/35) of trials and in combination with other funding agencies in an additional 9% (3/35), accounting for a 24% T A B L E 1

Table 5
. Each toplevel heading represents an aggregate of all specific sub-categoryThe distribution of impacts varies between 10% and 18% across each top-level heading with the 1% outlier exception of section 8. Social and cultural.Both section 3. Capacity building for future research and section 4. Policy and guidelines equally accounted for 18% of the total impacts observed.The most coded impact was 4.3.Clinical trial contributed to clinical guidelines, appearing in 22 instances across the T A B L E 2 Science and Research Priorities and associated Practical Research Challenges reported in case studies.Science and Research Priorities (case studies, n = 14) Number a May not equal 100% due to rounding.