International perspectives on suboptimal patient‐reported outcome trial design and reporting in cancer clinical trials: A qualitative study

Abstract Purpose Evidence suggests that the patient‐reported outcome (PRO) content of cancer trial protocols is frequently inadequate and non‐reporting of PRO findings is widespread. This qualitative study examined the factors influencing suboptimal PRO protocol content, implementation, and reporting, and use of PRO data during clinical interactions. Methods Semi‐structured interviews were conducted with four stakeholder groups: (1) trialists and chief investigators; (2) people with lived experience of cancer; (3) international experts in PRO cancer trial design; (4) journal editors, funding panelists, and regulatory agencies. Data were analyzed using directed thematic analysis with an iterative coding frame. Results Forty‐four interviews were undertaken. Several factors were identified that could influenced effective integration of PROs into trials and subsequent findings. Participants described (1) late inclusion of PROs in trial design; (2) PROs being considered a lower priority outcome compared to survival; (3) trialists’ reluctance to collect or report PROs due to participant burden, missing data, and perceived reticence of journals to publish; (4) lack of staff training. Strategies to address these included training research personnel and improved communication with site staff and patients regarding the value of PROs. Examples of good practice were identified. Conclusion Misconceptions relating to PRO methodology and its use may undermine their planning, collection, and reporting. There is a role for funding, regulatory, methodological, and journalistic institutions to address perceptions around the value of PROs, their position within the trial outcomes hierarchy, that PRO training and guidance is available, signposted, and readily accessible, with accompanying measures to ensure compliance with international best practice guidelines.


| INTRODUCTION
Patient-reported outcomes (PROs) enable the assessment of cancer, its impact, and treatment, from the patient's perspective, and are collected using validated self-reported questionnaires. PROs provide valuable information, including data on physical symptoms, treatment toxicities, psychosocial problems, global health-related quality of life, to assess treatment effectiveness, and tolerability. 1,2 Patient-reported outcome data support informed decisionmaking by patients from diagnosis and throughout treatment. 2 Its value has been recognized by key stakeholders including clinicians, funders, regulators, and policy-makers. [3][4][5] However, the growing literature suggests that the quality of PRO data may be compromised 6 through the omission of PRO-related content in trial protocols. 7,8 Moreover, PRO data are often poorly reported [9][10][11] or not included in trial publications. 12 Despite this evidence, a substantial, and growing, number of randomized controlled trials (RCTs) across all cancer types now include PROs as primary and/or secondary endpoints. 13,14 Poor PRO protocol content and subsequent poor outcome reporting reduce the extent to which PRO results reach and inform clinical interactions and decision making, while simultaneously devaluing the contribution of trial participants providing this information. Qualitative methods have provided important insights into the challenges when administering PROs in trials. 6 To date, there is limited qualitative research exploring the factors affecting PRO trial design, data collection and reporting, or patients' and clinicians' access to PRO data to inform decision making.
The Evaluation of Patient-Reported Outcome Protocol Content and Reporting in UK Cancer Clinical Trials (EPiC) study used mixed methods to investigate PRO protocol content and reporting in a cohort of international cancer clinical trials. Phase I highlighted inadequate PRO protocol components, widespread non-reporting of PRO trial results, and considerable delay and poor standards of reporting where PRO data were published. 8 More than one-third of the trials, involving 49,568 participants, failed to publish their PRO findings.
The current paper reports Phase II of the project: a qualitative study involving key stakeholders. Our aim was to examine and describe perceptions of the factors influencing suboptimal PRO protocol content, implementation, and reporting, including barriers, solutions, and examples of good practice.

| METHODS
The study was completed according to the published protocol, 15 conducting semi-structured face-to-face, and telephone interviews between May 2017 and January 2018. The study was approved by the University of Birmingham Ethics Committee (Ref: ERN_17-0085).

| Recruitment
Purposive sampling was used to recruit participants from four stakeholder groups: (1) trialists and chief investigators with experience of cancer trials collecting a PRO as a primary or secondary outcome; (2) individuals with lived experience of cancer; (3) international experts in PRO cancer trial design, including members of international cancer and quality of life research organizations, national advisory bodies and industry; and (4) journal editors, funding panelists, and regulatory agency representatives (Appendix S1). Approaches were via email, either directly, where contact details were available in the public sphere, or via the center from which they were identified where appropriate. Recruitment continued on a rolling basis until data saturation was reached. Eligible individuals were provided with a brief outline of the study and those interested in participating were invited to take part in an interview. Participants completed and returned a consent form prior to the interview or gave verbal recorded consent.

| Data collection
All interviews were conducted by a qualitative researcher (AR). The interviews were digitally audio-recorded and professionally transcribed verbatim. A preliminary topic guide was formulated in advance of the interviews informed by the research aims, which were iteratively refined to explore emerging themes. Additional prompts were developed for each participant group as required (Appendix S2).

| Analysis
Transcripts were analyzed by AR using directed analysis. 16 Findings from previous qualitative and review work, coupled with insights into Phase I of the project, were used to develop an initial coding framework. A flexible and iterative approach was used to continually develop and refine the coding frame, allowing for the emergence of novel themes. Additional codes were developed and included as the analysis progressed and the framework modified accordingly. 17 Transcripts were coded line by line and as the coding frame evolved, transcripts were recoded to ensure new themes were captured. The coding frame and sample codes were checked by DK and MC. Disagreements were resolved through discussion. 18 NVivo 11 19 was used for data management and to facilitate analysis.

| RESULTS
The interviews lasted for 30-60 minutes. For recruitment, participants were assigned to one of the four pre-defined stakeholder groups (trialists and chief investigators; those with lived experience of cancer; international experts in PRO design; and journal editors, funding panelists, regulatory agency representatives). However, several participants identified with or had features and experiences that resonated with more than one stakeholder group (Table 1).
Interviewees highlighted several barriers and facilitators to optimal PRO practice relating to study inception, PRO collection, analyses, data publication, and training and guidance ( Table 2).

| Study inception
During study inception, interviewees reported that PROs were often included in discussions at a late stage; or added as an "afterthought" to meet funding requirements for example, and were considered to be of lower importance in the hierarchy of trial outcomes. The relevance and priority of PRO endpoints in trial design were considered dependent upon the clinical characteristics of the cancer type and the nature of the intervention. Participants described how, where interventions were intended to prolong life, survival would be a key outcome, and PRO data may be of lesser importance to the research question. In cases where a range of curative interventions was available or in a palliative setting, PRO data had greater prominence. Other respondents commented on difficulties associated with selecting appropriate PROs in trials and perceived reluctance by some trial investigators to include PROs owing to concerns regarding their subjectivity or cost. Recommendations to address these issues included: the need to formulate a clear PRO rationale early in the trial design process which incorporated the patient perspective; identifying specific PRO aims and objectives; and ensuring involvement of PRO expertise during protocol development. Participants described how advances in cancer treatment supported more curative options, development of less toxic treatments, and greater awareness of the effects of living with consequences of radical interventions, leading to a growing awareness of PRO data and its place in cancer trials and clinical practice.

| PRO collection and analyses
Identified barriers centered around: perceived lack of standardized PRO administration compared to more "objective" clinical trial outcomes; concern around participant and staff burden associated with PRO completion; and a lack of communication with data collection staff regarding the importance of PRO data to the trial. Missing data were considered to be a significant challenge for analyses and concluding data. Subsequently, there was a view that the volume of missing data and its perceived poor quality could contribute to One of the real key components is a company understanding early in the development program why they are going to collect the data and if they are going to collect it, and this really impacts the overall data quality that you see later on. If you have not decided earlier, the company or drug developer, that you are going to collect PRO and then you only decide later on in the development program when you realize that maybe a payer or someone else might ask for that data, usually the quality of the data, the hypothesis-generating information and methodology is done quite poorly and it is quite difficult to conclude from that data [006, Trialist/Chief Investigator] PRO expertise in protocol development [The PRO advisory group] make recommendations and will sometimes provide some sample packs … We do try to get as much of that sort of stuff into the protocol as we can, so that it is written in stone as … part of the protocol and any deviation from that is, a deviation to the protocol … administration processes, that are too detailed for the protocol might go into a site training document or an appendix that is used as instructional material or a checklist for the site [010, International Expert]

Inclusion of patient perspective
Often the issues that researchers think are important to get out there, are not important to the patients. They are more interested in … quality of life measures that sometimes the researchers do not put such great store on. They are more bothered in whatever the primary end point was and it is very rare for a primary end point to be a quality of life. It is usually progression-free survival or overall survival [033, Lived Experience] (Continues)

New focus on gentler treatments
The treatment has advanced to a point where is not just a matter of, can we keep you alive, but, can we keep you alive and comfortable, can we improve your survival and improve your symptoms, then it starts becoming more important to actually be able to assess the symptoms as opposed to when we have drugs that just, universally made people feel God awful. [

Barriers Selection of PRO instruments
The metrics may not be as well established as say, as in cardiac output like that in the randomized trial so the metrics may not be as well established, there may be less agreement about how to measure things there might be more subjectivity and that can be harder to operationalize, even administration may be somewhat looser or less rigorous compared to some more biological outcomes that can be standardized and where timing can be adhered to more so I think just describing the outcomes might be a bit more challenging, the fact that the outcomes may be less well-

Study phase and themes Example quote and source identifier and primary stakeholder group
Engaging trial participants There has to be some responsibility obviously on the patient, it has to be a shared task so improved tech partnership working is what is so important that the patient feels a part of the team and actually your data is incredibly valuable to us because it will inform research and clinical practice you know as an ongoing issue and it may be years down the line before you may see the benefit but you will be contributing to this so you are part of our team

Ranking of PRO among outcomes
If the PRO were only an exploratory end point within the protocol sometimes they are not even mentioned so they might have been mentioned as, "These were end points," but then you do not actually find any of the results or data actually in the publication at all. [006, Trialist/Chief Investigator]

Perceived interest in PRO findings
If you look at any of the main clinical trial report articles there might be a small paragraph on the quality of life outcomes regardless of whether they are really good or just no difference between the treatments. Mainly because the interest in both is on survival, progression-free survival, response rate, things of that nature which, and toxicity which seems to take more precedence. [005, International Expert] T A B L E 2 (Continued) (Continues)

Study phase and themes Example quote and source identifier and primary stakeholder group
Perceived disinterest from high-impact journals If survival's not there … the "harder outcomes" are not there in your paper, you do struggle. We start when we try to send papers for publications as high an impact factor as we can go but we usually know that we are going to get rejected. We get rejected by Lancet Oncology, for example. It is a classic. You just get a little pat on the head but, "Oh, send this to a nursing journal." British Journal of Cancer, some of the big names in the cancer field do not want to publish this sort of work so we work our way down the list. [ underreporting due to possible damage to reputation and limiting future funding prospects. Interviewees also highlighted a perceived lack of clarity surrounding PRO analysis methods as a key barrier. Others felt that a lack of trial resources often resulted in the prioritization of the primary outcome at the expense of PRO data. Facilitators to optimal implementation of PRO within trials included: ensuring adequate PRO coverage in the protocol to help foster "buy-in" from trial staff; engaging research participants by communicating the importance of their PRO data; development of a priori plans to minimize avoidable missing PRO data by identifying poor PRO compliance in real-time, establishing statistical management of missing PRO data in advance, and pre-specifying PRO analyses; and the use of innovative data-capture technology to reduce burden and increase data quality.

| Data publication
Discussion focused upon academic publication rather than inclusion in regulatory submissions or Health Technology Assessment applications. Interviewees highlighted issues around selective reporting of PROs, either linked to the significance of the primary outcome or of the PRO itself, or due to the "lower ranking" of PRO results by trialists. There was also a perception that PRO findings were of little interest to journals unless particularly "novel" or "dramatic," and that the inclusion of PRO data alongside the primary outcome could be impeded by restrictive word count limits. Interviewees discussed solutions including the generation of a dissemination plan that includes PROs; encouraging Patient and Public Involvement (PPI) throughout the trial to foster accountability and promote complete reporting; and the availability of the option to include PRO data as supplementary files or appendices. When asked about perceived demand for PRO data from journals, participants who had successfully led and published research using PROs in high-impact journals described how well-written papers aimed at appropriate journals would get published. Participants, including those employed as journal editors, noted that journals are keen to publish high-quality PRO data, particularly those from large-scale RCTs, depending on the clinical area.

| Training and guidance
A recurring theme was the perceived difficulty of acquiring and providing PRO training in practice. Respondents felt this was due to the lack of awareness of training resources; possible expense; limited time; and particularly

Study phase and themes Example quote and source identifier and primary stakeholder group
Burden upon researchers I think, as a researcher and especially as a single-handed researcher, that the bureaucracy and the paperwork are overwhelming and have become increasingly so over the last ten years. It might have modest benefits in some areas but it certainly can hold people back. I think if there is guidance, especially if it is mandatory guidance and it is impossible to publish if you do not follow that guidance, then you have to really support researchers to access that sort of stuff, rather than putting burdens in their way. for generalists, the notion that PROs were one of a multitude of competing for potential training needs, or that PROs were not within their remit. However, the majority of interviewees felt there was a role for all institutions involved in regulatory, funding, educational, methodological, and journalistic activities throughout the research process to ensure that PRO training and guidance was available, signposted, and readily accessible to stakeholders, with accompanying measures to ensure adherence and compliance to best practice guidelines.

| Additional themes
There were two unanticipated themes resulting from the interviews (Appendices S3 and S4), drawn from all stakeholder groups. The first relates to the usefulness of PRO data in meeting the information needs of people during diagnosis, treatment, and when living with and beyond cancer (LWBC

| DISCUSSION
Existing research has drawn valuable insights into the challenges faced when administering PROs in trials 6 ; however, the current study is the first investigation of factors affecting PRO trial design, data collection and reporting, or patients' and clinicians' access to PRO data to inform decision making. Our findings highlight a number of potential factors contributing to observed suboptimal PRO protocol quality and reporting in cancer clinical trials: low prioritization of PROs; lack of training; late inclusion of PROs in trials; uncertainty related to analysis; and poor implementation (leading to missing data and interpretation issues). These span all phases of trial development, implementation, and dissemination and require the concerted and coordinated effort by the cancer research community to address them, for the benefit of all cancer patients. A particular focus should include addressing perceptions around the value of PROs and their position within the hierarchy of trial outcomes, and adoption of international guidance around best-practice PRO trial design, data capture, analysis, and reporting.
Missing data were a recurring theme and was posited as a reason for skepticism around the value of PROs. This is reiterated in the literature, where high rates of missing PRO data continue to be reported. 10 It is hoped that the widespread adoption of recent guidance around design and measurement selection, implementation, and reporting strategies aimed at reducing the instance and impact of missing PRO data will support more effective capture of PRO trial data in the future. 20 Recent international cancer trials have successfully collected PRO data on a large scale, while demonstrating the ability to do so with very little missing data, minimal resource costs, and negligible investigator burden. 21 PRO data are being successfully collected in challenging trial settings, for example, those performed across several centers and including highly vulnerable cancer populations. 22 The need to include patients in the development, application, evaluation, and interpretation of PROMs is well established, 23 to ensure their acceptability and relevance. 24 A number of interviewees also supported the a priori development of statistical analysis plans including methods to address missing PRO data. The imminent publication of guidelines arising from the Setting International Standards in Analyzing Patient-Reported Outcomes and Quality of Life Endpoints (SISAQOL) Data Consortium should help support trialists to implement this recommendation. 25 There were diverging views around the non-publication of PRO data. 8,12 While several researchers cited the experience of journals' general reluctance to publish findings, particularly in primary trial manuscripts, interviews with journal editors suggested the route to publication was influenced instead solely by the quality of the individual manuscript. It is difficult to reconcile these opposing viewpoints. Open access international guidelines are available via the Consolidated Standards of Reporting Trials (CONSORT) PRO Extension, 26 based on the methodological framework for guideline development proposed by the Enhancing the Quality and Transparency of Health Research (EQUATOR) Network. 27 Intended to support researchers in raising standards of PRO reporting and if the broader pool of journal editors share the views of our interviewees, this could increase the probability of publication. Recent evidence also suggests an association between improved PRO reporting quality and the quality of the trial protocol, 8 30 Increased interest in PROs is echoed in the work of regulators, such as the Food and Drug Administration (FDA) recent patient-focused drug development initiative, 31 and the European Medicines Agency (EMA) guidance on the use of PROMs in cancer trials. 5 Despite this, evidence suggests that the impact of PRO data is limited. 32,33 Issues relating to PRO-related training are identified as a key theme, despite the availability of PRO training resources, [34][35][36] demonstrating the need to enable and uphold greater uptake.
In summary, our interviewees suggested future cancer trials should include more comprehensive PRO trial design and protocol development involving PRO expertise and patient input, with a focus on standardized administration. They also emphasized the need to minimize the burden for patients and staff; prevent missing data; address missing data with appropriate analysis methods; develop a priori PRO analyses and dissemination plans; and train staff. The use of PPI throughout a trial life-cycle was identified as integral to ensuring PRO data generated through cancer clinical trials are relevant and accessible to people with cancer. Unfortunately, the study findings suggest that stakeholders perceive and observe barriers to the interpretation and dissemination of PRO results, and that key factors that arise during the design and data collection of cancer clinical trials compromise this process.

| Clinical implications
Our qualitative results suggest that the specific features of cancer and treatment affect whether PROs are placed lower in the trial outcome hierarchy as compared to survival outcomes. This appears to shape the expectations of those in the field, conceptualizing trials and determining research questions, perpetuating the relatively low position of PROs in the trial outcome hierarchy. However, the interview findings also suggest that the relative position of PROs in the cancer trial outcome hierarchy could be changing over time. Participants described the growing demand for PRO data due to increasing awareness around the impact of cancer on individual quality of life and the considerable burden associated with the acute, mediumterm, and late effects of treatment. Participants described witnessing relatives and friends undergoing treatment and the lasting long-term consequences and how this informed the type of information they would seek while considering treatments for themselves. This is echoed in studies outlining the research priorities of those with lived experience of cancer, 37 reiterating the need to integrate meaningful PPI when setting the research agenda. Early and consistent PPI may lead to a shift in emphasis from survival or cancer progression in isolation to also include the quality of survival; patient-centered PRO rationale, aims, and objectives; selection of meaningful PROs, thereby making subsequent findings potentially more impactful; enhancing communication with participants around trial PROs; maximization of trial feasibility, recruitment, and retention; and minimization of PRO burden, participant study drop-out, and missing data. 20,[38][39][40]

| Study limitations
The strength of this work lies in the use of rigorous methodology; the broad expertise of the interviewees; and the inclusion of numerous stakeholder groups. However, a limitation is that international recruitment efforts were focused on the PRO and cancer methodology experts rather than across all stakeholder groups. A further limitation is that the study was at risk of self-selection and social desirability bias, whereby participants participated due to a pre-existing interest in PROs, portray their behaviors positively due to the nature of this study, with increased awareness of methodological issues and need for training. Despite this, the divergence of opinions identified indicates that individuals were sampled with a wide range of views related to PROs. Another possible limitation is that through the inclusion of people with lived experienced of cancer, there may be a bias toward the views of those who have survived and are LWBC. Their perspectives are more likely to reflect issues that pertain to the long-term impacts of cancer treatments. Similar studies in the future may benefit from the inclusion of people affected by cancer more broadly, including carers, and those who are bereaved.

| CONCLUSIONS
Our aim was to explore with stakeholders the factors influencing suboptimal PRO protocol content, implementation, and reporting. These qualitative findings suggest that a lack of training, understanding about the value of PRO data (that results in low prioritization of PROs in outcome hierarchy), difficulties associated with the numerous ways to analyze PRO data, and the expertise required to this end can undermine their planning, collection, and reporting can undermine their planning, collection, and reporting.