How to implement routine electronic patient‐reported outcome monitoring in oncology rehabilitation

Abstract Background Implemenation of patient‐reported outcomes (PRO) like quality of life can add the patient's perspective to traditional clinical outcomes of cancer rehabilitation in a structured and standardized way. Aim To present useful steps for a successful implementation of routine electronic patient‐reported outcomes (ePRO) monitoring. The presented steps are exemplified by describing the procedure applied in an Austrian inpatient cancer rehabilitation centre. Methods The suggested implementation steps are presented based on the structure of the replicating effective programmes framework, which was used for developing a pragmatic implementation strategy. Results We scheduled alternating trainings and process evaluations for audit and enhancement of procedures. In this way, the ePRO participation rates could be improved. Stakeholder involvement led to initiatives that included the integration of ePRO data into the medical discharge letter and the implementation of follow‐up assessments. Discussion Tailored changes in assessment procedures enabled the successful implementation of ePRO, which has been shown to be feasible before and after cancer rehabilitation. The continuous involvement of stakeholders paved the way for further projects initiated by medical staff as users themselves (inclusion of PRO data in the discharge letter and a comprehensive ePRO follow‐up using a versatile online patient portal).

eg, patients' health-related quality of life (QOL), disease-or treatment-related symptoms, functioning and well-being. Through PRO, the patient's subjective experience is quantified and can complement physician-reported data in a standardised manner, enabling gaining a more accurate picture of the patient's health status [4][5][6] and bringing up aspects (eg, sexual issues), which might otherwise go unnoticed or be underestimated by clinicians. [7][8][9][10][11] Established PRO instruments have undergone an elaborate scientific background and solid psychometric testing and must be interpreted in terms of the meaning and clinical relevance of absolute scores or changes in scores in order to be medically instructive.
Over the whole course of cancer treatment, in which rehabilitation and follow-up play an increasingly important role, 12 PRO can be used as a standardised screening to promote efficient use of time in medical consultations, improve the communication between patients and their therapists, ensure treatment continuity and facilitate patient-centred interventions as well as participatory decision-making. 13 Hence, similar to their benefits for routine active anti-cancer care, PROs can also be used in rehabilitation to screen patients for special care needs, tailor interventions to screening results, evaluate the treatment progress and collect data for performance measurement, which can as well feed into Big Data analyses. 14,15 It is reported that especially vulnerable cancer survivors (older age, lower education, depressive symptoms) are at a higher risk of dropping out longitudinal follow-up PRO assessments post-active treatment. 16 This might be associated with major problems in at least two respects: First, these patients might have less access to the care they actually need (especially if PRO data are used to support follow-up care) and second, this creates a considerable positive bias in data possibly used for Big Data analyses. Despite the many similarities in the application of PRO in clinical routine and in oncological rehabilitation, special attention must be paid, for example, to ensure that the choice of assessment instruments and the timing of the assessments are appropriately adapted to the purpose and setting of data collection.
Traditionally, paper-pencil questionnaires are used for PRO data collection; this is generally associated with relatively laborious working procedures (especially regarding data processing and storage), prone to data input errors and requiring vast human resources.
Electronic PRO (ePRO) monitoring can collect data in a cost-effective and secure way and, depending on the tolerance of missing values within the software, data quality is generally higher than that of traditional paper-pencil assessments. 17,18 The data is immediately available digitally, automatically scored and graphically presented and may be linked to the patients' electronic health record. 19 Electronic data processing, therefore, allows the use of ePRO data straight after questionnaire completion during the medical encounter. ePRO monitoring and specialised assessment software are increasingly incorporated into oncology routine, most frequently in settings including patients before treatment or undergoing active treatment, 20-23 but targeting cancer survivors after treatment as well. 24 With the intention to help overcoming common obstacles to implementing routine (e)PRO monitoring, consulting resources What's known • Patients are the experts for reporting their subjective health status and quality of life. The gold standard to assess such data is the use of validated patient-reported outcome (PRO) questionnaires.
• Due to the improved detection and treatment of cancer, the number of people dealing with an oncological disease, surviving it and needing rehabilitation is increasing.

What's new
• Implementing routine electronic quality of life assessments in oncology inpatient rehabilitation is feasible.
• An inclusive approach should involve all stakeholders to achieve the best result.
• Repeated evaluation and individual adjustments of the workflow support the adoption of new procedures.
F I G U R E 1 Components of the replicating effective programmes (REP) framework including actions taken for implementing ePRO provide useful information and generic guidance. 25,26 To the best of our knowledge, the literature on the use of ePRO monitoring in cancer rehabilitation is, however, still scarce, especially on the standardisation of implementation procedures. This paper presents useful steps for a successful implementation of routine ePRO monitoring in an oncological rehabilitation centre and explains by way of example how these steps were carried out in this particular case. The structure of the paper follows the four phases of the replicating effective programmes (REP) 27 (see Figures 1 and 3).

| PRE-COND ITI ON
The oncology rehabilitation centre was opened in January 2014, provides 120 inpatient beds and offers a two to three-week multidisciplinary inpatient rehabilitation programme for adult cancer patients (including various diagnoses) with a Karnofskyx index of >70.

| Think about stakeholders and an implementation strategy
The medical director of the rehabilitation centre approached the software distributors to obtain their expertise regarding the following topics: developing an ePRO implementation strategy, providing the necessary software (including maintenance and IT support), data management and analysis. Besides communications via phone and e-mail, six months before the opening of the rehabilitation centre an on-site meeting of stakeholders (medical director, institution representatives, directors of therapy, administration officers, front desk staff, psycho-oncologists and the software distributors) was held in order to develop a pragmatic implementation strategy on the basis of REP. 27 Figure 1 provides an overview of the single process steps for each of the four REP-phases.

| Set up general procedures of the ePRO monitoring
Patients' PRO data were assessed twice: before admission to rehabilitation and up to three days before discharge during their psycho-oncology closing meeting. Front desk staff responsible for ePRO monitoring was trained within another facility of the rehabilitation provider following their established working routine, which was adopted in exactly the same manner. For this reason, the same PRO instruments were used as in the other institution.
Furthermore, to keep procedures same way, PRO-assessment was initially implemented as paper-pencil assessment (patients received PRO instruments by mail, including a return envelope) and changed to ePRO after a six-month adaptation phase in June 2014 (patients received an information sheet providing a unique user name and initial password for online ePRO access). Irrespective of data collection mode, patients received a reminder phone call by the front desk staff, if they had not provided their data 10 days before admission.

| Define outcome parameters for successful ePRO implementation
Evaluating procedures is key to in order to recognise whether they work under real conditions, as they should. It is, therefore, important to define parameters that indicate whether certain goals have been achieved. To evaluate the success of ePRO implementation, the following outcome parameters were determined by the on-site stakeholders: 1. feasibility of ePRO assessments (ie, how practicable is ePRO in this given institution), operationalised by feedback prompted by users (medical and administrative personnel), feedback regarding the effect of applied adjustments of procedures; 2. patients' acceptance (ie, do patients complete ePRO), operationalised by patients' response rates (≥70%; in literature, patients' response rates to mailed or electronically distributed PRO ranges between 31% and 65% [28][29][30][31] ) and unstructured collection of feedback prompted by patients; 3. professional users' acceptance (ie, does the centre personnel, in particular medical and administrative personnel, engage in ePRO), operationalised by training participation rates (≥70%, corresponding to patients' response rates), unstructured collection of professional users' feedback.

| IMPLEMENTATION
The implementation and evaluation process consisted of each three partially overlapping user training sessions and evaluations. Please refer to Figure 3 for a flow chart of the (e)PRO and CHES implementation process.

| Check whether ePRO feeds into routine care as intended
Based on ePRO-scores, the default frequency of four one-onone psycho-oncology sessions was increased to up to eight sessions. Neither administrative staff nor psycho-oncologists reported any problems using the recommendations based on the automated algorithm or individually adjusted treatment plans to be inadequate. Approached staff members and stakeholders showed great interest in the provided trainings and evaluations with very high participation rates (at least 80%, only very few nonparticipants resulting from illness or vacation), even though attendance was voluntary.

| Evaluate whether everything works as planned
During a two-day participating observation of a software company member in October 2014, practical help in software handling F I G U R E 3 Flow-chart of (e)PRO implementation at the rehabilitation centre was provided. The staff's feedback showed that the response rates were rather low (22%) and that they did not feel well prepared for ePRO-related critical inquiries of patients. To improve response rates, the cover letter to patients was refined, presenting ePRO as a substantial part of the routine rehabilitation procedure. Furthermore, phone support for front desk stuff was provided between October 2014 and December 2014. By the end of December 2014, patients' PRO response rates had increased from 22% to 83%.

Many patients gave a negative feedback about the question-
naire SSS-PSD being intrusive and not appropriate. As the adequacy of this measure with regard to content is disputable in a rehabilitation context, it was excluded from the ePRO assessment.

| Improve and re-evaluate procedures accordingly
In April 2015, a stakeholder (medical director, institution representatives, directors of therapy, administration officers, front desk staff, psycho-oncologists and the software distributors) update meeting took place, discussing the current status of the ePRO assessment procedure. Overall, no profound problems, regarding neither IT nor patient-related aspects, had arisen since the first evaluation.
Response rates of patients in ePRO assessments at the beginning of rehabilitation could be further increased to 98%. Since then, participation rates have settled at around 90%.

| Repeat as often as necessary
In September 2015, a further stakeholder meeting was dedicated to the discussion of how ePRO assessments can be successfully extended into clinical routine, providing patients' self-report data to all health care professionals for their individual use during patient encounter. In addition, the meeting included in-depth considerations about a feasible ePRO follow-up procedure.

| Check whether your outcome parameters for successful ePRO-implementation have been met
The predefined outcome parameters for successful implementation support the feasibility and acceptability of routine ePRO assessments:

| Think ahead
The existing IT infrastructure of the rehabilitation centre for the collection and use of ePRO enables the conduct of, eg, follow-up studies. A 1-year follow-up procedure (with assessments at 3, 6 and 12-month post-rehabilitation) was implemented in early January 2017 and will contribute to the long-term investigation of rehabilitation effects, which play an important role in terms of rehabilitation services' sustainability (eg, long-term benefit for patients, cost-effectiveness of rehabilitation). Reluctance to conduct PRO assessments continuously often relies on the fact that patients do not see any personal benefit in them and they get tired of just filling in questionnaires. Having access to their personal PRO scores and to tailored self-help advice, however, involves the patients in their care, provides feedback and can thus generate individual benefit and a greater willingness to participate. This very strategy is used to increase the attractiveness of follow-up ePRO assessments to patients and to prevent a common decrease in response rates during follow-up assessments after initially high inclusion rates. 37 The ongoing collection of follow-up data will accumulate a large pool of PRO data, which might be of interest to data mining or analytical reporting in the future.

| D ISCUSS I ON
We presented useful steps for achieving a successful implementation of a routine ePRO assessment and exemplified them by describing the process of such an implementation at an oncological inpatient rehabilitation centre. Data on patients' QOL and psychological distress before and after rehabilitation are reported elsewhere. 38,39 As suggested by Snyder et al, 26 the time structure of oncology rehabilitation offers practicable possibilities to smoothly incorporate ePRO in the given schedule. The fact that the assessment of ePRO has been included in the workflow of the rehabilitation centre from the very start has avoided any changes or disruptions as a result of (e)PRO assessment. Web-based ePRO assessments showed to be a feasible method for electronic data collection in cancer rehabilitation patients, which might be even preferred by patients and provide better data quality than paper-pencil assessments. 37,40

| Learn your lessons
Though already putting a lot of effort into the pre-condition and pre-implementation phases, we can report on some important les-  42,43 The initially low response rates suggested that many patients did not recognise any individual gain in (e)PRO assessments and seemed to perceive it as unnecessary add-on to regular rehabilitation procedures. Performing a detailed stakeholder analysis would have prevented the one-sided emphasis on the centre personnel by including patients and patient representatives on the stakeholder list. This could have prevented the aforementioned shortcoming by instantly developing a more adequate patient information sheet.
The second evaluation made obvious that the attitude of those staff members who had contact with patients posing critical questions about ePRO assessments plays an important role in their interaction with inquiring patients. 44 It is possible that the initial training before ePRO implementation lacked sufficient institution-specific educational content, as one major criticism of front desk staff was that their unfamiliarity with the concept, use and benefits of PRO as well as IT-related questions often made them feel uncomfortable in ePRO-related patient contact. An in-depth stakeholder analysis could also have been helpful in this respect by identifying the specific information needs at an early stage. The approach to combine several steps of ePRO assessment procedure evaluation with concerted training appears to have been a good way to enhance user involvement, which in turn seems to have positively influenced the interaction with patients, the collection of ePRO data and response rates. These observations go in line with other studies that report well informed, supportive staff and motivated patients (in a sense of perceiving the completion of ePRO to be relevant both for them personally as well as for their health care providers, eg, by triggering interventions) to be beneficial for successful ePRO assessments. [45][46][47] A summary of these most important lessons learned is presented in Figure 4 and can serve as a checklist of easily avoidable pitfalls.

| Conclusion
The implementation of ePRO prior to and after inpatient cancer rehabilitation was successful and feasible. This was achieved by fol-