Patients' experiences of engaging with electronic Patient Reported Outcome Measures (PROMs) after the completion of radiation therapy for breast cancer: a pilot service evaluation

Abstract Introduction Over 60 % of people who develop breast cancer will receive radiation therapy (RT) as part of their treatment. Side effects of RT may include inflammation, erythema, desquamation and fatigue. Electronic Patient Reported Outcomes Measures (ePROMs) enable patients to report side effects prior to their scheduled post‐RT appointment. This pilot service evaluation aims to explore patients' perceptions regarding the value of the ePROM system, ease of its use and barriers to using the system, after breast irradiation. Methods From July–November 2021, evaluation surveys were posted to 100 people who had received RT to their breast to explore their experience of using the ePROM. Ethical approval was obtained through Ulster University and the Western Health and Social Care Trust (WHSCT), Northern Ireland. Results Fifty‐two people responded to the survey, of which 27 respondents indicated that they had accessed the ePROM. Despite few participants experiencing significant side effects, the majority of participants recommended the ePROM indicating that it was an important source of support. Those who experienced significant side effects found the system to be prompt and effective. Barriers to accessing the ePROM included technical issues with the link, concerns about confidentiality and forgetting to access the link. Access to the ePROM increased with higher education levels. Conclusions This pilot service evaluation demonstrated that ePROMs are valued by patients and can provide rapid real‐time access to support, offering individual care and reassurance. For patients with longer RT schedules (>10 fractions), the introduction of ePROMs during RT was viewed favourably by participants. All patients may benefit from the option of receiving ePROMs post‐RT.


Introduction
Approximately 63% of patients diagnosed with breast cancer will receive radiation therapy (RT) as part of their primary cancer treatment. 1Over the last decade, 40 Gy in 15 fractions became the standard dose and fractionation for RT to the breast in early stage breast cancer. 2,3owever, in March-April 2020, the Institute of Cancer Research (ICR) recommended the use of 26 Gy in five fractions for all adjuvant treatment of early breast cancer based on findings from the Fast Forward trial. 2,4As nodal irradiation was not included in this trial, 40 Gy in 15 fractions remains the recommended standard for patients within this group. 2Selected patients may also receive a 'boost' to the initial tumour site resulting in up to five additional treatments to these standard schedules. 2While RT techniques have improved dramatically over the last decade, patients still experience localised side effects as a result of irradiating surrounding healthy tissue. 5Early side-effects of breast irradiation can appear within days to weeks and include inflammation, erythema and desquamation of the skin. 3As most breast irradiation is now delivered over 5 days, early side effects are less likely to be managed during RT treatment delivery.7][8][9] In the North West Cancer Centre (NWCC), one of two cancer centres in Northern Ireland, patients who receive breast irradiation are routinely reviewed 12 weeks post-RT.However, an audit conducted by NWCC in 2019, found that when these patients were reviewed 12 weeks post-RT, some patients reported higher-grade toxicities compared to those recorded during their final week of treatment.Despite this finding, the audit revealed that contact from patients prior to their 12-week appointment was rare, with patients mostly waiting until their appointment to discuss concerns.Hence, it was concluded that it may be necessary to assess some patients after RT for breast cancer, prior to the 12-week follow-up appointment, but without reverting to shorter follow-up times which is resource intensive and unnecessary for many patients. 10o address this concern, NWCC explored the possibility of implementing electronic Patient Reported Outcome Measures (PROMs).2][13] Numerous studies support the introduction of PROMs and suggest that they can increase communication between patients and clinicians leading to better detection of problems 14 and improved understanding of side effects. 15,16Collated findings by the National Institute for Health and Care Excellence (NICE) indicate that patients want to be active participants in their own healthcare 17 and the introduction of PROMs provides an outlet to support self-directed care.][20] Although PROMs are a useful tool when combined with routine practice, there are few studies that have explored PROMs as a standalone method of self-directed care. 21In addition, the majority of studies focus on the use of PROMs while the patient is receiving treatment, with few exploring its value for post-treatment care. 15,16Studies which have explored PROM use for post-treatment care 15,16 have not specifically looked at radiotherapy side-effects posttreatment and the timing of the PROM completion is unclear.Additionally, these studies required the patients to complete the PROM in the clinical setting and therefore have not tested the success of the ePROM when completed remotely. 15,16Subsequently, additional research is needed to obtain the patient perspective regarding the use of remote ePROMs for self-directed care post-RT. 15 address this gap in post-treatment care, an electronic PROM (ePROM) system, Penguin (Cievert Ltd, Gateshead, UK) was introduced into NWCC for digital self-directed care, to encourage reporting of acute toxicities in real time, in alignment with the NHS Long-Term Plan for access to digitised care. 22Due to the high number of patients who receive breast irradiation compared to other sites, this group of patients was selected for a pilot service evaluation of the system.The primary aim of this evaluation was to establish patients' perspectives regarding the value of the ePROM system, ease of use of the system and potential barriers to using the system, post-RT.

Study design
This pilot service evaluation involved a single cancer centre, NWCC, and utilised an anonymous postal evaluation survey to capture data regarding patients' experiences of accessing and using the ePROM.A survey was chosen as the information-gathering method, as it allows for data to be collected from a large, targeted population. 23Ethical approval for this evaluation was obtained through the Nursing and Health Research Ethics Filter Committee at Ulster University, as well as the Western Health and Social Care Trust (WHSCT).
Penguin, the ePROM software, was supplied as part of an ongoing departmental contract with Cievert at no additional charge and in line with local change control processes.

Participant recruitment
Participants were eligible to be enrolled into the study if they were a minimum age of 18 years old and completed radical RT for breast cancer in NWCC between July 2021 and November 2021.Radical RT treatment is defined as treatment given to cure or improve survival substantially. 24ll radical breast cancer patients who consented to RT from July 2021 were offered the opportunity to receive ePROM communications via e-mail to enable them to self-report side effects electronically after completion of RT.This information was provided to them by the lead author, who is also the radiation therapist primarily involved in the implementation of this system.Information regarding the ePROM was provided to them in their last week of treatment, where they also received information regarding the pilot service evaluation survey, at this time.Patients who expressed that they would like to use the ePROM post-RT were entered into Penguin, which generates an electronic questionnaire when their diagnosis is added.The ePROM was manually scheduled to be sent once per week between week 6 and week 12 after their last RT treatment.To access the link to the ePROM, participants were required to enter their postal code and date of birth.All participants were supplied with end-of-treatment literature and continued to have a face-to-face or telephone appointments scheduled at approximately 12 weeks post-RT (depending on their clinician's availability).The ePROM used a scoring system adapted from the Radiation Therapy Oncology Group (RTOG) and Common Terminology Criteria for Adverse Effects (CTCAE) grading system, 25 to improve patient comprehension.This adapted tool was already widely used within the oncology department for reporting of side-effects.For each question, clinicians agreed on threshold grades for each side-effect, beyond which responses were flagged for their attention (see Supporting Information, Appendix S1).If the participant selected these grades, an email alert was automatically sent to the designated team.The email was re-sent regularly to clinicians until the flag was actioned.Participants who returned their ePROM without significant issues did not need any intervention until their scheduled 12-week followup appointment.Participants could choose to complete as many of the ePROMs as they wished between week 6 and week 12 post-RT; there was no minimum number of completions required.Participants did not complete any ePROMs during their RT in this study.

Participant evaluation postal survey
Once the participants' 12-week RT follow-up date had passed, a Participant Information Sheet (PIS), including the evaluation survey (see Supporting Information, Appendix S2) was supplied with a stamped addressed envelope to elicit feedback regarding their experience of using the ePROM prior to their 12-week appointment.

Sample and sampling
Approximately 1450 women in Northern Ireland are diagnosed with breast cancer every year. 26As approximately 63% of these patients will be offered RT, 1 then approximately 914 women would be eligible for the study over a period of 1 year.As this pilot service evaluation was conducted over a period of less than 6 months and at least half of all patients with breast cancer would be treated in the other Northern Ireland cancer centre (Belfast City Hospital), then approximately 229 women may have been eligible to participate.
It is generally accepted that pilot studies should be large enough to provide useful information but do not need to adhere to the narrow margins of error and confidence levels associated with full studies. 27onsequently, the researchers aligned to Schober et al.'s 28 methodology, aiming for a margin of error of 10-15% on a 95% confidence level.Using the Raosoft sample size calculator, 29 it was, therefore, determined that 37-68 participants would need to be successfully recruited.
Based on an anticipated response rate of approximately 50%, 30 the first 100 participants who had access to the weekly ePROM, were sent an anonymous survey.

Data analysis
Descriptive statistics were used to analyse data for single categorical variables and included frequencies and percentages.Inferential statistics were used to explore whether participant demographic variables influenced participants' ability to access the ePROM.Age and educational level were assessed for normality using the Shapiro-Wilk statistic and visualisation of the graphical data 31,32 and assessed using the Mann-Whitney U test or independent t-test. 33Employment status and residential location were assessed using the chi-square test. 30,34An alpha value of 0.05 was set for statistical significance for all tests.
Qualitative data were analysed using thematic analysis where open, axial and selective coding was applied to the data as described by Williams & Moser. 35In the open coding phase, words and phrases were assimilated iteratively into codes by one researcher for each question with a qualitative component.The second researcher independently completed this first step strengthening the rigour of the data analysis. 36Open codes were then discussed and refined between the two researchers through reflection and reflexivity in a non-linear process until agreement of codes was reached, further increasing the rigour of the data analysis. 36Axial and selective coding were performed as a collaborative task between the two researchers where emergent themes were discussed and established.Due to the low volume of qualitative data, this process was relatively uncomplicated compared to the complexity inherent in interviews or focus groups.

Results
Of the 100 participants who were sent a feedback survey, 52 participants returned their completed survey detailing their experience of using the ePROM.

Demographic information
Participants most frequently indicated that they were employed but on leave (n = 19; 36.5%).Twenty-four participants (46.2%) indicated having an education level higher than secondary school.Participants most frequently reported being between 46 and 55 years old (n = 18; 34.6%) with 21 participants (40.4%) indicating that they were older than 55 years old.An equal number of participants reported living in either the town or countryside (n = 19; 36.5%).
Participants most frequently indicated that they had received 10 fractions of RT (n = 19; 36.5%),followed closely by 5 fractions (n = 18; 34.6%).Fifteen participants (28.8%) indicated that they received ≥15 fractions of RT.All participants identified that they had surgery and a further 25 participants (48.1%) identified that they received chemotherapy in addition to surgery.Four participants (7.7%) indicated that they required wound dressing and four participants (7.7%) indicated that they required emotional support during RT.Nine participants (17.3%) identified that they required physiotherapy for complications related to their cancer, during RT treatment.See Table 1 for full demographic details.

Access to and completion of the ePROM
Twenty-seven participants (51.9%) identified that they were able to access the ePROM.Reasons reported for not accessing the ePROM included issues with the link, issues with access and not receiving the email.Some comments included: Could not access this because it told me each time that it had expired" (P21) "Unfortunately the email was sent as a spam and I could not access the email.I tried several things but still could not open it.

(P1)
Twenty-four participants (46.2%) identified that they completed an ePROM.Common reasons reported for not completing an ePROM included poor internet connection and participant's own forgetfulness.Comments included: Poor internet.

(P16)
Five participants (9.6%) identified that their ePROM required a response from the RT team.See Table 2.
Four of the five participants who required a response from the RT team, identified that they were contacted by a member of the team; the remaining one participant did not respond to this question.These four participants indicated that they were contacted within 1-3 days of submission of the ePROM.Three of the four participants who required a response, commented that the quality of care that they subsequently received was adequate, good and very good respectively; the other 2 participants did not complete this question.See Table 3 for full details.
Fifteen participants (28.8%) identified that they felt that the ePROM aided the review process, with comments such as: It put my mind to rest.

(P6)
It informed the questions asked in the telephone follow up appointment.

(P26)
Yes because she already had knowledge of my progress.

(P35)
Other participants' comments indicated their uncertainty regarding the value of the ePROM.Three of the four participants that answered no, commented: Not sure that this happened.

(P10)
My response to the ePROM wasn't specifically mentioned at my 12 week review.

Section B
The majority of participants (n = 36; 69.2%) identified that they felt the ePROM was sufficiently explained, with only three participants (5.8%) reporting that it was not sufficiently explained.Of the three participants who felt that the ePROM was not sufficiently explained, two participants felt this was due to their own memory of the information session.See Table 4. Comments from the latter included: If it was explained, I don't remember.

(P41)
. . .because I was stressed, I did not understand what I was being told or forgot it. . .

(P6)
Participants most frequently (n = 30; 57.7%) indicated that they were not apprehensive about using the ePROM system, with only six participants (11.5%) reporting apprehension.Comments from the latter included: I thought it was a scam at first, but then emailed the WHSCT and found out that it was legitimate.

(P14)
The majority of participants (n = 32; 61.5%) identified that they would recommend using an ePROM to report side effects post-RT.See Table 4 for full details.
One participant commented: If rolled out I would say it would allow clinicians to deal with issues quicker and time efficient.

(P40)
Only four participants (7.7%) reported that they would not recommend it, with comments including: Prefer speaking to a professional directly.(P4) I would prefer to talk to a clinical on the phone or in person.

(P23)
Two of the four participants commented that they would not recommend it because the link did not work (P28, P37).
The majority of participants (n = 31; 59.6%) agreed that the ePROM should also be sent out while they were receiving RT treatment.Of the 15 participants (28.8%) who received ≥15 RT treatments, 12 of these 15 participants agreed with this statement with the remaining three participants not answering this question.The six participants (11.5%) who did not think this should occur all received either five or 10 RT treatments.Two of the six participants commented that this was because they had no side effects during the treatment.

Comments included:
because you need to finish the treatment and see what the following week is like.

(P19)
The majority of participants (n = 36; 69.2%) indicated that they felt that they received enough information and support to help manage post-RT care.Comments included: I was given a checklist to monitor my health during and after radiotherapy that was very helpful.

(P12)
The information and support I received prepared me for the side effects and timeline associated with them, especially in the first month after radiotherapy.

(P26)
Of the three participants (5.8%) who disagreed, only one comment was provided: No to be honest they were friendly every day during treatment but when it's over you're on your own.(P29) Eleven participants (21.1%) added additional comments, a cross-section of which is shown in Table 5.

Relationship between participant demographics (Section A) and access to the ePROM (Section B)
Out of the 52 participants who completed the evaluation survey, one participant did not indicate whether they assessed the ePROM and therefore the data analysis in this section is based on the responses of 51 participants.
To assess if education level had an impact on whether participants accessed the ePROMs, education level was allocated a score of 1 to 5 for increasing participant education level from completion of primary school (score of 1) to receipt of a postgraduate degree (score of 5).Seven participants either did not indicate their level of education or chose 'other' so were removed from the data analysis.Therefore 44 participants were included in this analysis.The researchers explored the alternative hypothesis which was that those with higher education levels would access the ePROM more frequently.
To assess if age had an impact on whether participants accessed the ePROM, age was also allocated a score of 1 to 8 with the lowest age range being allocated a score of 1 and the highest age range being allocated a score of 8. Three participants did not complete this question resulting in 48 participants being included in the data analysis.The alternative hypothesis proposed that accessing the ePROM would increase with decreasing age.
Due to the visualisation of the graphical data and a Shapiro-Wilk statistic of <0.05 in both data sets, the data was considered not to be normally distributed for educational level or age.Consequently, a one-tailed nonparametric Mann-Whitney U test was utilised to analyse this data.This test demonstrated that there was a significant difference (Mean rank score 18.68 vs. 25.69;U = 163.5,P = 0.029) between the education level of participants who accessed the ePROM compared to those who did not access the ePROM; those who accessed the ePROM having higher levels of education.A one-tailed independent t-test also confirmed the finding with a Pvalue of 0.018 (t = À2.253;df = 41.92) and mean scores of 2.65 versus 3.38 for those who did not access the ePROM and who did access the ePROM respectively.Therefore the alternative hypothesis was accepted; access to the ePROM increases with increasing educational level.
Using a one-tailed non-parametric Mann-Whitney U test, no significant difference was found in age between those who accessed the ePROM and those who did not (U = 286, P = 0.974); the mean rank scores being just over 24 in both groups.Therefore, the null hypothesis was accepted and it was concluded that there was no difference between the groups.
Due to the low number of participants who indicated that they lived in the city or in rural areas, the participants were further categorised as either living in highly populated areas (city or town: n = 24) or lower populated areas (village, countryside or rural: n = 24).Three participants did not indicate where they lived resulting in analysis based on 48 participant responses.A 2 9 2 matrix Chi-square test was used to assess the relationship between where participants lived and whether this impacted their access to the ePROM.This test indicated that there was no significant association between accessing the ePROM and residential location X 2 (1, n = 48) = 0.751, P = 0.386.
Similarly, employment status was re-categorised into 2 categories; employed (n = 26) and other (including retired) (n = 13).A 2 9 2 matrix Chi square test was used to access the relationship between employment status and whether this impacted participant access to the ePROM.This test indicated that there was no association between accessing the ePROM and employment status X 2 (1, n = 48) = 1.326,P = 0.25).
A statistician was consulted to confirm the findings in this section.

Discussion
Participants' perspectives regarding the value of the ePROM system The majority of participants who contributed to this pilot service evaluation, recommended that the ePROM should be integrated into post-RT care for reporting of sideeffects.They indicated that the ePROM assisted them in rapidly reporting arising problems and felt reassured by its ability to accurately inform their RT team of their "Maybe a little clearer who the emails are from or maybe write out the email address so that we recognise who it is from exactly.Being a traumatic time we don't always hear or remember what is said."(P6) "I wasn't clear on whether I should complete the form multiple times (this may have been said to me, I can't recall).I had no major issues so I didn't feel the need to report my progress when I felt well from an early stage."(P12) "For serious and painful side effects I would probably prefer telephone/ face to face consultation but I did not experience these and was prepared and equipped to deal with pinkish hue, heat, slight itchiness explained during face to face consultation" (P26) "I think if rolled out it would be very beneficial as not all patients are the same and their treatment may incur questions that are not the "norm" so if clinician aware then it does save time."(P40) Another common reason for not completing the ePROM was concern regarding confidentiality and/or apprehension to open a link from an unknown sender.To address this issue, ePROM suppliers should ensure inclusion of the RT department name into the subject line of the email and/ or amend the body of the email to include hospital specific information.Given that scam e-mails have been determined to account for >70% of all e-mails sent, 44 participation is likely to be positively impacted by clearly identifiable e-mail addresses, names and titles.
A small number of participants described forgetting to complete the ePROM or having insufficiently absorbed information regarding the ePROM completion process.With almost half of the participants in the study having also completed chemotherapy, this group is vulnerable to fatigue and 'chemo brain' symptoms including impact to memory, attention and processing speed. 45Some patients also present for RT with high levels of anxiety. 14,39All of these factors can impact retention of information, emphasising the importance of explaining ePROM information to patients more than once; 1 week prior to the end of their RT treatment and again on their last day of RT.
No association was found between participants' ability to access the ePROM and individual characteristics including employment status, age and residential location.Previous publications which have found lower access to digital services like email based on patients' increasing age found that people over 75 years of age are most impacted. 46,47iven that few participants in this study were over 75 years old, future study of this participant age group would be useful and add to these findings.While globally internet access in more rural areas has been a barrier to healthcare, 48 no differences in internet access were noted between participants living in highly populated areas compared to those in lower populated areas.
An association was found between education level and ability to access the ePROM with those with higher levels of education accessing the ePROM most frequently.Health literacy is frequently demonstrated to be correlated to education level 49 and lower health literacy can negatively impact access to ePROMs. 50Computer literacy is also an aspect which increases with higher education level. 51This findings further emphasises the need for allocating sufficient time with patients during ePROM information sessions to ensure that patients feel confident in understanding both how to access the ePROM and what is being asked in the ePROM.

Strengths and limitations
Given that the average patient survey response rate is between 35 and 50%, 29,52 it can be concluded that the response rate (52%), shows acceptable engagement with and delivery of this pilot service evaluation.However, there are limitations of this study which should be addressed.One limitation is that the ePROM was only recently implemented in this single centre and therefore teething issues related to the technology, hampered data collection.Additionally, a limited number of participants reported significant side-effects and therefore it is difficult to accurately establish the efficiency of this system.Very few participants reported that they utilised other services while receiving RT, so it is difficult to draw conclusions on their perception of access to Allied Health Professionals for services like wound dressing and emotional support.

Conclusion
This pilot study demonstrated that incorporation of the ePROM into post-RT care is a feasible option and can enhance care after breast irradiation.Post-RT ePROMs have the potential to offer patients rapid access to care and optimise the distribution of follow-up appointments for patients based on their individually assessed needs.Breast irradiation schedules vary considerably so the timing of ePROMs post-RT should be optimised to capture and manage side-effects for each patient as sideeffects arise in real time.Post-RT, ePROMs offer reassurance and support to patients while waiting for their first post-RT follow-up appointment.Consideration should be given to offering access to ePROMs during RT to patients with longer RT schedules.ePROMs during and after RT treatment could help the management of not only physical side-effects, but also patients' psychological wellbeing.
This study highlights the logistics/communication issues which can arise with administration of ePROMs post-RT where patients are no longer seeing healthcare professionals for guidance.Consistent with previous findings, 53 this study demonstrated that in the early stages of implementation, refinement of the ePROM process may be necessary through frequent service evaluation to ensure that the needs of the targeted population are being met.
While the feasibility of incorporating ePROMs after cancer treatment has recently been explored in other studies, 38,[53][54][55] few publications have explored the use of ePROMs with patients post-RT for breast cancer.Larger multicentre studies are needed to fully understand the potential benefits of ePROMs post-RT for breast RT and other RT treatment sites.Such multicentre studies could be used for predicting future outcomes and the impact of this system on resources 21 for a range of cancer sites.

ª
2023 The Authors.Journal of Medical Radiation Sciences published by John Wiley & Sons Australia, Ltd on behalf of Australian Society of Medical Imaging and Radiation Therapy and New Zealand Institute of Medical Radiation Technology.

Table 1 .
Demographics of participants.
ª 2023 The Authors.Journal of Medical Radiation Sciences published by John Wiley & Sons Australia, Ltd on behalf of Australian Society of Medical Imaging and Radiation Therapy and New Zealand Institute of Medical Radiation Technology.

Table 2 .
Participants' access to and use of the ePROM.

Table 3 .
Participants' report on the response that they received from the radiotherapy team through engagement with the ePROM system.

Table 4 .
Participants' perceptions of information received regarding the ePROM and the value of the ePROM.

Table 5 .
Comments from participants regarding use of the ePROM.