Evaluation of a remote monitoring app in head and neck cancer follow‐up care

Abstract Background A remote monitoring app was developed for head and neck cancer (HNC) follow‐up during the SARS‐CoV‐2 pandemic. This mixed‐methods study provides insight in the usability and patients' experiences with the app to develop recommendations for future use. Methods Patients were invited to participate if they were treated for HNC, used the app at least once and were in clinical follow‐up. A subset was selected for semi‐structured interviews through purposive sampling considering gender and age. This study was conducted between September 2021–May 2022 at a Dutch university medical center. Results 135 of the 216 invited patients completed the questionnaire, resulting in a total mHealth usability score of 4.72 (± 1.13) out of 7. Thirteen semi‐structured interviews revealed 12 barriers and 11 facilitators. Most of them occurred at the level of the app itself. For example, patients received no feedback when all their answers were normal. The app made patients feel more responsible over their follow‐up, but could not fulfill the need for personal contact with the attending physician. Patients felt that the app could replace some of the outpatient follow‐up visits. Conclusions Our app is user‐friendly, makes patients feel more in control and remote monitoring can reduce the frequency of outpatient follow‐up visits. The barriers that emerged must be resolved before the app can be used in regular HNC follow‐up. Future studies should investigate the appropriate ratio of remote monitoring to outpatient follow‐up visits and the cost‐effectiveness of remote monitoring in oncology care on a larger scale.


| INTRODUCTION
Approximately 560.000 patients are diagnosed with head and neck cancer (HNC) each year worldwide. 1 Follow-up protocols prescribe clinical follow-up for three years to lifelong after curative treatment, consisting of several outpatient follow-up visits per year. 2 Patients have stated that the current follow-up regimen is intensive. 3 Some report that the frequency of visits is excessive and that they prefer a less rigorous follow-up schedule. 4,5 In addition, not all follow-up visits are essential from a medical point of view-some are mainly intended to reassure the patient. 3,6,7 Not only patients, but also healthcare professionals are willing to de-intensify HNC follow-up, provided that the doctor-patient relationship is adequately maintained. 8 Finally, the current follow-up schedule pressures healthcare systems and contributes to high costs. 9,10 Using a remote monitoring application (RMA) within HNC follow-up might actively engage patients in their care, optimize patient information delivery, reduce the number of required outpatient follow-up visits and thereby reduce healthcare costs. 11 RMAs are designed to collect patiententered data, such as patient-reported outcome and experience measures, which can be received and interpreted by the hospital involved. We specifically developed an RMA for HNC patients during the SARS-CoV-2 pandemic and used it as an alternative to outpatient follow-up visits since the pandemic led to a significant decrease in outpatient capacity at hospitals all over Europe. 12 Our RMA remained available as an add-on to regular follow-up care when outpatient follow-up visits were resumed.
Before further implementation into clinical practice, the RMA had to be evaluated and optimized. Therefore, this study aimed to investigate patients' experiences with remoting monitoring using our RMA during the SARS-CoV-2 pandemic. Our primary objectives were to evaluate the app's usability from a patient's perspective and to gain insight into the barriers and facilitators for using the RMA. In addition, we aimed to evaluate differences between patients over and under 65 years of age, as there is limited knowledge about the older oncology patient's experiences with and use of digital health services. 13 2 | METHODS

| Study design and setting
This study used a mixed-methods methodology, starting with a survey among HNC patients, followed by indepth interviews among a subset of these patients. This study was performed at the Radboud university medical center in Nijmegen, The Netherlands. This is one of the largest Dutch head and neck oncology centers, with approximately 500 newly diagnosed patients annually. The local ethics committee considered the study exempt from further review (dossier number 2020-6941). The consolidated criteria for reporting qualitative research (COREQ) checklist was used 14 (Appendix A, Table A1).

| Study population
Patients were eligible to participate in this study if they were treated for HNC and were currently in HNC follow-up. According to the Dutch guidelines, this follow-up consists of five years of prescheduled visits after treatment with decreasing frequency. 15 Other criteria were that they were fluent in Dutch and 18 years or older. HNC patients who used the RMA at least once were invited to complete the mHealth App Usability Questionnaire (MAUQ), a validated questionnaire that objectively evaluated the usability of mHealth apps. 16 Subsequently, qualitative interviews among a subset of participants were used to explore barriers and facilitators for using the RMA.

| The remote monitoring application
Our RMA is designed to monitor HNC patients at home. The RMA was developed using EPIC MyChart (EPIC EHR, Verona, Wisconsin) and, therefore, integrated within the Electronic Health Record. 17 Patients receive automatic monthly notifications for self-monitoring, which is achieved by completing a short questionnaire and examining the head and neck area using video instructions by themselves or a relative. See Figures 1 and 2. Case managers, who are supportive healthcare professionals coordinating HNC care, are automatically notified by a message from the electronic health record in case of potential abnormal findings. These include that the patient experiences symptoms consistent with possible disease recurrence, requires psychosocial support, has a question, or uploaded a photo or video that needs to be checked. The case manager reviews the patients' results, and decides whether the treating physician needs to be consulted and if the patient needs to visit the outpatient clinic.

| mHealth app usability questionnaire
The MAUQ consists of 21 questions and was developed to evaluate the usability of a mobile health app by statements on ease of use and satisfaction, system information arrangement and usefulness, which can be answered using a 7-point Likert scale (strongly disagree-strongly agree). 16 To translate the English version of the MAUQ to Dutch, translation guidelines for validated questionnaires were followed. Multiple forward-and backward translations were conducted by two independent professional translators from the Radboud University, department Radboud in'to Languages. Discrepancies were discussed and resolved by previous mentioned translators, TE, RK, and GB. 18 The final version of the Dutch MAUQ and the original English version are included in Appendix A, Methods A1 and A2. All eligible patients received an invitation to complete the MAUQ through CastorEDC, an electronic data capture platform. 19 After completing the F I G U R E 1 Screenshot of the remote monitoring application. Examples of the Dutch questions which the patient could answer with yes ("Ja") or no ("Nee").

F I G U R E 2
Screenshot of the remote monitoring application. A 3-minute video was incorporated to explain self-examination of the head and neck area. Patients were asked if they felt or saw anything abnormal.
MAUQ, patients were asked if researchers could contact them for an in-depth interview. The quantitative data was collected between September and November 2021. All answers were processed anonymously.

| Interviews
Patients were selected through purposive sampling based on gender and age from the population who completed the MAUQ and agreed to participate in the interview. 20 A semi-structured interview guide was developed by two researchers (CW, TE) based on literature published about usability of mHealth applications and the questionnaire results. Two experts in qualitative research (RK, RH) reviewed the interview guide. The interview guide covered four topics: use of the RMA, content of the RMA, influence of remote monitoring on perceived care, and future perspectives on remote monitoring in HNC care. Questions were open-ended and were optionally followed by questions to expand on each topic. See Appendix A, Methods A3, for the interview guide.
After two interviews, three researchers (CW, DS, TE) reviewed the interview guide and made minor adjustments accordingly. Interviews were conducted until data saturation, the point at which no new information was mentioned in the interviews, was reached. 21 Two researchers (CW, DS) were trained in interviewing and conducted the interviews in January and February 2022. No one was present besides the interviewer(s) and the participant. Participants were informed about the role of the researcher and the study goals. There was no relationship between the participants and the researcher, nor did the researcher benefit from certain outcomes. Written and verbal informed consent was achieved before each interview. Participation was voluntary, and patients could withdraw from the study at any moment without consequences.

| mHealth app usability questionnaire
The mean and standard deviation (SD) for each subscale and the total usability score were calculated according to the MAUQ instruction guide. 16 A Mann-Whitney-U test was used to compare the total scores of patients under 65 to those of 65 years and above. Two-sided p-values of <0.05 were considered statistically significant. IBM SPSS Statistics for Windows, version 25 (IBM Corp., Armonk, N.Y., USA) was used for all analyses.

| Interviews
Thematic analysis as described by Braun & Clarke in 2006 was used as follows to analyze the data that emerged from the interviews. The interviews were recorded to be anonymously transcribed by an independent third-party company. Participants did not receive the transcripts for corrections because they were transcribed verbatim. Two researchers (CW, DS) read and labeled the interviews by open-ended coding using Atlas.ti (Scientific Software Development GmbH, Berlin, Version 9). Coding differences were discussed by four members of our team (CW, DS, TE, RK) until consensus was reached. There were three moments of reflection during the coding process. The codes were compiled into subthemes. Two researchers (CW, DS) merged these subthemes into the six levels of the barriers and facilitators framework published by Grol and Wensing. 22 Again, differences were discussed until a consensus was reached. Afterwards, results were discussed with a fifth researcher (RH) and adjusted accordingly.

| Participants
In total, 216 patients who at least used the RMA once were invited to complete the MAUQ. After three reminders, 135 patients completed the questionnaire, resulting in a 63% response rate. The mean respondent age was 66.2 (± 9.7) years. Thirteen semi-structured interviews were conducted for the qualitative part of this study. The interviewees' ages ranged from 54 to 71 years, with a mean of 63.8 (± 5.0). The majority (n = 8) were male. Other participant characteristics are shown in Table 1.

| mHealth app usability questionnaire
The total mHealth usability score as measured by the MAUQ was 4.72 (± 1.13) on a scale from 1 to 7. The subscale scores for ease of use and satisfaction, system information arrangement, and usefulness were 5.0 (± 1.23), 4.74 (± 1.15), and 4.42 (± 1.24), respectively. The statements "the app was easy to use" and "it was easy to learn to use the app for me" scored the highest: 5.39 and 5.47, respectively. Patients also felt confident that the RMA sent the information they entered to their healthcare provider (score: 5.27). The statements "I had many more opportunities to interact with my  There was no significant difference between the two groups (p = 0.627).

| Interviews
Due to the SARS-CoV-2 regulations, interviews were performed by telephone, with a mean duration of 31 minutes (range 16-51). Data saturation was reached after 11 of the 13 interviews. A total of 12 barriers and 11 facilitators emerged and were classified into five levels of Grol and Wensing's framework. The innovation level covered nine of the 23 barriers and facilitators. 22 No barriers or facilitators fit the economic and political context, and this level was, therefore, omitted. The results are shown in Table 3.

| Innovation
The interviewees indicated that the RMA was easy to use. The questions were straightforward and focused on detecting possible cancer recurrence. However, patients would like to provide more nuanced answers and suggested adding a free-text field to elaborate on the answer or having more answer options than "yes" and "no." Interviewees encountered some problems while conducting self-examination of the head and neck. The instruction video could not be saved or rewound. In addition, some interviewees pointed out that the self-examination explained in the video did not specifically address investigation of their tumor localization.  An advantage of using the RMA was that it is less timeconsuming than a hospital visit. Interviewees also stated that follow-up by the RMA could reduce the frequency of hospital visits by extending the time between visits. When patients answer "yes" to questions within the RMA, a notification reaches the case manager, who contacts the patient to verify the results. When patients answer "no" to each question, all results are normal, and they receive no feedback from the RMA. Patients who always had a normal result indicated that they were unaware of what happened with their data.
'So, if I enter "no" in every question, I'll get "You have completed all your assignments again", but I won't get, after two days, "We've looked into it, glad it's going so well".'-Patient #3.

MAUQ subscales and corresponding statements a Mean (SD)
20. I felt confident that any information I sent to my healthcare provider using the app would be received.

| Individual professional
Healthcare professionals clearly demonstrated how to selfexamine the head and neck area in the video. However, patients also reported that an explanation of interpreting the findings was lacking.

| Patient
Patients reported not having to learn new things to use the RMA. Besides, the RMA allowed patients to be more involved with their illness and symptoms. Patients felt more self-responsibility when using the RMA compared to outpatient hospital visits. Some considered this positive, as it gave them more control over their follow-up. Others reported a negative effect, as they sometimes felt unnecessarily preoccupied with their symptoms.
One barrier mentioned was the difference in how physicians and patients examine the head and neck. Some patients expressed less confidence and knowledge in their way of examining than that of a physician.

| Social context
Some patients reported needing help from a relative to complete the self-examination as a barrier to using the RMA. A facilitator was that the RMA made patients feel more connected to the hospital.
Patients reported missing personal contact with healthcare professionals when using the RMA. The RMA could not offer the personal attention that they experienced during a physical visit.

| Organizational context
The RMA made it easier for patients to contact the hospital when needed. Patients also stated that the RMA allowed them to monitor their complaints at their preferred time.
The RMA was introduced during the SARS-CoV-2 pandemic. Patients received an email with information about the RMA. According to some patients, this introduction was 'If they had explained something in the video like "look for a bump or look for hard parts or whatever", it would have been clearer. The video does explain very clearly where to examine, but not what to examine.'-Patient #7.
'The RMA signals: listen up, you need to check yourself. Because the RMA reminds me of that, I also sometimes perform the self-examination when I sit quietly for a while, without having received a notification from the RMA.'-Patient #7.
'Imagine; it's beautiful weather and you go for a walk. You get home, and you get that notification again. You have to do another one of those examinations. Then I feel all kinds of things again. It makes me insecure.'-Patient #4. 'The doctors know what to feel, but I don't. I don't feel that, I don't know what to feel, what to look for. They do explain it in the video, but I can't do anything with that.'-Patient #2.
"I do think it's a good thing to keep using the RMA, because it also reminds people again of: "the hospital is still thinking about me, they read the answers I give and want to know how I am doing".-Patient #12. 'I do miss being able to go back to the hospital. They call and then they ask, "How are you?", and that's it. I can examine my neck, but I don't think that's the solution.'-Patient #8.
'I can complete the RMA at the time I prefer. The notification can come in spontaneously sometime during the day. Once I have seen it, I will fill it out in the evening when I have the time.'-Patient #7. incomplete and unclear. Some patients reported the lack of flexible endoscopic examination as a shortcoming.

| DISCUSSION
This mixed-method study provides insight into the usability of an RMA developed by a Dutch university medical center during the SARS-CoV-2 pandemic and patients' experiences with this RMA in HNC follow-up care. The overall usability of the RMA assessed with the mHealth Usability Questionnaire was good, particularly in terms of ease of use. The fact that there were no significant differences in MAUQ scores for patients over and under 65 years of age supports the assumption that our RMA is suitable for patients of all ages. Semi-structured interviews revealed 12 barriers and 11 facilitators for use in daily practice from patients' perspectives. Self-responsibility and the ability to perform checkups at their preferred time were mentioned as facilitators of using the RMA. Barriers included the interpretation of self-examination of the head and neck area and the lack of personal contact with their treating physician.
Patients felt that the RMA provided quick communication with their healthcare providers in case they filled out that they were experiencing physical symptoms. Van den Brink et al. also concluded that an electronic health system allowed for early detection of problems in HNC care that required direct intervention. 23 They hypothesized these problems could have led to adverse events had they been discovered during later outpatient visits. This could also be the case in our study, although we did not review individual reported symptoms. The MAUQ showed that patients were confident that information sent through the RMA would be received by their healthcare providers. The interviews revealed that patients received no feedback from the RMA in case all their answers were normal. During the development process of the RMA, we chose to only review abnormal patient answers to keep the automated monitoring process as efficient as possible. An explanation on when healthcare providers will review answers should be added to the RMA to better inform the patient.
The MAUQ did not cover all topics that emerged from the interviews. First, patients reported that the RMA made them more alert to symptoms of possible cancer recurrence in daily life and gave them a higher sense of self-responsibility. Bouaoud et al. also concluded that mHealth applications enable the early detection of health problems and improve HNC patients' self-management. 24 While our patients felt the instructions on examining the head and neck area were clear, directions on interpreting their findings were perceived as insufficient. A solution might be to practice the selfexamination with a healthcare provider before starting remote monitoring to increase patients' confidence in distinguishing normal from abnormal findings. However, Addeo et al. described a number of reasons for nonoptimal patient communication, including a lack of time or staff. 25 Time for patient education should, therefore, be scheduled. Also, the reliability of patient-examination in comparison to physician-examination could be studied in the future. Second, some patients clarified that the RMA did not fully apply to every HNC localization. For example, the instruction video does not cover examining the oral cavity. Duman-Lubberding et al. also reported that patients felt that Oncokompas, an e-health application for self-management after cancer treatment, was not tailored to their individual needs. 26 Our RMA could be personalized more by adding specific instruction videos for various HNC localizations. The RMA's complexity should be considered, as this could negatively affect the overall usability. Finally, patients felt that remote monitoring could not replace personal contact with their treating physician. Chen et al. reported clinicians' willingness to de-intensify HNC surveillance and also expressed the importance of maintaining the patient-physician relationship. 8 Therefore, we suggest that our RMA could be used to reduce the frequency of routine visits in individualized follow-up care, but not fully replace outpatient visits. Furthermore, shared decision-making on whether or not to use this tool in follow-up is essential because some patients could benefit significantly from the RMA, while others could be negatively affected due to increased preoccupation with their disease. As discussed by D'Allessandro et al., there is no "universal choice of treatment," or in our case, follow-up. 27 Future studies should also focus on integrating remote monitoring into HNC follow-up care while adequately maintaining the patient-physician relationship, and meeting the need for 'When do you get a notification? When do you not? It was a bit off now, which is understandable during Covid, but the context was a bit lacking.'-Patient #6.
'At a checkup appointment at the hospital, they will look down the throat with a tube to keep a close eye on everything. That is something that cannot be replaced by the RMA.'-Patient #13.
supportive care for cancer patients. Another aspect that needs further investigation is the cost-effectiveness of integrating remote monitoring in cancer follow-up care.
The main strength of this study is the mixed-methods design. Other studies have also investigated the feasibility and usability of remote monitoring in head and neck oncology using quantitative methods. 28,29 The results showed that remote monitoring of symptoms is feasible and that the applications were useful, which is consistent with our findings. However, through our qualitative analysis, additional barriers and facilitators were found, which can be used to optimize our RMA further.
This study also has some limitations. We cannot completely exclude selection bias. Patients with a higher educational level or more affinity with technology could be more likely to use the RMA and more inclined to participate in this study. One patient who had used the RMA once decided not to continue it because he was not satisfied with it. Unfortunately, this patient was not willing to participate in this study. There may be more patients who feel the same way. It would be interesting to study patients who chose not to start or continue using the RMA to understand the barriers for use better. Unfortunately, patients who started using the RMA during the pandemic did not give consent for participation in a research study nor for researchers to extract information from their medical records, making it difficult to compare patients who used the RMA with a general HNC population. Also, we did not have access to detailed demographic and clinical data of patients that solely agreed to complete the MAUQ because of privacy reasons. Therefore, we could not investigate whether certain characteristics such as HNC-site or -stage were related to differences in MAUQ scores. Finally, the response to the MAUQ was 63% after sending thee reminders. One could argue that a considerable amount of patients did not respond. However, our response rate was higher than the average email-survey response rate of 51% among surgical patients recently described in a systematic review by Meyer et al. 30 It should be noted that our RMA focuses on changes in physical function that may indicate recurrence. Although disease surveillance is one goal of cancer follow-up care, monitoring functional and psychosocial status is also important. Dutch HNC patients routinely receive symptomrelated questionnaires focusing on these domains through the Dutch Head and Neck Audit. 31 Our RMA and this study focus on detecting recurrence-related symptoms.

| CONCLUSION
This study shows that our RMA is user-friendly, making patients feel more connected to the hospital and more alert to new symptoms. The RMA can be improved by following patient suggestions, such as explaining how to interpret self-examination of the head and neck and giving patients feedback when the monitoring results are normal. Patients indicate that remote monitoring could reduce the frequency of outpatient visits, providing that the physician-patient relationship is adequately maintained. As such, the RMA could help to relieve the pressure on HNC follow-up care in the future. Since detecting recurrences or second primary tumors is an important goal of HNC follow-up, the next step would be to study the effectiveness of disease detection through remote monitoring.

FUNDING INFORMATION
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

CONFLICT OF INTEREST STATEMENT
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

DATA AVAILABILITY STATEMENT
The data that support the findings of this study are available from the corresponding author, CW, upon reasonable request.

PRECIS
Remote monitoring with an app makes head and neck cancer patients feel more in control over their follow-up and could reduce the frequency of outpatient follow-up visits. Data saturation Interviews were conducted until data saturation (the point at which no new information was mentioned in the interviews) was reached.

10
Transcripts returned Transcripts were not returned to participants for comments and/or corrections.

Domain 3: analysis and findings
Number of data coders Coding was performed individually by two researchers (CW, DS). There were several moments) of discussion and reflection of the codes between the researchers (CW, DS, TE, TK, RH).

11
Description of the coding tree The subthemes were merged into 5 levels of the framework of Grol and Wensing.

11
Derivation of themes Subthemes were derived from the data. These were categorized into themes from the Grol and Wensing framework.

11
Software Atlas.ti (Scientific Software Development GmbH, Berlin, Version 9) 11 Participant checking Participants did not provide feedback on the findings. Participants will receive the study manuscript after publication.

Reporting
Quotations presented Quotations are presented in the result section to illustrate the findings and are identified by participant number.

Results section Data and findings consistent
There is consistency between the data presented and the findings. Results section

Clarity of major themes
Major themes are clarified in the results section.

Results section
Clarity of minor themes Minor themes are described in the results section and discussed in the discussion section.
Results and discussion section The app has a link to Oncokompas at the end. Did you click on it?
• If yes: are you using Oncokompas due to this link? Why yes/no? • Do you think the link to Oncokompas is of added value to the remote monitoring app? Why?

Theme 3: Influence of remote monitoring on perceived care
What do you think of remote monitoring with the app compared to standard follow-up visits at the hospital?
• What do you think are the advantages of remote monitoring with the app compared to hospital checkups? Why? • What do you think are the disadvantages of remote monitoring with the app compared to hospital checkups? Why? • What do you miss in remote monitoring with the app compared to hospital checkups? Why? • Do you trust remote monitoring with the app? Why? • Has anyone from the hospital ever contacted you because of your answers in the app? What happened? What did you think of that?
• What do you think of the help the hospital offered in using the Home Monitor app? What was good? What could be better? • How do you feel when you use the app? (Anxious, reassured, etc.) Does this differ from follow-up visits at the hospital? • Does the app affect how often you are reminded of your illness? Why?
Theme 4: Remote monitoring in the future What do you think of the app being used more often in the future?
• How would we best introduce the app to patients? And when? • What do you think of being monitored with the app instead of standard hospital follow-up visits? Why? • What do you think of a combination of follow-up visits in the hospital and follow-up with the app? • Would you recommend the app to other patients? Why? • Are there any areas of improvement for the app that we have not yet discussed? If so, which ones? • What would you have wanted to know before you started using the app? • What would you tell a patient who was about to start using the app? • What would the ideal aftercare look like for you? And what role could the app play in that?

Wrapping up
• Are there any topics that we have not discussed yet that are of importance to you? • Do you have any remaining questions?