Implementation of a “real‐world” learning health system: Results from the evaluation of the Connected Health Cities programme

Abstract Background The “learning health system” has been proposed to deliver better outcomes for patients and communities by analyzing routinely captured health information and feeding back results to clinical staff. This approach has been piloted in the Connected Health Cities (CHC) programme in four regions in the North of England. This paper presents the results of the evaluation of this program conducted between February and December 2018. Methods Fifty nine semistructured interviews were completed with a mix of CHC programme staff and external partners who had contributed to the delivery of the CHC programme. Interviews were audio recorded and transcribed verbatim. This also included the review of project documentation including project reports and minutes of project group meetings, in addition to a short online survey that was completed by 31 members of CHC programme staff. Data were analyzed thematically. Results Two overarching themes emerged through the thematic analysis of participant interview: (a) challenges in the implementation of learning health system pathways, and (b) benefits to the CHC approach for both staff and patients. In particular, time constraints in delivering an ambitious program of work, data quality, and accessibility, as well as the long‐term sustainability of the CHC programme were noted as key challenges in implementing a LHS at scale. Conclusions The findings from this evaluation provide valuable insight into creating learning health system at scale, including the potential benefits and likely challenges.

chronic disease have led to an increased demand in health and social care services. At the same time, the amount of health data being collected and stored is vast, while the technology and analytic tools needed to analyze "big data" has been developed. 4,5 In the North of England, there is an ever-increasing health gap.
These deep-rooted and persistent inequalities have resulted in Northern populations regularly found to be less healthy than those who live in the South. 6 This health gap can be found across all social groups and among both men and women. 7 Furthermore, there is a 2-year life expectancy gap between those who live in the North and the rest of England, with premature death rates 20% higher for those living in the North across all age groups. 6,8 "A learning healthcare system is one that is designed to generate and apply the best evidence for the collaborative healthcare choices of each patient and provider; to drive the process of discovery as a natural outgrowth of patient care; and to ensure innovation, quality, safety, and value in health care" 9 Thus, a "learning health system" should be able to deliver better outcomes for patients and communities by analyzing routinely captured health information and feeding back results to clinical staff. 10,11 This approach is being piloted in the The CHC programme started in January 2016, with seven core deliverables set out by the DHSC forming the basis for establishing four regional LHS. These are presented in Table 1. In 2018, the final year of program delivery, an evaluation was commissioned to provide an assessment of the CHC programme in relation to progress toward these seven deliverables. However, in doing so, two key issues were raised.
First, while the number of pathways varies for each region, the CHC programme was tasked with developing at least two pathways per region. Our funders requested that eight care pathways were included as part of the evaluation process; however, there are more than 16 in delivery. Table 2 shows the four CHC regions in relation to the eight care pathways chosen for inclusion in the evaluation with a brief overview of the work planned for each pathway.
Second, in designing the evaluation, it became clear that no bench marking data or any other form of monitoring data or information had been formally collected during the program delivery period. Therefore, it would be difficult to accurately measure impact and calculate any economic benefits or potential savings to the health system since the CHC programme's inception in January 2016.
Furthermore, halfway through the data collection period, the DHSC changed the CHC programme deliverables and scope of the evaluation. To enable this, the project as a whole was extended beyond the original end date of December 2018 to end of 2019. As a result, the original evaluation changed from being an "end of program" evaluation to an interim evaluation.
The purpose of this article is to present the results emerging from the interim evaluation with an emphasis on the emerging benefits and key challenges faced by the CHC programme in implementing learning health system across the North of England.

| METHODS
This evaluation took place between February 2018 and December 2018. To begin with, an inception meeting with the project management group and review of key project documentation took place.
Semistructured informational interviews were conducted with key CHC staff from each of the pathways and the central CHC hub in mid-February to late April 2018. A total of 28 key staff participated in these interviews. The aim of these informational interviews was to establish which two pathways would be put forward for evaluation; what they considered to be the greatest challenges; any issues they foresaw; successes and unintentional outcomes of the CHC and a consideration of future challenges with regard to the CHC programme deliverables.
This allowed the scope of the evaluation to be determined, the  Description of deliverable

Deliverable 1
The establishment of data sharing strategies and data sharing agreements for each CHC region.

Deliverable 2
The establishment and delivery of governance arrangement for the sharing and usage of data for each CHC region, across the North and the United Kingdom.

Deliverable 3
The optimization of Ark workforce arrangements, including the identification of long-term CPD requirements the establishment of new skill bases.

Deliverable 4
The creation of the Ark as an analytical platform for investigating linked data.

Deliverable 5
The analysis of eight care pathways, identification of any pathway variations, and proposals for any improvements if possible.

Deliverable 6
The creation and implementation of frameworks for potential integration with R&D partners and the future rising of Foreign Direct Investment.

Deliverable 7
The production of a CHC business model suitable for scaling across the North and sustainable for delivery in the NHS. Using technology and data to improve the diagnosis and treatment of stroke • Improve the recognition of stroke by paramedics to maximize the proportion of acute stroke patients taken directly to a specialist stroke center for timely expert care and minimizing the number of nonstroke patients entering the stroke pathway. • Provide timely and focused referral to neurosurgery for patients in Greater Manchester with stroke caused by a brain hemorrhage. • Ensure that all patients get all the right treatments that they need to reduce the risk of another stroke when they are discharged from hospital.
To improve stroke recognition by paramedics by linking ambulance data to data at Salford Royal; using primary and secondary care data to create a large cohort of stroke and TIA patients for creating a predictive model of patients who are at high risk of stroke; and using acute trust data to identify predictive factors of early deterioration and death.

North East North Cumbria
Predictive modeling for unplanned care • To develop predictive modelling tools for unplanned care forecasting to support demand management and service planning in relevant health and social care services.
To produce statistical models that can be used by health/local authority/ other analytics teams to produce daily forecasts up to 6 mo in advance with the pertinent associated uncertainties and variations in urgent and emergency care.

SILVER: Smart Interventions for Local Vulnerable Families
• To develop data sharing agreements to allow the linking of existing health data across multiple health agencies via one platform that provides recommendations to key workers.
To link data across multiple agencies including health (physical and mental), social care, criminal justice, housing, and education to develop a more complete Learning Health System.
North West coast Development of a learning system for alcohol • To be able to inform health professionals about local clinical care.
Improving the way information is collected, analyzed and shared between agencies and service users (Continues)

| Data collection
In May 2018, lead pathway and regional partners were sent the online survey to be cascaded to staff working on the CHC programme. This was to gain a broad understanding of CHC staff experiences across the different pathways in relation to the program deliverables, as well as their views on the challenges, benefits, impact, and successes. A reminder was sent out 4 weeks after the original mailing, with pathway leads reminding staff to complete the e-survey.
Qualitative research was undertaken with a selection of CHC staff for each pathway and region. In late May 2018, initial semistructured confidential interviews were conducted. Detailed discussions with a cross section of pathway and regional staff enabled the development of a balanced narrative of key achievements and challenges across the CHC programme. These were then used to inform recommendations and the development of case studies. Interviews were carried out until October 2018. In order to consider the wider benefits and challenges of the CHC programme within each region, interviews were also conducted with a number of stakeholders who sit within the CHC programme but were not directly involved in activities in the eight care pathways or regional activities.

| Data analysis
Our data analysis utilized a thematic approach 12,13 where data from the documentary review, survey, and interview data were triangulated to ensure consistency in our findings. Thematic analysis is a widely used method to identify, organize, analyze, and report patterns or themes within qualitative research data. 14 Themes were compared across participants and documents with data analysis taking place alongside data collection.

| RESULTS
Two overarching themes emerged through the thematic analysis of participant interviews, online survey, and documentary analysis:  we try to cope with these barriers that people have put in.
Participants working with data expressed concerns about the data quality, highlighting the following specific issues: missing data, incorrectly coded data, and duplicated data. Criticisms were expressed that the data were paid for and in some cases, the data itself came from NHS Digital, and therefore, paid for data should be of better quality and standardized.

| Subtheme: Long-term sustainability and commitment
There did not appear to be a consistent level of commitment from both DHSC as a funder and some partners within the CHC programme. The main challenge for each CHC region was to ensure that the partnership involved the right senior people in order to ensure commitment and direction at a high level. Each of the regions utilized a different governance structure, with mixed results. Only one region was successful from the start in fully implementing its governance structure, with senior staff in clearly defined roles, a clear regional vision, and operational staff, including dedicated project managers for each care pathway, had resulted in quicker progress made. Key issues that affected the other regions were a lack of clear vision and agreed set of regional objectives. Some staff noted that trying to get everyone to work out what was the common ground and then develop and implement strategies that would work for all was very time-consuming.
All participants were concerned that all work completed to date will have been for short-term gain rather than a long-term investment in the North of England. Participants interviewed that did not work on the CHC programme noted the lack of secure data repositories for research purposes in the North of England besides the CHC programme and that DHSC needs to invest more in the North of England to reduce health inequalities: There are the issues about the sustainability of the CHC programme,

| Subtheme: Different working cultures and priorities
Conflicts in the way different partners work were noted in all regions.
In particular, the different pace of work and changing funding landscape between academia, the NHS, and industry created tensions within regions. Some of this was often due to the differences in language used by each partner, as one member of staff commented:

| Subtheme: Communication
Internal communication was a challenge across the CHC programme.
Getting all regions to communicate and cascade information had been especially difficult to achieve in some regions or within individual partner organizations. Many participants felt "disconnected" to the wider CHC programme, with communication blockages appearing both within regions, as well as overall as a program. As several participants commented, "we don't seem to have the connected element of the 'Connected Health Cities' project". As a result, many participants felt isolated from both the regional and overall CHC programmes of work. Furthermore, CHC funding has allowed partnerships to explore innovative ways of working with data without the constraints associated with traditional funding streams. As one participant reflected:

|
It has allowed us to explore the possibilities more freely than what we would have been able to do. I think that having the funding there to do this has been fantastic, because it would be hard to find someone to fund something like this…you know, I don't think NIHR would have funded it.

| Subtheme: Benefits for patients of CHC programme activities
Across all CHC regions, as well as the CHC programme as a whole, patients and members of the public have had an increased level of involvement. Participants working on care pathways noted how useful and beneficial it was to have insights from patients, members of the public, and health and social care staff in their projects. In some cases, the patient voice was key to pushing forward a piece of work when an NHS Trust might have been hesitant.
In other pathways, gathering patient and public views and engagement were seen as being critical elements of the CHC programme, to the extent that some regions have a specialist PPI role within their partnerships to enable the voice of patients to be heard.
As one participant observed:

| DISCUSSION
The focus on developing learning health system in recent years has the potential to deliver better outcomes for patients and communities by analyzing routinely captured health information and feeding back results to clinical staff. 15,16 For a health system to be able to learn from the data it collects there is a need for a suitable infrastructure and working culture that supports the routine application of learning cycles. 17 This suggests that both need to exist if a LHS is to be successful in practice.
This interim evaluation provides an in-depth look at the key challenges of implementing a LHS at scale in the North of England.
In particular, the CHC programme had to develop and implement In attempting to implement the CHC programme, the evaluation noted a number of challenges in implementing the CHC programme, which have been described in the results as "sub-themes". These include: time constraints; data; long-term sustainability and commitment; different working cultures and priorities; and communication. While other LHS studies have described challenges in obtaining data from an accessibility perspective, 18 here we have been able to provide further insight into the organizations in England that would need to be approached to gain approvals for data sharing in a LHS.
Challenges around data accessibility and quality in the CHC programme have also been discussed elsewhere. For example, issues around the heterogeneity of patient records, and differences between routine data and data collected for the purposes of research, means that data analysts cannot assume that patient data provides the full or accurate clinical picture of a care pathway, nor the population as a whole. 19,20 As noted in Reference 18, in their systematic review of adopting a LHS in practice, the CHC programme has found similar challenges in legal bases for data sharing agreements among participating organizations, building trust and follow-on funding.
Despite the challenges that have been reported here, and in other LHS studies, we have been able to gain some early insight into the benefits of using a LHS. While it is too early in the delivery of the CHC programme to ascertain patient outcome impact, the multi- to ongoing delays in obtaining data approvals and there had been no baseline data collected at the start of the CHC programme. As a result, this evaluation changed from being an "end of program" evaluation to an interim evaluation. Therefore, a full end-of-program evaluation of the CHC programme would be able to consider the benefits, challenges and patient outcomes in greater depth.
On a more practical level, one of the key learnings from this interim evaluation has been the importance of building trust with all organizations involved in both care pathway and CHC region levels.