The ultimate question? Evaluating the use of Net Promoter Score in healthcare: A systematic review

Abstract Background Patient experience is a complex phenomenon that presents challenges for appropriate and effective measurement. With the lack of a standardized measurement approach, efforts have been made to simplify the evaluation and reporting of patient experience by using single‐item measures, such as the Net Promoter Score (NPS). Although NPS is widely used in many countries, there has been little research to validate its effectiveness and value in the healthcare setting. The aim of this study was to systematically evaluate the evidence that is available about the application of NPS in healthcare settings. Methods Studies were identified using words and synonyms that relate to NPS, which was applied to five electronic databases: Medline, CINAHL, Proquest, Business Journal Premium, and Scopus. Titles and abstracts between January 2005 and September 2020 were screened for relevance, with the inclusion of quantitative and qualitative studies in the healthcare setting that evaluated the use of NPS to measure patient experience. Results Twelve studies met the inclusion criteria. Four studies identified benefits associated with using NPS, such as ease of use, high completion rates and being well‐understood by a range of patients. Three studies questioned the usefulness of the NPS recommendation question in healthcare settings, particularly when respondents are unable to select their service provider. The free‐text comments section, which provides additional detail and contextual cues, was viewed positively by patients and staff in 4 of 12 studies. According to these studies, NPS can be influenced by a wide range of variables, such as age, condition/disease, intervention and cultural variation; therefore, caution should be taken when using NPS for comparisons. Four studies concluded that NPS adds minimal value to healthcare improvement. Conclusion The literature suggests that many of the proposed benefits of using NPS are not supported by research. NPS may not be sufficient as a stand‐alone metric and may be better used in conjunction with a larger survey. NPS may be more suited for use in certain healthcare settings, for example, where patients have a choice of provider. Staff attitudes towards the use of NPS for patient surveying are mixed. More research is needed to validate the use of NPS as a primary metric of patient experience. Patient or Public Contribution Consumer representatives were provided with the research findings and their feedback was sought about the study. Consumers commented that they found the results to be useful and felt that this study highlighted important considerations when NPS data is used to evaluate patient experience.

metric and may be better used in conjunction with a larger survey. NPS may be more suited for use in certain healthcare settings, for example, where patients have a choice of provider. Staff attitudes towards the use of NPS for patient surveying are mixed. More research is needed to validate the use of NPS as a primary metric of patient experience.
Patient or Public Contribution: Consumer representatives were provided with the research findings and their feedback was sought about the study. Consumers commented that they found the results to be useful and felt that this study highlighted important considerations when NPS data is used to evaluate patient experience.

K E Y W O R D S
Friends and Family Test, healthcare, improvement, measurement, Net Promoter Score, patient experience, quality

| INTRODUCTION
Improving patient experiences in healthcare is increasingly becoming a core strategic imperative and is fundamental to global healthcare performance. 1 The term 'patient experience' refers to 'the sum of all interactions shaped by an organization's culture that influence patient perceptions across the continuum of care'. 2 While patient satisfaction and complaint resolution are important, focusing solely on these has limitations, which have become increasingly apparent. 3,4 A key driver for the increased focus on patient experience is the correlation between experiences of healthcare and the safety and quality of care provision. 5 A positive patient experience is associated with fewer adverse safety events, more favourable perceptions of safety event handling (including disclosures) and a lower risk of litigation in the aftermath of a safety event. 6 Due to the strong association with quality and safety in healthcare, enhancing the patient experience is identified as a goal of the 'Quadruple Aim' framework, developed by the Institute for Healthcare Improvement 7 and is used as an indicator for healthcare accreditation with the National Safety and Quality Health Service Standards. 8 The inclusion of patient experience as an outcome indicator has necessitated the development of measurement tools for capturing, monitoring and benchmarking performance.
While many healthcare organizations strive to improve the patient experience, it has proven difficult to rigorously evaluate.
There is no single, best-practice method to measure patient experience, 9 and this limits the ability to benchmark patient experience across departments, organizations and countries. 10 A number of measurement tools have been developed to assess patient experience, such as the multi-item Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey, which is often used in the United States, 11 and the 12-item Australian Hospital Patient Experience Question Set (AHPEQS) developed by the Australian Commission on Safety and Quality in Healthcare. 12 As healthcare organizations regularly report on outcome measures, surveying of patients typically produces a large volume of data and increased demands associated with data management. Therefore, to simplify data management and reporting, healthcare organizations may prefer to use information that is condensed into a single performance indicator, or 'composite measure'. 13 Patient experience is a complex and multifaceted phenomenon, however, which can be difficult to condense into a single outcome indicator.
One method to achieve this is using the single-item Net Promoter Score (NPS). Created in 2003, NPS has been used in a variety of industries around the world, including banking, insurance and technology. 14 In recent years, NPS has been adopted into healthcare settings, frequently for the purpose of system-level benchmarking. 15 NPS is a popular surveying method that is used globally and has been dubbed 'the ultimate question'. 16 NPS consists of a two-part questionnaire. Firstly, respondents are presented with a rating question: 'How likely is it that you would recommend our business/service to a friend or colleague?' on a scale from 0 (not likely) to 10 (very likely). This is followed by an openended, free-text item, which enables respondents to provide the main reason for their score. Based on the rating that is provided, responses are categorized into three groups: 'Detractors' (ratings 0-6), 'Passives' (ratings 7 and 8) and 'Promoters' (ratings 9 and 10).
The overall score, known as the 'Net Promoter Score', is calculated by subtracting the percentage of Detractors from the percentage of Promoters, therefore NPS can range from −100 (i.e., all Detractors) to +100 (i.e., all Promoters; see Figure 1). 16 NPS is widely used in healthcare settings internationally. For instance, in 2012, the UK National Health Service (NHS) implemented a modified version of the NPS survey, called the 'Friends and Family Test' (FFT), which became a nationally mandated measure to monitor consumer satisfaction and healthcare quality performance. 9 This replaced the NPS numerical rating (i.e., 0-10) with a 5-point Likert scale to measure willingness to recommend health services ADAMS ET AL. | 2329 (which ranged from 'Extremely Likely' to 'Extremely Unlikely'; see Figure 2). For improved comprehension, FFT will be categorized as NPS in this review.
NPS may be appealing as a simple and quick single-item measure to evaluate patient experience performance, however there is a shortage of evidence regarding the suitability of the NPS to assess patient experience in healthcare settings and the limitations of its use. 16

| Exclusion criteria
Studies were excluded if the NPS was used as an outcome measure within a research study and there was no evaluative data about the use of the NPS. Due to the focus on evaluation of peer-reviewed research, other forms of publications (such as case studies, commentaries, and editorial pieces) were not eligible for inclusion.

| Study identification
The study identification was conducted in consultation with a medical    The following data were extracted: author, year, location, methodology, study objectives and key results regarding the review objectives.

| Assessment of study quality
Studies were critically appraised using the 16-item Quality Appraisal for Diverse Studies (QuADS) tool, which has demonstrated good reliability and validity to evaluate multiple types of study designs. 18 C.
A. conducted the quality review for all included studies, and these were re-assessed by reviewers (R. H. and R. W.), with ratings compared to determine the agreement between the two reviewers.
Inter-rater reliability was calculated using the κ test and 67.3% agreement was achieved, which indicates substantial agreement for quality appraisal.

| Data synthesis
Due to the range of studies included, a narrative synthesis was performed to combine the findings from the different studies. This involved identifying the key findings from each study, and a summary of these findings was synthesized into a narrative text form. Coding occurred with input from three researchers (C. A., R. H. and R. W.), and regular meetings were held (fortnightly) to discuss and reconcile any disagreements, which produced the final thematic categorization.

| Excluded studies
Studies were excluded after full-text review (n = 64) because studies were the wrong study design, such as editorials (n = 52), wrong comparator, such as did not evaluate NPS (n = 10), or were not conducted in a healthcare setting (n = 2).
Stakeholders were consulted about the preliminary review findings to support the analytical process. In this study, consumer representatives were provided with the preliminary findings and feedback was sought about the approach to categorizing the research data. Their feedback was used to finalize the themes and categories that were used in the paper. Consumer representatives highlighted the importance of the findings, including relevancy to patient experience measurement and healthcare system improvement.

| RESULTS
A total of 468 studies were obtained from the database search.
Duplicates were removed, and 256 studies were subject to title and abstract screening. A total of 180 studies were excluded in the first stage of screening, most commonly because the studies were commentary articles. As a result, 76 articles were subject to fulltext screening, from which 64 studies were excluded, most commonly because the studies did not evaluate the use of NPS.
Finally, 12 articles were included in the systematic review (see Table 1). The search and selection process is outlined in Figure 3.

| Characteristics of included studies
Ten out of twelve studies were conducted in the United Kingdom, which may reflect the national adoption of NPS measurement (via FFT). One study was conducted in the Netherlands, and NPS was also researched in developing countries (Mumbai, Kenya, and Nigeria).
Although NPS originated in the United States, there was a lack of USbased research. Research was conducted in a range of healthcare settings, including acute hospitals (5), dental practices (3), general practitioner (GP) clinics (2), orthopaedic centres (2), family planning centres (1) and community mental health services (1). One study was conducted in both general and dental practices. The majority of studies examined the use of NPS to evaluate adult patient experience (n = 11), while one study focused on paediatric patient experience.
The majority of the studies (9) used a survey format, two studies used interviews and one study was a scoping review.

| Study quality
Assessment of study quality using the QuADS tool demonstrated variation in the quality assessment criteria for the included studies.
The highest scoring areas for study quality included: statement of research aims and objectives, description of data collection procedure and appropriateness of study design to address the research aims. Conversely, lower scoring areas of study quality included: theoretical underpinning to the research and justification for the analytical method selected. In addition, the majority of studies (8 out of 12) received a nil score for evidence that the research stakeholders have been considered in the research design of conduct, which highlights the need for better consumer and stakeholder engagement with research codesign.
The narrative synthesis identified three categories in accordance with the research objectives: context and factors for NPS use, application of NPS for service improvement and considerations for implementing NPS (and managing NPS data) in the healthcare context.

| Evaluative evidence of NPS in healthcare
From the 12 studies reviewed, four studies identified that there were benefits associated with using NPS to evaluate patient experience. 14,[19][20][21] Research indicated that NPS can be easily used by a wide range of patients, including adults with low health literacy 19 and children. 20 Furthermore, NPS surveying has shown high completion rates by patients, with the ability to generate a large volume of data. 14  • Case mix should be taken into account when comparing scores between hospitals and departments.
• NPS differences between procedures suggest no overarching score should be given without case-mix adjustment. Koladycz • Clients (in India) understood the NPS question without additional explanation.
• Participants may feel challenged to offer suggestions for improvement.
• Targeted feedback may be necessary to identify actions for improvement, including asking specific questions.
• Staff reported that NPS was easy to complete.
• NPS can be used effectively in lowresource settings with low literacy. • NPS is less valid as a summary of patient experience than a global rating.
• Patients thought it was unusual to recommend healthcare.
• NPS may oversimplify results, so should be used as an addition to the survey results. • NPS is easy to implement, but data had minimal impact on quality improvement.
• Suggest removing NPS questions and providing a more targeted, specific survey. • NPS survey was completed by a wide range of ethnicities.
• NPS survey is useful but may add an administrative burden.
• NPS may raise awareness about the importance of feedback. • NPS may produce value if part of wider data collection, and with a larger sample of service users.

Sizmur, Graham and
• Caution is needed when comparing NPS from patients with different prognoses.
• It may be better to develop a quality score from multi-item measures, which would provide more breadth and improved reliability.
Abbreviations: FFT, Friends and Family Test; GP, general practitioner; NPS, Net Promoter Score; PREM, patient-reported experience measure; PROM, patient-reported outcome measure.
particularly if consumers have limited choice of healthcare providers. 16,21,22 For instance, in a study of GP clinics, patients were frequently confused by the recommendation question, particularly because they had limited or no option to choose their GP. 22 This review identified that the most useful component of NPS surveying was the patient comments section, which was noted to be beneficial in 4 out of 12 studies. 19,21,23,24 The comments section in NPS is well-used by patients, typically completed by more than threequarters of respondents, 23 and these comments may help to contextualize the quantitative results gathered from NPS surveying. 25 For that reason, following a review of the FFT (in 2014), the open-ended question became mandatory due to the perceived value of these comments. 22 Alternatively, some healthcare practitioners have reported insufficient detail in the patient NPS responses, which may lack specific information about the causes of patient dissatisfaction 22 and/or constructive comments about how to improve healthcare services. 19

| Applications of NPS for monitoring and benchmarking
Although NPS has often been implemented for the purpose of monitoring and benchmarking performance, researchers have raised concerns about the validity of NPS data when used to compare healthcare services. Three studies found that NPS results can be influenced by a variety of system and service factors 14,21,24 which suggests that NPS may be indicative of a wider range of system and service features apart from patient experience. Two studies found significant variation in NPS based on condition and intervention, such as mental health 21 and type of joint replacement surgery. 14 For example, one study identified significant variations in NPS between procedures, with NPS of total hip replacement as 71, while NPS of total knee replacement was 49. 14 As such, caution may be required when comparing NPS between prognoses and conditions. 21 In addition, NPS may be influenced by various factors, such as mode of administration, gender and patient age. 24 For instance, two studies reported that patients over the age of 70 had a lower willingness to recommend services. 21,24 When analysing NPS data, therefore, differences between demographic groups may also need to be considered. NPS may be frequently promoted as a comparison and benchmarking tool, yet researchers have recommended that comparisons of results should be localized. 24,26 For example, the comparison of NPS results may be restricted to a single hospital site. 24 Additionally, two studies noted that NPS can be influenced by differences in cultural norms and expectations, which may limit the ability to compare and benchmark NPS data internationally. 16 an insufficient measure, 14,19,21,28 noting that composite measures can oversimplify results and should be used with caution. 13 Despite the lack of specificity, this information may still provide broad but useful insights. 26 One study recommended that summary scores should be only used to supplement the results of a larger survey set 16 and three studies suggested that NPS may be better suited as part of the larger feedback process. 14,19,28 Researchers have proposed that multi-item instruments may be more useful than NPS to provide a greater breadth of evaluation and improved reliability. 21 For these reasons, rather than being a singular encompassing metric, NPS may be better utilized as a starting point to better understand patient experience, which can help to identify areas that require further investigation and detailed examination. 19

| Patient and staff attitudes towards NPS
Two studies noted that the NPS question was well-understood by a diverse range of patients, including those with low literacy. 19,20 However patients may also experience concerns about the recommendation question, particularly if they do not have a choice of healthcare provider. 22 While patient attitudes are quite positive, there are mixed results about staff attitudes towards NPS (from both managerial and clinical staff). One study reported positive staff attitudes towards NPS, which was perceived as being quicker and easier to implement than existing survey methods, 19 yet this contrasts with the results of two studies that reported negative employee attitudes towards NPS. 22,23 In one study, staff in dental practices reported NPS to be time-consuming and resource intensive to implement and maintain, 23 while in a study of 42 GP practices, only 10% of GP practices expressed positive attitudes towards NPS, 22 with staff reporting that NPS provided minimal useful information in comparison to existing survey methods.

| DISCUSSION
When measuring patient experience, methods for evaluation need to be capable to assess the complexity of healthcare delivery, and also suitable to the particular healthcare setting. 10 Although NPS has been used in a wide range of service industries, it has not been validated for use in the healthcare setting. The aim of this systematic review was to evaluate the suitability of the NPS to assess patient experience and to identify potential limitations of its use in the healthcare setting. Healthcare organizations may seek to maximize patient experience, but there is currently no best-practice measurement tool to measure patient experience. 9 14,21,24 Furthermore, NPS ratings may also be affected by cultural considerations, including cultural differences in rating norms, 16,19 which may have an impact on the ability to compare NPS results internationally. As such, NPS may be more useful when monitoring longitudinal performance (such as change over time) for a more specific group, such as one hospital site, department and/or patient cohort. 24 NPS may also be better used to evaluate discrete cohorts, such as categorized according to admission type (i.e., emergency or elective admission), with localized and site-specific NPS evaluation.
Finally, although NPS is often implemented to support healthcare improvements, this review finds that NPS has limited ability to support healthcare improvement. 13 While NPS can generate a large volume of data, due to its high completion rates and ease of use, 14

| Strengths and limitations
This review provides a useful synthesis of evidence for many healthcare organizations that are currently using NPS or are considering implementing NPS. The use of a systematic review methodology of peer-reviewed literature strengthens this review. suggests that NPS may be insufficient as a stand-alone metric.
Instead, NPS may be more effectively used as part of a more comprehensive feedback process, such as multi-item surveying.
Research does not support the use of NPS for widespread benchmarking, hence NPS may be more suited to assess localized performance. Overall, there are limitations to the use of NPS, so healthcare organizations may need to consider a range of factors before selecting NPS as a primary measure of patient experience.