Patient involvement in the development of patient‐reported outcome measures: a scoping review

Abstract Background Patient‐reported outcome measures (PROMs) measure patients’ perspectives on health outcomes and are increasingly used in health care. To capture the patient's perspective, it is essential that patients are involved in PROM development Objective This article reviews in what ways and to what extent patients are involved in PROM development and whether patient involvement has increased over time. Search strategy Literature was searched in PubMed, EMBASE, MEDLINE and the Cochrane Methodology Register. Inclusion criteria Studies were included if they described a new PROM development. Data extraction Basic information and information regarding patient involvement in development phases was recorded. Main results A total of 189 studies, describing the development of 193 PROMs, were included. Most PROMs were meant for chronic disease patients (n = 59) and measured quality of life (n = 28). In 25.9% of the PROM development studies, no patients were involved. Patients were mostly involved during item development (58.5%), closely followed by testing for comprehensibility (50.8%), while patient involvement in determining which outcome to measure was minimal (10.9%). Some patient involvement took place in the development of most PROMs, but in only 6.7% patients were involved in all aspects of the development. Patient involvement did not increase with time. Conclusions Although patient involvement in PROM development is essential to develop valid patient‐centred PROMs, patients are not always involved. When patients are involved, their level of involvement varies considerably. These variations suggest that further attention to building and/or disseminating consensus on requirements for patient involvement in PROM development is necessary.


Introduction
Patient-reported outcome measures (PROMs) assess health outcomes from the patients' perspective. 1,2 PROMs were originally developed for the use in clinical research as a way to measure treatment effectiveness. However, PROMs are now increasingly used in clinical practice to monitor and to improve the care for individual patients and in health policy and management, for example in the English National Health Service (NHS), to measure the performance of health-care providers 2,3 or by the American Centres for Medicare and Medicaid Services (CMS) to award incentive payments for eligible professionals. 4 Patients are more and more considered to possess important experiential knowledge on health and health care, a source of information that is relevant for improving quality of care. [2][3][4][5] To truly capture the patient's perspective, it is essential that patients are involved in PROM development, 5-9 as only patients can determine which health outcomes are relevant for them [9][10][11] and whether the questionnaire captures these outcomes in a comprehensible and understandable manner. [12][13][14][15] Besides, if a questionnaire fails to represent the patients' perspective, it may result in patients failing to complete the questionnaire and a negative impact on the validity. 16 Only a few studies investigated patient involvement in the development of PROMs, and only for a small number of PROMs, or for a specific disease. 7,17 Nevertheless, there are concerns that there are many PROMs in use where no patients were involved during the development process. Haywood investigated the quality and acceptability of PROMs used in chronic fatigue syndrome/myalgic encephalomyelitis and found no clear evidence of patient involvement in the development of PROMs. 17 Other research using interviews with patients uncovered conceptual difficulties and questionnaire design problems among a few well-known PROMs which might have been prevented by involving patients in the development. 7 It appears that although patient involvement is increasingly accepted as an essential part of the development process, we are still using PROMs which were developed without patient involvement.
Although patient involvement appears to be neglected in PROM selection and at least in the development of earlier PROMs, there is some agreement in the literature on how patients should be involved. There are several phases in the development where the input of patients can be used to create a more patient-centred PROM. To ensure that the health outcomes and domains measured with PROMs are relevant for patients, patients may partake in identifying core outcomes, 10,11 for example by participating in a reference group or focus groups. 18 After establishing which outcomes should be measured, patients should be involved in the generation of items by either focus groups or interviews. 12,15,18 While interviews can be used to obtain a number of personal feelings and opinions on a subject, which can be especially useful for eliciting the opinions of minority groups, focus groups may be useful for obtaining opinions that are likely to reflect the majority. 1,19 Choosing between these methods may also be dependent upon the patient population or practical considerations. 20 Item development may have some overlap with the phase where core outcomes are identified, because although they have different purposes, both could take place using the same qualitative methods. After item development, it is important to determine that the questionnaire is comprehensible and that the content is valid for patients. [12][13][14][15] Structured cognitive interviews are an established way to ensure this 12,13 by enabling researchers to determine how items are interpreted by potential respondents and how a response is formed. 13,15 It appears that there are at least three phases in which patients can be involved in the development of PROMs, determining important health outcomes, item generation and checking for comprehensibility and content validity. However, as mentioned before, PROM development procedures are not standardized regarding patient involvement and the few studies who Health Expectations, 20, pp.11-23 development. 7,17 As patient involvement is such an important factor in the development of PROMs, 5, [9][10][11][12][13][14][15]21 it is of the utmost importance that all the parties involved are aware of the level of patient involvement of a PROM if it comes to selecting a PROM or interpreting PROM results. Additionally, awareness of the current situation regarding patient involvement is necessary so that all parties can act accordingly to ensure the development and use of valid patientcentred PROMs. Therefore, by conducting a scoping review, this study aims to review the level of patient involvement in PROMs development. We expect to find more patient involvement in recently developed PROMs as compared to older PROMs, as the importance of the patients' perspective is increasingly stressed. [2][3][4][5] In summary, our research questions are as follows:

Scoping review
This study is a scoping review. Scoping reviews are used to give a unique overview, in this case of patient involvement in the development of PROMs. For a scoping review, studies are not excluded based on type of study (as long as a description of a PROM development was included), type of participants, type of health care or development techniques. Furthermore, studies are not assessed on aspects of methodological quality. Characteristics of a scoping review can be a lack of a narrow review questions, the inclusion of studies which have employed a range of data collection and analysis techniques, the quality of the included studies is not assessed and the subject has not been subjected to a review before. 22 Although we did not conduct a systematic review, the PRISMA statement 23 was followed where possible.

Search strategy
The PROM very slightly or shortened a PROM without making any fundamental changes.

Post hoc exclusion criteria
Studies were excluded if they did not concern a published manuscript or if the manuscript was written in a language other than English, German, Dutch or French. As only two manuscripts were written in a different language, the impact was limited.

Study eligibility
Author DB independently scored ten per cent of the abstracts on eligibility, after which disagreements were discussed. The inter-rater agreement on study eligibility, calculated using the kappa statistic, was 0.71 (95% CI: [0.50, 0.92]). Full-text selection by a second reviewer was deemed unnecessary as the reasons for exclusion were mostly very clear. However, authors DD and DB independently assessed the eligibility of 10 full texts. Additionally, full texts which were excluded because the questionnaire that was developed measured the process of care instead of outcomes, were discussed among the authors. There were no disagreements. See Fig. 1 for details on the literature selection process.

Data extraction
The basic information that was collected from each article included the first author, the year of publication, the country, the health problem or treatment associated with the PROM, whether the patient group concerned adults, parents or children, whether the PROM is generic or specific, the health outcome measured, and the care type associated with the health problem.
To gain insight into patient involvement in the development of PROMs, additional information was recorded which is displayed in Table 1.
Authors DD and DB extracted the data for 10 studies. Disagreements were discussed, after which the categories were further specified. Main discussion points were as follows: 1. Generic or specific? Initially, we defined a PROM as generic if everyone in the general population could answer the questions and a PROM as specific if it covered a specific disease. However, several PROMs were not generic, but covered health problems relevant for more than one disease, such as pain or fatigue. Conventional definitions 24 would label this category as specific as it measures a specific concept. However, most of these questionnaires can be used for a far larger group of patients than a disease-specific questionnaire could, which is why we gave this group of questionnaires the label 'semi-generic'. 2. What is regarded as patient involvement in determining which health outcome to measure? Although ideally patients are involved from the start of the development by asking them what outcomes are important to them, this is extremely rare. Therefore, we broadened this category to include cases where patients contributed to the development of the framework or domains within an established outcome or where patient suggestions were not restricted in any way.

Search flow
As shown in Fig. 1, 4489 studies were identified from the databases PubMed, EMBASE, MED-LINE and the Cochrane methodology register. After a search for duplicates, 329 were removed, leaving 4160 studies. Of these studies, 219 studies remained after selecting abstracts which appeared to meet the inclusion criteria. Finally, a full-text selection resulted in 189 studies. The studies excluded during full-text selection were mostly presentations (n = 10), adaptations of already existing PROMs (n = 9) or patientreported experience measures (n = 4).

Study characteristics
The 189 studies were published from 1980 until 2014 and described the development of 193 PROMs. The PROMs were mostly developed in the USA (n = 69), the UK (n = 62) and Germany (n = 17). Most PROMs were developed for a specific patient group (n = 152) or few patient groups (n = 32). Nine questionnaires were generic. Of the 193 PROMs, 180 questionnaires were meant to be filled out by adults and one by parents. A range of patient populations were included, from haemophilia (n = 6) and  mental health patients (n = 5) to diabetes patients (n = 4). The majority of the PROMs were meant for chronic disease patients (n = 59) or for patients undergoing surgery (n = 25). Most PROMs measured solely quality of life (n = 28), health-related quality of life (n = 21) or health status (n = 15). Both relatively unknown PROMs and much used PROMs such as the SF-36, the EQ-5D and the KOOS were included in the review.

Patient involvement
In the development of 74.1% of PROM patients were involved in some way, leaving 25.9 per cent of PROMs without any patient involvement ( Table 2). Patients were mostly involved in the item development (58.5%), especially using interviews (31.6%). Testing for comprehensibility closely followed item development (50.8%), while patients were never involved in determining which outcome should be measured.
However, in some cases, patients were not limited to just the predetermined outcome and all suggestions were included (1.0%), and in some cases, patients were involved in developing the domains or a framework (9.8%). These examples of patient involvement together suggest some limited patient involvement in determining which outcome to measure in some PROMs (10.9%). In 6.7% of the PROM development studies, patients were involved in all the development phases we documented. The level of patient involvement did not differ dependent on whether the PROM was developed as a measurement of a health outcome of surgery, cancer, chronic disease, mental health or a lifelong disorder (v 2 (6) = 7.063, P = 0.315). There is a small significant difference between the levels of patient involvement for the three time periods

Discussion
As patient involvement in the development of PROMs is essential for measuring patient relevant outcomes, 5-9 this study aimed to review whether and to what extent patients have been involved in the development of PROMs. In most PROM development studies, some patient involvement is recorded, but only in a few cases, patients have been involved throughout the development process. Surprisingly, in more than a quarter of PROM development studies, patient involvement was not documented at all. Other research looking at patient involvement in the development of PROMs for certain health problems suggests similar or even less favourable results for patient involvement. 7,17 Besides patient involvement in development phases, this review also investigated whether patient involvement increased with time. We expected that, as the patients' perspective is increasingly regarded as important, 2-5 patients would be more involved in recently developed PROMs. There appeared to be some differences in patient involvement, but none that suggest an increase of patient involvement over time. This is especially surprising because patient involvement is not only recommended by many researchers, 5-9 a few years ago it has also become a requirement if the PROM is used in medical product development. 26 However, in ISOQOL's reporting standards for the use of PROMs in randomized controlled trials, 27 the issue of patient involvement was rejected after consideration. Similarly, the COSMIN checklist, which assesses the methodological quality of studies on measurement properties of PROMs, 21 barely mentions patient involvement. It only asks to check whether the items are relevant to the study population and whether the questionnaire was pre-tested, for example using cognitive interviews. Apparently, patient involvement is still not always considered important enough to be included in reporting formats. Additionally, although many benefits may be gained by involving patients during the development process, patient involvement can also result in budget and time issues, 9,28 which may prevent PROM developers from involving patients. Differences in requirements regarding patient involvement and negative consequences of patient involvement may explain the variation we found in the level of patient involvement and the lack of a significant increase in patient involvement over time.
If patients are still not always involved, and if the involvement varies in intensity (from letting some patients comment on the questionnaire in writing after completing it, to the whole process of organizing focus groups, interviews and cognitive interviews), what does this mean for the validity of the questionnaires and for the use of their results by patients, health professionals and health insurers? Patient involvement is seen as essential to ensure the content validity of the questionnaire. [12][13][14][15] Involving patients by focus groups or interviews during item development ensures a better understanding of illness experi-ence 20 and could help challenge tacit models on which PROM designs are based. 29 Furthermore, the content of the questionnaire may be less relevant to patients if they are not involved in the development. This, in turn, may lead to a negative attitude towards the questionnaire and failure to answer the questions. All this can negatively affect the validity of a survey. 16 Based on the findings of this review, it appears that the content validity of many PROMs can be improved. This is troubling as PROMs are increasingly used to inform patients, to improve the care of individual patients and to assist purchasers. 2,3,30,31 PROMs may be used to improve the care of individual patients by increasing health professionals' awareness of, and ability to, address patients' concern. 32 However, to use PROMs for this purpose, it is essential that a PROM is used which reflects the patients' needs and concerns accurately. 33 Furthermore, patient information based on PROMs may not sufficiently cover aspects that are most relevant to patients.
In the Netherlands, as well as in the English NHS and the American Medicare and Medicaid system, the use of PROMs in models of payment by results is a particularly relevant issue. In the Netherlands in 2006, managed competition has been introduced. In this system, health insurers should contract providers based on quality and costs of care. To do so, health insurers need instruments for measuring the quality of the care delivered by providers. 34 Besides measures of effectiveness and safety and patient experience measures, 35 PROMs are slowly introduced in the Netherlands to assist health insurers in selecting the best providers for contracting. In theory, this is a good thing, as PROMs are meant to reflect the patient's perspective 1 and research suggests that patients have a different view on which aspects of their care are important to them. Identifying and understanding which aspects of care are the most important to patients may be the key to good care. 36 However, if the validity of PROMs may need to be improved and if PROMs do not always reflect the patient's perspective, health insurers may want to ask themselves whether they are going to select health providers for the right reasons.

Limitations
The present findings should be regarded with some caution because of study limitations. First, patient involvement was recorded using the documentation of the development process in the included articles. The results are therefore dependent on the quality and accuracy of the documentation. Patient involvement may go unnoticed if it is not documented in the article describing the development. Second, reporting standards may have improved over time and although patient involvement was recorded almost as many times in older publications as in recent publications, a lower standard of reporting may mean that levels of patient involvement in earlier studies have been underestimated. Third, not all development studies were documented in detail and sometimes PROM developers used one method for several development phases, which occasionally made it difficult to separate the development phases. However, to ensure that patient involvement was not underestimated, we recorded any suggestion that a method was used for more than one phase as patient involvement during several development phases. Fourth, as we wanted to compare PROMs on patient involvement during the three phases of development, we selected studies describing one or more of these phases. However, sometimes PROMs are further developed after the conclusion of these phases. As further development is very difficult to identify and the methods used for further development can vary greatly, we were unable to include this. Nevertheless, it is possible that for some of the PROMs included in this review, a higher level of patient involvement has been practised following the initial development. Fifth, although we were able to compare the PROMs on patient involvement in three development phases, we could not compare the PROMs on the level of patient involvement during these phases. The amount of information that is given in publications varies hugely and often lacks detailed information on patient involvement. Sixth, statistical analyses were performed on data which was not derived from a systematic review. Although systematic review guidelines were followed where possible, the included papers may not be a complete reflection of all PROM development papers, which may have implications for the interpretation of the results.

Implications
This review may have several implications for the use and development of PROMs. First, a consensus should be reached on how patients should be involved in the development of PROMs. In this review several studies, describing opportunities to involve patients were brought together to use as a guide for extracting data concerning patient involvement from the papers. However, to ensure that all PROMs incorporate patient involvement throughout their development phases, it is important that one, easy to use, guideline is created. Perhaps inspiration may be gained from best practice examples such as the Genetic Counseling Outcome Scale 37 and the Breast-Q, 38 where patients were involved throughout the development process. In the development of an instrument for fatigue in rheumatoid arthritis, patients were even given the role of research partners. 39 Preferably the guideline is developed involving researchers, professionals, relevant organizations and patients to ensure input from all stakeholders and a widespread acceptation of the guideline.
Besides a guideline for patient involvement in PROM development, the implementation of a guideline for how to report the methods used to involve patients in publications is also necessary. As patient involvement is necessary to ensure that the questionnaire measures the patients' perspective, 5-9 existing PROMs should be selected based on the level of patient involvement. However, certain information about the development process is necessary for comparing PROMs on patient involvement. Although an attempt at guiding researchers in reporting patient involvement has been made, 40 studies still offer varying amounts of information about their development process. This makes it very difficult to select a PROM based on its level of patient involvement. A guideline that is supported by all relevant parties and enforced by journals should help ensure that all publications include information needed to make an informed choice for a PROM. Third, besides patient involvement in the development of PROMs, patients may also be involved in the interpretation and presentation of PROM results. Patients may be involved in the interpretation of PROM results using importance ratings. Importance ratings may give an indication of patient's values, needs and expectations regarding health care. 41 Importance ratings enable the weighting of the PROM results according to the importance patients allocate towards aspects of the PROM. Furthermore, as patients can use PROM data to make an informed decision concerning health-care providers or treatments, 30,31 PROM results should be freely available and easy to read. Patient involvement in the presentation of the results could help easy use of PROMs by patients.

Conclusion
Although some patient involvement takes place in most PROM development studies, the level of patient involvement varies greatly. Furthermore, in more than a quarter of PROM development studies, no patient involvement was recorded. This lack of patient involvement throughout the development may have consequences for how well the questionnaire reflects the patient's perspective, which in turn may result in limited benefits for the use of PROMs in individual care, for making decisions by patients between treatments and health professionals and, finally, for rewarding health professionals or even selectively buying health care from certain health professionals. For PROMs to truly measure the patient's perspective, further attention to building and/or disseminating consensus on requirements for patient involvement in the development of PROMs is necessary and existing PROMs should be carefully selected on the level of patient involvement.