Using co‐creation and multi‐criteria decision analysis to close service gaps for underserved populations

Abstract Background Navigating treatment pathways remains a challenge for populations with complex needs due to bottlenecks, service gaps and access barriers. The application of novel methods may be required to identify and remedy such problems. Objective To demonstrate a novel approach to identifying persistent service gaps, generating potential solutions and prioritizing action. Design Co‐creation and multi‐criteria decision analysis in the context of a larger, mixed methods study. Setting and participants Community‐dwelling sample of older women living alone (OWLA), residing in Melbourne, Australia (n = 13‐37). Convenience sample of (n = 11) representatives from providers and patient organizations. Interventions Novel interventions co‐created to support health, well‐being and independence for OWLA and bridge missing links in pathways to care. Main outcome measures Performance criteria, criterion weights , performance ratings, summary scores and ranks reflecting the relative value of interventions to OWLA. Results The co‐creation process generated a list of ten interventions. Both OWLA and stakeholders considered a broad range of criteria when evaluating the relative merits of these ten interventions and a “Do Nothing” alternative. Combining criterion weights with performance ratings yielded a consistent set of high priority interventions, with “Handy Help,” “Volunteer Drivers” and “Exercise Buddies” most highly ranked by both OWLA and stakeholder samples. Discussion and conclusions The present study described and demonstrated the use of multi‐criteria decision analysis to prioritize a set of novel interventions generated via a co‐creation process. Application of this approach can add community voice to the policy debate and begin to bridge the gap in service provision for underserved populations.


| INTRODUC TI ON
Older women living alone (OWLA) may be burdened by complex care needs due to a constellation of co-morbidities that may include cancer, cardio-vascular disease, cognitive decline, incontinence, frailty, diabetes and their related complications. [1][2][3] Complex needs are typically met with complex treatment pathways and a sequence of transitions from community-based care to acute care to sub-acute care. 4 Bottlenecks, service gaps and access barriers may prevent or delay transitions and/or preclude a return to independent living. 4 Bottlenecks, service gaps and access barriers may be particularly problematic for social and community services (such as short-term home care, food and transportation services, or ongoing community-based care). For example, discharge home from acute or subacute care may not be possible if informal care from family members is not available and supply of affordable short-term home care is limited. 4,5 Put simply, "complex needs are often at odds with health care systems designed to treat patients needing acute episodic care" (p2123). 6 This has not gone unnoticed by policymakers and fundholders.
In Australia, the development of existing programmes for promoting health independence was motivated by recognition of the challenges faced by older people. 4,5 Older people living alone were a particular concern, partly because their complex care needs "…require greater emphasis on care planning and coordination in order to navigate the transition (from health services) back to the community and remain there safely" (p3). 4 In addition, non-government organizations and local government authorities have developed social care programmes to help older people stay well and avoid or delay transition to residential care. 7,8 Despite the availability of such programmes, navigating treatment pathways remains a challenge for older people with complex needs. Around one in five high-need older people in high-income countries report care coordination problems. 6 Additional financial pressures and logistical/psychological barriers may partly explain why older people living alone sometimes report lower levels of service utilization and worse patient care experiences than populations with comparable levels of need. [9][10][11] Older women may be further disadvantaged by structural barriers to economic and social participation. 12 This is not to exclude the fact that other populations with complex needs face barriers to participation and experience difficulties in navigating treatment pathways. 13,14 Here, OWLA represent one of many underserved populations that may benefit from a novel and person-centred approach to identifying persistent service gaps, generating potential solutions and prioritizing action.
The present study sought to address the particular challenges faced by OWLA in accessing health, social and community services that are consistent with their needs and preferences. To engage OWLA and stakeholders in this task, we integrated co-creation 15 and multi-criteria decision analysis (MCDA) 16,17 in a novel application of existing methods. The sections that follow (a) review existing methods and identify weaknesses that may be addressed by our approach to integrating co-creation and MCDA, and (b) describe and demonstrate the use of our approach for identifying persistent service gaps, generating potential solutions and prioritizing action in an OWLA population. Results include a set of "high priority" strategies that were specifically co-created to support the well-being and independence of OWLA and bridge missing links in pathways to care.
To inform future applications, we discuss several challenges we encountered in applying MCDA to the set of interventions generated by the co-creation process. Despite these challenges, we conclude that the methods described here can add community voice to the policy debate and help close persistent service gaps for underserved populations.

| BACKG ROU N D
A broad range of methods have previously been applied to (a) identify persistent service gaps, (b) generate potential solutions and/ or (c) prioritize action. To identify persistent service gaps, mapping the number of relevant services or providers against the number of potential users for different catchment areas can help to identify locations, services or location/service combinations with potential shortages. 18 Typically, this service mapping approach limits the set of potentially relevant services by first identifying all existing services in the relevant catchment areas and then applying exclusion criteria to remove services not currently offered to the population of interest. 18 As a result, service mapping is not well suited to identifying service gaps for which there is no existing solution or that have arisen due to a mismatch between service design and the needs and preferences of potential users.
To identify service gaps not identified by service mapping, consumer engagement can provide a patient-centred perspective 19 and is increasingly recognized as an essential element in service design and quality improvement. [20][21][22] Consumer engagement also has the potential to generate solutions and prioritize action. 20 While consumer engagement can take many forms, one recent example of good practice in consumer engagement (a) surveyed perceptions of service quality in a representative sample of service users to prioritize broad areas for improvement, (b) convened a Consumer Action Group of hospital staff and consumer representatives to develop and implement quality improvements and (c) used feedback from subsequent iterations of the consumer survey to fine-tune quality improvements and prioritize next steps. 19 When prioritizing areas for improvement in this example, respondents to the consumer survey allocated a fixed number of "points" across their top five areas for improvement (eg improved parking, information, hospital catering) to indicate a relative preference for changes in one area over another. When developing potential quality improvements, the Consumer Action Group followed an adapted version of the Breakthrough Series model, with an emphasis on collaborative learning rather than co-design or cocreation. While this approach has the potential to identify service gaps for which there is no existing solution and to develop novel solutions that meet consumer needs and preferences, it may prove difficult to replicate these outcomes without a structured process for co-creation and without a more detailed understanding of consumer preferences.
Integrating structured processes for co-creation and preference elicitation into an overarching consumer engagement framework therefore offers a potential remedy for the weaknesses of some existing methods. With regard to preference elicitation, MCDA can provide a detailed understanding of consumer preferences, including a description of potentially relevant performance criteria (components of "value" to consumers) and the relative weight given to these criteria by different types of consumer. This type of information can be used and re-used to predict how service improvements would be "valued" by existing users, potential users and partners/ carers, helping decision makers to navigate the often complex tradeoffs required when setting health-care priorities. 23 With regard to co-design or co-creation of services, existing frameworks offer structured processes for generating novel improvements or interventions that are consistent with consumer needs and preferences. 15,24 For example, one such process 24 was structured as a series of steps or phases: commencing with research and analysis "using qualitative methods… to gain deep insights into end users' context, wants and needs" (p20); progressing to idea generation "typically led by the design team" (p21); idea testing and refinement "using paper-or experience-prototyping" (p21); and concluding with evaluation and priority setting "to increase collective ownership of the outcome, and to encourage shared responsibility for progress" (p23). The use of this type of structured process is claimed 25 to generate improvements that are "more likely to be fit for purpose, acceptable, valuable and enduring" (p406) than alternative approaches to service improvement or redesign.
While existing frameworks for co-creation have clear strengths in identifying persistent service gaps and generating potential solutions, this is not the case when it comes to prioritizing action. To prioritize action, existing frameworks for co-creation typically rely on global assessments of "importance" or "value" 26 or simple tools to visualize the performance of potential improvements across just two or three dimensions. 24,27 Neither approach yields a sufficiently detailed description of consumer preferences for rational and transparent priority setting or for the purposes of predictive modelling.
Priority setting mechanisms embedded within existing frameworks for co-creation would therefore need to be repeated when additional improvements or new interventions come under consideration, and may be unsuitable to support funding decisions.
A mirror image of this trade-off operates for MCDA. Multi-criteria decision analysis delivers the detailed and durable understanding of consumer preferences that is missing from existing frameworks for co-creation but lacks a structured process for generating novel service improvements or interventions that meet consumer needs and preferences. The complementary strengths of MCDA and cocreation suggest that combining the two methods will deliver a framework for consumer engagement that carries all of the strengths and none of the weaknesses of one or other of these methods used in isolation.

| ME THODS
To provide an overarching framework for patient engagement, we adapted an existing framework for co-creation, 15 with MCDA employed to meet two of its six essential elements. Implementing the first four elements of the framework (listed below) generated ten co-created strategies/interventions to support health, well-being and independence for OWLA and bridge missing links in pathways to care: 1. Engage-establishing meaningful relationships with participants.
3. Explore-learning about experiences and identification of improvement ideas.
4. Develop-work with participants to turn ideas into improvements (ie strategies or interventions).
A detailed description of methods for generating our set of cocreated interventions has been reported elsewhere. 28 The final two elements of the framework (listed below) were completed using MCDA to "decide" which of the ten co-created strategies/interventions should be further developed and implemented, and "prepare for change": 5. Decide-choose which improvements to make, and how to make them (ie set priorities).
6. Prepare for change-turn improvement ideas into sufficient information to prompt action by key decision makers.
While the "decide" element of the framework could be undertaken with a number of different objectives in mind and using a broad range of methods, we took a person-centred approach and prioritized the co-created strategies/interventions in line with their "value" to OWLA using MCDA. To "prepare for change," the use of MCDA made two key contributions. First, it generated detailed information regarding performance criteria, criterion weights, performance ratings, summary scores and ranks reflecting the relative value of interventions to OWLA. Second, it provided a mechanism for reconciling client and stakeholder perspectives regarding which interventions should be considered "high priority." The present paper describes and demonstrates application of MCDA to our set of co-created interventions.
The interventions generated by the co-creation process potentially differed on several performance criteria including effectiveness in improving health, well-being and independence; out-of-pocket cost; location of service; and accessibility/availability. Moreover, there was potential for each of the co-created interventions to do better on some criteria but worse on others so that it was not possible to clearly identify an intervention that performed "best" across all criteria. In such circumstances, the relative weight given to each of the criteria matters and MCDA can be particularly valuable.
Multi-criteria decision analysis entails five steps: (a) identify the set of relevant interventions and the set of criteria against which interventions are to be evaluated, (b) quantify the relative importance of each of the criteria and express this information as "weights," (c) specify a measurement scale for each criterion and evaluate the performance of each intervention on each of the criteria to obtain "performance ratings," (d) combine weights with performance ratings to obtain a "score" (typically a weighted average) summarizing each intervention's performance across all criteria, and (e) rank order interventions based on their summary scores. 29,30 By reducing the complex task of priority setting to this series of relatively simple steps, MCDA can "help organize and distil the large amount of information required for effective and defensible decision-making" (p128). 29 In the present application, our MCDA exercise included two iterations of Step 2, with feedback and discussion between elicitation of first-try and final criterion weights to promote deliberation and produce "better" weights. 31 In other studies, MCDA has also included a Step 6 wherein the rank order implied by weighted averages has been validated or revised by key stakeholders during group discussion of criteria, weights, performance and priorities in a deliberative process. 30 MCDA has previously been used to incorporate patient preferences into health-care decision making for a wide range of diseases and interventions. 17,23 For the present study, an online

| Step 1-Identifying relevant interventions and performance criteria
The ten interventions generated by the co-creation process (Table 1), plus a "Do Nothing" alternative, formed the set of relevant interventions for the MCDA exercise. Here, "Do Nothing" characterizes the set of services, programmes and interventions currently available to respondents ("do nothing extra" rather than the absence of intervention or services).
To identify relevant performance criteria, semi-structured interviews were conducted with 37 participants (mean age: 73 years, age range: 57-94 years; owner-occupier: 68% [25/37]; retirement village: 30% [11/37]). We identified 26 themes in interview scripts, not all of which could be characterized as criteria for distinguishing between preferred and less preferred interventions. For example, the theme of "marital life" was referenced 97 times by 57% (21/37) of participants but these references were predominantly descriptive of participants' past marital life and transition to current living arrangements. Similarly, the theme of "retirement/work life" was referenced 99 times by 46% (17/37) respondents but these references were predominantly descriptive of participants' past employment, health-related barriers to employment and transition to retirement.
Other themes of peripheral relevance to the task of identifying relevant criteria for evaluating alternative interventions included "volunteering/charity work," "thoughts on living alone," "thoughts on retirement villages," "social services accessed," "services accessed (other than social services)" and "current service gaps." To identify performance criteria of particular relevance to OWLA, we excluded themes of peripheral relevance and then ranked remaining themes by total references and by number of respondents referencing each theme. We then mapped themes to criteria suitable for evaluating alternative interventions. For example, general cost of living, cost of housing, household budgeting and out-of-pocket cost as a barrier to accessing health and social services were referenced 78 times by 57% (21/37) of respondents. We collected these references into a "finance" theme and mapped this theme directly to "Finances" in our list of potentially relevant criteria. In other cases, multiple themes could be mapped to the same criteria. For example, the themes of "support independence," "transport" and "importance of living independently" were mapped to "being independent" in our list of potentially relevant criteria. This mapping is analogous to "bottom-up" construction of the value tree, clustering characteristics of interventions into higher-order objectives that are themselves subcomponents of overall value. 16 Table 2 ranks potentially relevant themes by total references and number of respondents referencing each theme. Table 2 Table 3 describes characteristics of the study sample in which we elicited criterion weights (Step 2) and performance ratings (Step 3). Table 4 describes "first-try" and "final" criterion weights for OWLA and stakeholder participants. Appendices S1 and S2 provide a detailed description of methods for elicitation of "first-try" and "final" criterion weights.

| Step 2-Weight criteria
For OWLA participants, "Health & Wellbeing" carried the highest weight (most important), followed by "Being Independent," "Safety & Trust," "Finances," "Accessible," "Enjoyment" and "Relationships" (least important). Differences between OWLA and stakeholder weights were small and statistically insignificant for the first-try elicitation (t ≤ 1.21, P ≥ 0.24). However, these small differences in importance weights still resulted in differences in importance rankings.
For stakeholder participants, "Being Independent" carried the highest weight (most important), followed by "Health & Wellbeing," "Safety & Trust," "Accessible," "Finances," "Enjoyment" and "Relationships" (least important). The minor differences between OWLA and stakeholder weights were further reduced after deliberation, with the closer correspondence in final weights due mostly to a shift in stakeholder weights. This closer correspondence in final weights was also seen in importance rankings, with OWLA and stakeholder importance rankings based on final weights matching importance rankings based on first-try OWLA weights. When movements in OWLA and stakeholder weights were pooled, the largest changes in weight between first-try and final elicitations were observed for "Health & Wellbeing" (+2.6%) and "Safety & Trust" (−2.1%). Table 5 describes performance ratings for each intervention against each criterion. On average, "Do Nothing" was rated lowest across all criteria by both OWLA and stakeholders, largely as a consequence of the fact that co-created interventions "added something" to the "Do Nothing" alternative. For positively valued criteria, "adding something" might plausibly result in better performance, provided that the relevant intervention "does no harm" with respect to the relevant criteria. Neither OWLA nor stakeholder samples identified a dominant intervention that performed best across all criteria. For OWLA, "Handy Help" received the highest mean rating on "Health & Wellbeing," "Finances," "Being Independent," "Accessible" and "Safety & Trust" but "Good Neighbour" received the highest mean performance rating on "Relationships" and "Exercise Buddies" received the highest mean rating on "Enjoyment." For stakeholders, "Exercise Buddies" received the highest mean rating on "Health & Wellbeing," "Enjoyment" and "Relationships" but "Handy Help" received the highest mean rating on "Finances," "Being Independent" and "Safety & Trust."

| Steps 4 and 5-Summarize performance across
all criteria and rank from "best" to "worst" Table 6 summarizes scores and ranks for our set of interventions.
Differences between OWLA and stakeholders with respect to firsttry criterion weights and with respect to performance ratings were carried through and reflected in differences on intervention scores and ranks. When intervention scores were calculated from OWLA data, the most preferred intervention was "Handy Help," followed by "Volunteer Drivers," "Post-Op Stay," "Exercise Buddies," "Good Neighbour," "MyServiceMap," "Hour-4-Hour Barter," "Shortcut to Services," "Friendly Visitor," "Safe Boarder" and "Do Nothing" (least preferred). The top two interventions remained the same when intervention scores were calculated from stakeholder data but we see "Post-Op Stay" drop to fifth position and a reordering of lower ranked interventions.
The closer correspondence between OWLA and stakeholders on final weights also resulted in a closer correspondence in intervention scores and ranks. When intervention scores were calculated using TA B L E 1 Co-created interventions

Label Description
Mobility and ability "Handy Help" Handy person to do small tasks such as change a lightbulb, replace batteries in smoke detector or flip mattress "Post-Op Stay" Patient transfer to 3-star hotel for overnight stay post day surgery or minor procedure, with nurse on duty "Exercise Buddies" Find-a-buddy service to match older women to an exercise buddy with similar needs and interests Transport "Volunteer Drivers" Volunteer drivers using private vehicle, with no restrictions on reason for travel (ie not just medical appointments) or time spent at activity Social connections/participation "Good Neighbour" programme Letter from local council or MP, encouraging people to connect or reconnect with older residents in their community "Friendly Visitor" programme Volunteer visiting programme, matching volunteers with older residents in the local community

| Principal findings and innovations
The present study demonstrated a novel application of co-creation 15  Total references for all n = 37 respondents and mean (SD) references per respondent.
preferences at each step of the co-creation and MCDA processes.
In previous applications of MCDA, criterion weights have sometimes been derived using relatively complex quantitative methods such as discrete choice experiments (DCEs). 29 Some studies have flagged potential difficulties in using this approach to obtain weights from community members. 16

| Limitations
The present study offers a number of lessons for researchers and policymakers. First, the co-creation process yielded a set of 10 interventions, designed in collaboration with OWLA participants to  (29); 5th a Calculation of scores and ranks was based on each participant's criterion weights and performance ratings, except for n = 1 from the stakeholder group where missing data for performance ratings were replaced with mean performance ratings for stakeholder participants.
TA B L E 6 Intervention scores and ranks complete elicitation tasks and one-on-one computing assistance to access and navigate the decision support software. The level of support required suggests that-for an OWLA population-postal or remote online administration of an MCDA exercise is unlikely to prove fruitful. Researchers considering application of our approach in other underserved populations may encounter similar barriers but should give due consideration to online administration in a larger sample if levels of computer literacy and education permit.
Finally, feedback and discussion between elicitation of "firsttry" and "final" criterion weights were designed to promote deliberation and with the intent of producing "better" weights. For unfamiliar or complex trade-offs, preferences (eg as reflected in criterion weights) may not be fully formed or readily accessible and may instead need to be "constructed" by respondents.

| CON CLUS ION
The present study described and demonstrated a novel approach to integrating co-creation and MCDA. Results suggest that OWLA continue to encounter service gaps and barriers to service use. This is despite the introduction of government programmes designed to facilitate and coordinate care for patients with complex needs. Results also suggest that giving voice to community preferences via the approach described here may help to close persistent service gaps and lower access barriers. Further work is underway to ensure that these community preferences are given due to weight by decision makers.

ACK N OWLED G EM ENTS
We gratefully acknowledge the contribution of all participants. This work was supported by a Lord Mayor's Charitable Foundation-Innovation Grant (Healthy & Cohesive Communities), City of Melbourne, Victoria, Australia.

CO N FLI C T O F I NTE R E S T
None.

DATA ACCE SS I B I LIT Y
The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions.