Using collaborative logic analysis evaluation to test the program theory of an intensive interdisciplinary pain treatment for youth with pain‐related disability

Abstract Intensive interdisciplinary pain treatment (IIPT) involves multiple stakeholders. Mapping the program components to its anticipated outcomes (ie, its theory) can be difficult and requires stakeholder engagement. Evidence is lacking, however, on how best to engage them. Logic analysis, a theory‐based evaluation, that tests the coherence of a program theory using scientific evidence and experiential knowledge may hold some promise. Its use is rare in pediatric pain interventions, and few methodological details are available. This article provides a description of a collaborative logic analysis methodology used to test the theoretical plausibility of an IIPT designed for youth with pain‐related disability. A 3‐step direct logic analysis process was used. A 13‐member expert panel, composed of clinicians, teachers, managers, youth with pain‐related disability, and their parents, were engaged in each step. First, a logic model was constructed through document analysis, expert panel surveys, and focus‐group discussions. Then, a scoping review, focused on pediatric self‐management, building self‐efficacy, and fostering participation, helped create a conceptual framework. An examination of the logic model against the conceptual framework by the expert panel followed, and recommendations were formulated. Overall, the collaborative logic analysis process helped raiseawareness of clinicians’ assumptions about the program causal mechanisms, identified program components most valued by youth and their parents, recognized the program features supported by scientific and experiential knowledge, detected gaps, and highlighted emerging trends. In addition to providing a consumer‐focused program evaluation option, collaborative logic analysis methodology holds promise as a strategy to engage stakeholders and to translate pediatric pain rehabilitation evaluation research knowledge to key stakeholders.


| INTRODUC TI ON
Pain-related disability affects eight percent of youth. 1,2 Within the pediatric pain context, pain-related disability is defined as pain which impairs youth's ability to perform age-appropriate activities relevant to daily life. 3,4 Due to the complexity of these impairments, intensive interdisciplinary pain treatment (IIPT), a specialized multidisciplinary rehabilitation intervention, is viewed as the treatment of choice. [5][6][7][8][9] To be considered an IIPT program, three or more disciplines (eg, pain specialist, psychologist, physiotherapist) must work together, in an integrated manner, guided by a shared rehabilitation philosophy. 7,10,11 The aim of IIPT intervention is self-management, whereby youth and their parents actively engaged in managing pain, and resume participation in age-appropriate activities. 12 Although these programs exist worldwide, their comparison and reproducibility are complicated by poor descriptions of the intervention components, and a lack of transparency in how the components produce the anticipated outcomes. 12,13 Moreover, stakeholders' perceptions of the value of these programs are missing from the evidence, rendering judgment of their worth difficult.
Integrated knowledge translation (IKT) is a model of collaborative research, where researchers and stakeholders engage together to produce mutually beneficial research and optimize healthcare delivery. 14 Stakeholder engagement is increasingly recognized as essential and believed to increase accountability, broaden the underlying value base, and enhance the relevance and utilization of the research findings. 15,16 However, how best to engage stakeholders is less well known. To date, stakeholder engagement in the evaluation of interventions, like IIPT, has been limited. [16][17][18][19] Interventions like IIPT are recognized as complex. According to the Medical Research Council, a complex intervention is described as one that contains several interacting components, requires various behaviors to be exhibited by both those delivering and those receiving it, incorporates different groups and organizations, and includes many different outcomes, all the while exhibiting flexibility or tailoring. 18 The interaction of these multiple components can be represented as a program theory, defined as the specific activities by which an intervention achieves its anticipated outcomes. 20 Furthermore, it can be illustrated by a logic model, a visual map of this theory. 21 Stakeholders have unique experience and knowledge of the contextual factors, and how these may have influenced the implementation of an intervention. 22 Without creating an in-depth understanding of how complex interventions work and under what condition, treatment outcomes become difficult to explain and are poorly understood. 23 Currently, an explicit theorization of IIPT and its context is lacking in the pediatric pain-related disability intervention literature. 12 Theory-based evaluation is an approach that may facilitate stakeholder engagement. 24 It aims to explain how and why programs work (or fail) in different contexts and for different stakeholders. 24 Logic analysis, a relatively new theory-based evaluation methodology, theorizes a program by mapping the links between the intervention components and the anticipated outcomes (ie, program theory), highlights contextual influences, and evaluates the plausibility of the program theory against existing evidence and experiential knowledge. 25,26 Logic analysis uniqueness lies in its theoretical examination of the core intervention characteristics, which must be present to achieve the desired outcomes, and in its identification of the critical conditions necessary for implementation and production of these outcomes. 25 It is useful in uncovering causal pathways that may be discernible but not always perceptible. 27 Furthermore, it helps reduce uncertainty about the program theory inherent to complex interventions, provides a preliminary evaluation of the theoretical and empirical foundation of the intervention, and is valuable in recognizing the strengths, weaknesses, and areas of improvement in the program theory. 25,26,28 Evaluations, using logic analysis, have yet to be applied in pediatric health or rehabilitation interventions, such as IIPT. Furthermore, some methodological gaps exist, including how to engage stakeholder. 29 In an attempt to broaden the application of this evaluation approach in pediatric health and rehabilitation, this article aims to provide details on the logic analysis methodology including the strategies targeting stakeholder inclusion, the data collected, and the analyses used. To do so, we will present an example of its application in a preliminary evaluation of an implemented IIPT for youth with pain-related disability and share the findings assessing whether this IIPT was theoretically designed to achieve its desired outcomes.

| Study context
With funding from a large philanthropic donation, the IIPT in Western Canada was conceived in response to a growing number of youth presenting with pain-related disability. This cohort-based IIPT was influenced by the day-hospital model described by Logan et al. 9,30 The 6-hour daily IIPT operated 5 days per week in a dayhospital setting and included individual, and group psychology, physical, family, occupational, art, music, and recreation therapies, as well as classroom time with a qualified teacher. Weekly nursing and physician consultations were also incorporated. All providers had specific training and experience working with youth with pain-related disability. Activities emphasized self-management knowledge acquisition and skill development, with a focus on restoring function and returning to age-appropriate activities. Treatment intensity and frequency, the disciplines involved, and the discharge timeframe were individualized and contingent on the achievement of patient-identified goals established at treatment commencement. Participants received on average 119 hours of scheduled treatment, with an average length of stay of 5 weeks. Once implemented, an evaluation was requested by decision-makers to determine the program's value and to identify any improvement recommendations.

| Study design
To determine whether the core intervention components and critical contextual conditions were present to produce the desired outcomes, a direct logic analysis was used. 26,27,29 This evaluation was part of a larger participatory study for which ethical approval was obtained.

| Participants
An expert panel of representatives from stakeholders involved in the treatment designed for youth with pain-related disability was identified by facility leadership and recruited via email invitation.
The 13-member panel consisted of five clinicians, a program coordinator, and healthcare manager, all of whom had experience (range 2-15 years) treating youth with pain and/or disability (eg, pain-related disability, cerebral palsy). Also included were two teachers with over 10 years of experience academically supporting youth with an array of physical and mental health conditions, two youth managing pain-related disability, and their parents. As no standards exist to guide the appropriate number of stakeholders to engage in a panel, guidance was gleaned from the consensus building literature, where a diverse group of 5-15 participants is recommended. [31][32][33]

| Procedures
To foster an environment conducive to stakeholder engagement, several activities preceded the evaluation process. First, a charter of the role and responsibilities was created and, once agreed upon, was signed by all expert panel and research team members. Additionally, educational resources and training sessions associated with the logic analysis methodology were provided (eg, logic model creation, scoping review processes). The 3-step logic analysis process described by Brousselle & Champagne 26 was then followed (see Figure 1). Table 1 provides a summary of the processes and procedures used in each sequential step. Additional details for each step are provided below.

| Step 1. Logic model construction
In this first step of the 3-step logic analysis methodology, three data collection methods were used to generate the data required to construct a stakeholder representation of the logic model. These included document analysis, stakeholders' surveys, and group discussions. All available historical documents (see Table 2 for full list) were analyzed. A stakeholder survey was developed by the research team guided by the semi-structured interview question for constructing a logic model proposed by Gugiu and Rodriguez-Campos 34 (see Appendix S1). Once developed, it was distributed electronically to the expert panel to supplement the document data. A form, founded on the logic model components and their definitions, was used for data extraction of the documents and a deductive analysis followed. 35 The same process was then repeated for the survey data. The extracted data from the document and the survey analysis were used to populate the various components (ie, resources, research, activities, process, outcomes, contextual factors) of a draft logic model. Six group meetings with the expert panel, facilitated by a member of the research team, were held for the purpose of gathering missing information about logic model components and to clarify inconsistencies. Using various F I G U R E 1 Association between the logic analysis steps and results communication strategies (eg, face-to-face, FaceTime, telephone, and email), all expert panel members participated in all six discussions. More specifically, at the first meeting, the program goal and objectives were discussed. A dialogue updating each logic model component, the linkages between the components, and the influential contextual features followed in the five subsequent meetings (see Table 1). New iterations of the logic model, based on expert panel feedback, were distributed between meetings, and the iterative process continued until agreement was reached. The sixth iteration was adopted.

| Step 2. Conceptual framework development
The purpose of developing the conceptual framework, the second step of the 3-step logic analysis methodology, is to examine the intervention's main components and determine whether the optimal conditions have been assembled to achieve the desired outcomes.
The aim is not to complete a systematic synthesis of the literature, but instead to create a representative synthesis of the most recent and meaningful evidence across various fields upon which the scientific validity of the logic model is examined. 26,29 To develop TA B L E 1 Summary of logic analysis steps, processes, and procedures the conceptual framework, the 6-stage scoping review process described by Levac et al 36 was followed and included the stages outlined in Table 1. A scoping review was the evidence synthesis method chosen as it summarizes a range of evidence in order to convey the breadth and depth of a field. 36 As suggested in logic analysis methodology, review studies were favored. 26 Further details about each scoping review stage are provided below.

Identifying the research question
The research question identified by the expert panel was founded in the primary objective of the IIPT, as identified in Step 1 of the logic model methodology. More specifically, the following question guiding the search: "What components should an IIPT designed for youth with pain-related disability adopt to promote self-management, self-efficacy and participation in age-appropriate meaningful activities?" Identifying relevant studies MEDLINE, CINAHL, and PsycInfo electronic databases were consulted using the following key words: chronic pain; pain-related disability; chronic conditions; disability; pediatric* or pediatric*, self-manag*; self-efficacy; participation. The target population was broadened to include youth with chronic conditions and disabilities for which pain is an important symptom, along with those with pain-related disability. It has been argued that youth with chronic conditions and disability share more comparable challenges than differences and that disease-specific orientations minimize the efficiency with which solutions for these challenges can be identified. 37

Study selection
To be included, studies had to incorporate youth, aged 12-18 years (as per the age inclusion criteria of the evaluated IIPT), be related to self-management, self-efficacy, and/or participation in meaningful activity (ie, leisure, recreation, or activities that promote productivity (eg, school, work)), and have a multi-or interdisciplinary focus.
Retrieved titles and abstracts were screened by two reviewers for relevance. Entire manuscripts were then examined. Reference lists were inspected, yet no additional studies were identified. Once completed, original manuscripts cited in the review studies were scanned for additional relevant information.

Charting the data
A data extraction form (as per the categories outlined in Table 3) and procedures were developed and validated by the research team.
Once consensus was achieved, the extraction process was completed by KH.

Collating, summarizing, and reporting the results
Data were coded, categorized, themed, and then culminated into a table format (see Table 4). An initial draft of the conceptual model was presented and discussed with the expert panel to explore the meaning, clarity, and consistency of the thematic interpretation.

Consultation
As identified in Table 1, the expert panel members were involved in the scoping review in the initial three stages of the review, provided consultation throughout the process, and assisted in the re-interpretation of the data in the context of IIPT.

| Step 3. Evaluation of the program theory
The third and final step of the logic analysis methodology consisted of comparing the constructed logic model with the developed conceptual framework. 26  Recommendations upon which consensus was achieved were then shared with hospital leadership.

| Program documents
Fifteen key program documents and 13 stakeholder surveys were used to construct the draft logic model. Although the documents contained many important program details, when closely compared, inconsistencies emerged (see Table 2  Education initiatives should target peers, classmates, teachers, and community leaders (eg, coaches) 46,49,59 Program activities Psychoeducation, combining information and skills training, is the focus of self-management interventions 38,55 Parent education, parent-to-parent support, and using parent coaching approaches are effective in fostering independence in youth self-management 39 Experiential approaches, varying delivery methods (group, individualized, Internet-based), peers learning opportunities, and skill mastery experiences should be provided 38-41 Communication, assertiveness, and advocacy training are a need identified by youth to promote shared decision-making with professionals 39,41 Opportunities for youth to create their own patient-professional relationships can be enriching 41 Peer-to-peer learning and mentoring is an emerging model showing promise 45 Activities that build independence, life, and leadership skills should be promoted 56 Opportunities for youth to create their own patient-professional relationships can be enriching 44,56 Self-awareness (eg, journaling), selfdirected learning (eg, web-based resources), and spiritual program activities, using a variety of learning methods and mediums (eg, health professionals, parents, Internet-based modules) should be included 44,45,57 Biofeedback, self-regulation, relaxation, mindfulness, cognitive-behavioral therapy, value-based goal identification nurture self-efficacy 58 Successful accomplishment of assigned tasks and generalization of prior successes, and graded exposure to fear-eliciting activities are also beneficial 58

TA B L E 4 Conceptual framework
Individualized and group-based interventions are effective when combined 48 Physical and leisure activity selection should be guided by mutually agreed upon participation goals and identified through coaching approaches 48 Training parents and youth on how to advocate for social inclusion and how to adapt and modify the activity and environment are effective strategies to minimize participation barriers 46 Sport and leisure activity counseling and social skills training should be available 48 Coaching on how to communicate about the condition and the supports required may be beneficial for this population in peer and school settings 46,48,49 More complex age-specific in-person sessions expanding social skills training to peer interactions, conflicts (eg, bullying), and intimate friendships may also be beneficial for older adolescents 59,60 Program outcomes Increased knowledge and skills in problem-solving, decisionmaking, and advocacy have been described 38 Improvements in self-efficacy, psychosocial well-being, and family functioning, along with reduction in social isolation, school absenteeism and pain have been demonstrated 41 Reduced family and parent burden, reducing healthcare utilization, and improving overall health outcomes and quality of life have also been reported 38 Benefits to physical, emotional, and school functioning have been recognized 42 Self-efficacy has been identified as a key contributor to chronic disease selfmanagement, to promoting of longterm behavior change, to improving the appropriateness of healthcare utilization practices, and to enhancing health quality of life 43 Participation improved academic performance, social interactions, mental and physical health, and helps develop life purpose and meaning 46,62 Creating the ideal context (Continues)  Table 2).

| Expert panel surveys
Contextual factors were also identified in the survey responses (see Appendix S3). Internal factors were linked to program structure and team dynamics, while external factors were related to building community-based partnerships and securing future program funding. Although these factors helped to further understand the context and the conditions deemed essential for success, questions remained.

| Group meetings
At the first expert panel group meeting, a new program objective drafted and distributed prior to the meeting was validated. The program objectives became "To provide youth with pain-related disability and their parents the knowledge, skills, and tools to self-manage their pain, build their self-efficacy, and promote their participation in meaningful activities, despite their pain." Furthermore, based on expert panel discourse as per the member below, the program reach was extended to include school and community personnel.
Our target population should include parents and the school, but also others in their community environment.

(Clinician 1)
Some activities and processes were omitted, while others were added, or further detailed. Program activities, which provided support, most valued by parents and youth were underscored. Youth also recognized activities that should be added to further improve their outcomes. Such activities focused on self-advocacy and the need to facilitate their transition back to their community following the program. The expected outcomes were adjusted and further elucidated based on panel member's experience.
In terms of long-term outcomes, it should be how much knowledge is retained. Because if you can refine the application of that knowledge; and once you build routines, you've found a way to make it work for you.

(Youth 1)
Finally, contextual factors believed to be essential for program success were discussed, and agreement was reached. These factors were associated with the preprogram screening, access to specialized health human resources, and participant characteristics. Figure 2 illustrates the final agreed upon logic model. Table 3 outlines the details of the 19 articles selected for the conceptual framework development and the deductive framework used to extract the data. All population samples included children and adolescents with a variety of disabling conditions for which pain is an important symptom.

Logic model components Self-management interventions Building self-efficacy Fostering participation
Program resources Program should be publicly funded 61 A variety of health disciplines with specific training and expertise in pediatric pain 7,12,61 A clinical and research training role, along with a public education (eg, school personnel) and advocacy mandate should be fulfilled by the program 61 Youth with variety of pain conditions, regardless of the type and origin, and their parents should be targeted 7,12,61 TA B L E 4 (Continued) Table 4 synthesizes the salient evidence of the conceptual framework, its relationship with both the logic model components, and the themes supportive of the program's key objectives. Further description is provided below.

| Conceptual framework summary
Promoting self-management Self-management, defined as a person's ability to acquire and apply the skills and knowledge to manage their symptoms, is learned with the support of one's family, community members (eg, friends, peers, teachers, coaches), and healthcare professionals. 38 Chronic conditions are experienced within the perspective of everyday life contexts (ie, peers, family, school, occupation, leisure, community). 38,39 Although medical management is important, emotional coping and role (social participation, occupation) management should also be considered. 40 Effective medical self-management is contingent on youth acquiring independence, knowledge, and skills. 41 Psychoeducation and skills training are the cornerstones of selfmanagement programs. 7,41 Parental education and parent-to-parent support are effective in addressing the gradual shift of self-management responsibilities to youth. 39 Support from social networks, including peers, has also emerged as a facilitator. [38][39][40][41] Many additional effective activities and promising emerging approaches are presented in the conceptual framework (see Table 4).

Building self-efficacy
Self-efficacy, defined as a youth's confidence in their ability to function effectively while in pain, 42 is critical to self-management, to appropriate healthcare utilization practices, and to enhancing health-related quality of life. 43 Effective activities for building selfefficacy were highlighted in the framework (see Table 4). Appealing to youth's preferred information seeking practices is considered pivotal to the process, with web-and application-based resources holding promise for this population. 44,45 Enhancing participation in meaningful activities Participation, defined as one's involvement in life situations (eg, education, employment, recreation, and community living), is an important pediatric rehabilitation outcome. 46,47 Social supports (eg, school personnel, peers) are important facilitators to achieving participation. 46 Moreover, effectively communicating about one's condition and requesting the supports required within various contexts (eg, in school, with peers) are important skills for increasing participation. 46,48,49 Other associated activities are presented in Table 4.

Creating the ideal context
Contextual conditions essential for program success were also found in the literature. Admission criteria across IIPT programs worldwide are similar, of which, pain impacting function, and youth and parent commitment to a self-management approach dominate. 7,12 Other contextual factors are highlighted in the conceptual framework (see Table 4).

| Evaluating the intervention theory
When detailed IIPT components, their links, and anticipated outcomes were systematically compared to the conceptual framework, generally speaking, the scientific evidence supported the program theory plausibility. Furthermore, interconnectivity between the three IIPT program objectives was illustrated. Below the IIPT program, strengths are presented, followed by recommendations for improvements.

| IIPT strengths
Regarding refining the self-management intervention for youth, our IIPT intervention aligned well with the evidence contained in the conceptual framework. As per the evidence, psychoeducation was acknowledged as a valued tenet of the program. Many With respect to fostering participation in meaningful activity, the IIPT included several components deemed effective based on the evidence. Sports, recreation and leisure counseling, advocacy education, and youth and parental training in activity and environment modifications were activities already incorporated in the IIPT and for which conceptual framework scientific support existed.
Transition meetings with school personnel, part of the current program discharge process, were acknowledged by youth and parent expert panel members as an opportunity to foster collaboration with teachers, which coincided with the conceptual framework evidence.
Youth expert panel members not only valued these meetings, they requested additional tools to further facilitate their ongoing advocacy initiatives in this context postdischarge.
Finally, concerning creating an ideal context to achieve the anticipated program outcomes the IIPT fulfilled many of the prerequisite conditions identified in the conceptual framework. When compared, the IIPT admission criteria, key program features, and team memberships shared many similarities with studies included in the conceptual framework.

| IIPT improvements
When comparing the logic model to the conceptual framework, three

| D ISCUSS I ON
The purpose of this article was to detail the logic analysis methodology and to share the findings of the program theory testing of an IIPT using this approach. As a collaborative IKT approach, this evaluation methodology proved helpful in many ways. First, logic analysis provided an opportunity to create a shared understanding of the complexity of IIPT among stakeholders, highlighting previously unidentified intervention and context interactive mechanisms. Stakeholder engagement was critical in ensuring the accuracy, validity, and the integrity of the implemented IIPT description. Engaging stakeholders in logic analysis has been previously recommended. 29 Particularly unique in our application of this methodology was the involvement of patients (ie, youth with pain-related disability) and their caregivers. The premise of engaging patients beyond the level of research subjects reflects a growing desire for more ethical, democratic, and moral practices. 52 However, the absence of parent and youth voices in the published evaluation of pediatric pain rehabilitation interventions, including IIPT, is a gap recognized by many. 17,19,38,53 In our evaluation, their engagement resulted in identifying youth and their parents' program expectations, as well as recognizing their ongoing challenges following program discharge.
Also noteowrthy was the causal mechanisms identified by youth and parent expert panel members, as experiential knowledge was acknowledge in the scientific evidence incorporated into the conceptual framwork. Building this shared understanding within the expert panel proved valuable in later prioritizing program refinements.
Furthermore, organization constraints highlighted by the health manager provided important insight into selecting recommendations that were feasible to implement within the program context. Third, the inclusion of expert panel members into the conceptual framework development could be enhanced. In previously described logic analysis processes, the conceptual framework phase was completed by the evaluator only. Although the expert panel members were included in many stages of the conceptual framework construction, incorporating stakeholders in the data extraction and theming processes of the scoping review could be added if appropriate oversight was provided.
Theory-based evaluation provided an opportunity to further detail the causal path of IIPT rehabilitation intervention, leading to a better understanding of these interventions, and evaluated the plausibility of the program theory in achieving its anticipated outcomes. Stakeholders were implicit to this process. The methods presented in this article, where scientific and experiential knowledge were weighed in a similar manner, provided a collaborative, pragmatic, and realistic approach, representative of the clinical environment in which most healthcare providers conduct evaluation. Engaging stakeholders, including parents and youth, in the logic analysis represents a catalyst for better understanding complex of pediatric pain rehabilitation interventions, such as IIPT, and their evaluations. Furthermore, it represents a novel IKT method to narrow the ongoing knowledge-to-practice gap existent in the field.

ACK N OWLED G M ENTS
We would like to thank all members of the expert panel who were so willingly sharing their opinions and expertise with us. The first author was also supported by the Vanier Canada Graduate Scholarship, by the Pain in Child Health (PICH) and the Canadian Child Health Clinician Scientist Programs.

CO N FLI C T O F I NTE R E S T
The authors have nothing to declare.