• Open Access

Using AI to Understand Key Success Features in Evolving CTSAs

Authors


Correspondence: JD Kusch (jkusch@mcw.edu)

Abstract

A vital role for Clinical and Translational Science Award (CTSA) evaluators is to first identify and then articulate the necessary change processes that support the research infrastructures and achieve synergies needed to improve health through research. The use of qualitative evaluation strategies to compliment quantitative tracking measures (e.g., number of grants/publications) is an essential but under-utilized approach in CTSA evaluations. The Clinical and Translational Science Institute of Southeast Wisconsin implemented a qualitative evaluation approach using appreciative inquiry (AI) that has revealed three critical features associated with CTSA infrastructure transformation success: developing open communication, creating opportunities for proactive collaboration, and ongoing attainment of milestones at the key function group level. These findings are consistent with Bolman & Deal's four interacting hallmarks of successful organizations: structural (infrastructure), political (power distribution; organizational politics), human resource (facilitating change among humans necessary for continued success), and symbolic (visions and aspirations). Data gathered through this longitudinal AI approach illuminates how these change features progress over time as CTSA funded organizations successfully create the multiinstitutional infrastructures to connect laboratory discoveries with the diagnosis and treatment of human disease.

Introduction

As science grows more complex, the “NIH Roadmap” purposely focuses on efforts that engage researchers, scientists, and clinicians to ensure both efficient and effective discovery.[1] The complexity of today's biomedical research problems requires scientists to move beyond their own discipline and explore new organizational models for team science.[2] The Clinical and Translational Science Award (CTSA) is expressly designed to link scientists, clinicians, communities and industries to expedite the application of research findings into real world practice.[3] At no other time has there been such a need for a robust multidirectional flow of information to support advancements in knowledge of biologic systems and the development of new tools applied at the bench and the bedside.[2] CTSA evaluators, primarily through the use of quantitative strategies, are beginning to understand the critical features of the multidirectional flow of information. Complimenting traditional “metrics” associated with quantitative measures, evaluators are employing qualitative approaches, like appreciative inquiry (AI) to gain insights into the dynamic and structural components of the organization and the participants. Gaining this understanding is even more critical when a CTSA is composed of different organizations (e.g., universities, medical colleges, health care systems), as each brings its own structure and culture to the partnership. AI methodology explicitly facilitates this cross-organizational understanding which can further foster innovations in social-organizational arrangements and processes.[4] This report describes how an AI evaluation strategy can result in new knowledge, models, and actionable intelligence[5] to guide and shape CTSA organizational infrastructure success.

Methods

The AI approach implemented by the Clinical Translational Science Institute (CTSI) of Southeastern Wisconsin uses one-on-one qualitative interviews conducted by an interdisciplinary, interinstitutional team. Each team member, recommended by their institutional CTSI executive committee representative as highly respected and a skilled interviewer/listener, conducts one AI interview every 3–6 months. Interviews are conducting using a standard protocol consisting of three major sections: successes, the ideal CTSI, and challenges (Figure 1). Interviews are recorded for quality, transcribed and analyzed by a qualitative analyst and confirmed by team members. To date, interviews have been conducted with CTSI leaders and trainees. Interview protocols were recorded, transcribed and analyzed using standard grounded theory approach to qualitative analysis with verification of themes by interviewers.

Figure 1.

Major AI interview protocol sections.

To broaden the utility of the findings, CTSI identified an evidence-based and well-established organization analysis model. Bolman and Deal's model uses four frames of reference to examine and understand organizational frameworks: (1) structural (organization of groups and structures); (2) human resources (maximizing group dynamics, training); (3) political (managing power and conflict, resources); (4) symbolic (aligning values with actions).[6] These thematic results will be presented within this interactional framework.

Results

To date, three thematic findings have emerged. Consistent with qualitative reporting methodology each theme will be presented using brief narratives of successes and opportunities. Each theme will be further elucidated by providing in parentheses the applicable Bolman and Deal frames to highlight that success emerged from the synergy obtained by attending to at least three of the four frames.

Developing open communication

Open communication connects partnering organizations together and ensures that each institution actively supports the shared CTSI goals and functions. However, it is a developmental process that evolves as mutual respect and trust grow, as indicated in the following AI interview excerpt with a CTSI key function group director:

“I can easily speak with the new chancellor at [name of partner institution]…about what is going on in the CTSI, and he will know exactly what I'm talking about. That is not…what we previously had or something that I would expect. [Being] open at the executive committee level about problems and perception[s]…has had [an] incredibly, positive effect.”

Open communication is facilitated by ongoing interactions among CTSI members through regular contacts/meetings. On-going interactions provide opportunities for CTSI members to communicate about new developments and creative synergies. More specifically, successful meetings occur at regular intervals, consider travel limitations, engage in a shared identification process of priorities and action items, and involve willing members from a wide variety of institutions and departments. These meetings are organized, focused, solution-oriented, and foster an environment of open communication to support and promote innovation.

Opportunities for such dialogue are facilitated by formal focus group discussions like the one described in the quote below from another key function group director AI interview:

“One of the most satisfying experiences I had was the first focus group we did with a group of veterans representing the veterans’ organizations. We talked with them about their understanding of clinical and translational research, how they would be involved ….They were so anxious, and so well informed…to become involved and to talk about the importance of research to them… That was one…of seven or eight focus groups that we did that with. Some of the ideas that are coming out of those focus groups are now coming to fruition into Science Cafés …. I think it is just coming to that common understanding and then capitalizing on that knowledge that they have.”

As this excerpt demonstrates, becoming directly involved with stakeholders at multiple levels (political, structural, human resources) fosters communication and generates new approaches. The willingness to listen to the individual needs both inside and outside of the partnering institutions (symbolic), makes people feel their time is valued and leverages knowledge across resource users.

Creating opportunities for collaboration

Successful collaborations can emerge at CTSI sponsored events which are explicitly designed to promote successful collaborations. AI interview data reveal that these event-stimulated collaborations emerge when there is multidisciplinary participation, senior oversight of projects, getting interested researchers to attend (symbolic, structural, political, human resources), and explicitly designed opportunities (e.g., breakout, buzz groups) that promote conversation and exploration by researchers. Such planned events draw people from partnering institutions and beyond. The following excerpt from a pilot grant awardee interview illustrates these elements:

“Several years ago…there was a CTSI workshop where they brought together a number of different people who were interested in pursuing functional imaging…. I was placed at a table with other people who studied movement and were also interested in functional imaging…. We had an opportunity to sit at that table and just brainstorm about possible projects that could be interactive and collaborative. [T]here were two people who were at that table who honestly were people I probably should have reached out to before to collaborate, but just hadn't made the time. That sometimes can be hard…when your day is busy and you just don't bother to do it…. Coming together at the meeting that the CTSI had planned, was the seed crystal that got this collaboration going…. We kept meeting and we knew that there was an opportunity to submit a funding proposal for the CTSI several months later and…we ended up…creating a proposal to examine differences between upper extremity and lower extremity control–how the nervous system controls the upper limbs as compared to the lower limbs using functional magnetic residence imaging.”

Another CTSI structured collaboration effort resulted in the formation of the Milwaukee Evidence-Based Practice Institute, comprised of clinicians and educators who meet several times per year to explore best practices for evidence-based instruction for students enrolled in programs at CTSI partner institutions. These illustrative success stories highlight the types of outcomes that may not be otherwise captured through standard tracking and evaluation efforts. The natural dialogue of AI interviews captures these experiences and outcomes in radiant details and can be used to highlight CTSI outcomes.

Ongoing attainment of milestones at the key function group level

Early interviews (i.e., those collected within the first 6 months of receiving the CTSA) reflect a broad sense of success (symbolic). Analysis of these interviews reveals generalized optimism that the CTSI goals and visions will become reality, while acknowledging the need for time to crystalize and actualize each one. Details regarding CTSI related infrastructure processes and strategies in these early interviews was vague. Over time, the generalized stories of successes and projected successes continues, but interviewees began sharing more detailed, focused stories of measurable success (political, structural, human resources) evolving around the crystallization of shared values and visions (symbolic). Two early success interviews from key function group codirectors illustrate this finding. “During the past year I have participated with CTSI faculty on a number of proposals which never ever would have happened if it had not been for the CTSI…I've actually enjoyed those interactions.” “I think we are building partnerships…quite well with UWM [and] Marquette, which will be evolving. But this one…started a little bit early so I would say a higher peak position than some other programs or other relationships.”

A recent AI interview with another key function group co-director illustrates the “crystallization” that emerges over time. “One of the things that really might turn out to be extremely important is our interaction with [name of partner institution]. They are doing physical therapy on stroke patients and they are using MRI to look at the degree of stroke damage. One of the patients they were studying—half of her brain was gone. With a MRI, we could not even see it.… Yet, she was functioning as ski instructor. So, obviously she was able to use both limbs effectively. Normally we think that the only way that happens is through crossover from the cortex. From that one experiment we learned that crossover can occur in lower brain areas and the brain stem and spinal cord. That could be revolutionary!” As the CTSI grows and matures, so too do perceptions of heightened appreciation and recognition of successful outcomes experienced by participants.

Discussion

Benefits of AI as an evaluation strategy for CTSAs

A valuable attribute of AI interviews is the candid communication among interviewer and interviewee. The semistructured nature of the AI interviews mimics natural conversation between two people. Interviews elicit sincere emotions and sentiments surrounding critical successes and challenges impacting clinical translational research processes. The anonymous interview approach provides a “safe” platform for interviewees to openly share views and perceptions.

CTSI stakeholders, including executive committee members, key function group directors, and staff receive thematic findings from the interviews through oral and written reports. The findings serve as focal points for discussion surrounding how to improve the CTSI. Findings also provide the impetus to celebrate successes among all CTSI members (symbolic). By reviewing the findings, the principal investigator and other leaders gain new insights which they report were not previously available. These new insights may hold the key to identifying ways in which translational research infrastructure processes and resources may enhance the quality and efficiency for translational investigators and other CTSI resource users.

As CTSAs strive to identify and streamline the translational research infrastructure and processes, an AI approach provides a deeper awareness and understanding of these processes. For example, while communication is a commonly identified organizational success feature, pragmatically it can be achieved by simply enhancing communication practices within those key function groups that play a role in enhancing the efficiency in which new infrastructure changes occur such as biomedical informatics or resources.

The CTSI of Southeast Wisconsin currently serves as a model CTSA for bridging IRB processes through joint agreements across seven major health and education institutions in Wisconsin. The efficiency of this process is attributed to a number of factors including the meeting and communication practices identified through AI interviews, per the key function leader(s). In the near future this IRB agreement will extend to additional areas in Wisconsin achieved by using the same communication and efficiency strategies identified through the AI process. As a success-oriented model, AI conditions individuals to realize the valuable ingredients involved in achieving success. Through this process, they become engaged in a shared process of imagining what could be—a responsibility traditionally reserved for those in leadership roles. Identification of success strategy ingredients can be shared with others in the CTSI to recreate future successes. Beyond demonstrating that success requires the dynamic interaction involving all four frames (structural, political, human resources, symbolic), qualitative data reveal the progression of perspectives as a CTSA funded organization successfully creates the infrastructure to support translational science.

Conclusion

CTSAs represent a unique opportunity to facilitate rapid movement of research into useful patient treatments and interventions. To make this vision a reality, CTSAs must utilize evaluation approaches which capture successful practice(s). AI represents a unique approach to facilitate the process of gathering rich information about processes and procedures in sufficient detail to inform action. The need for more qualitative measures has been recognized and realized through the recent approval and formation of the CTSA Qualitative Evaluation Methods Interest Group comprised of evaluators and other key function group members representing over 20 CTSA affiliated institutions. Over time, this workgroup may track and compare findings to assess the generalizability and increase the robustness of AI findings to inform CTSA stakeholders. In summary, an AI evaluation approach promotes widespread communication and vision building within and among CTSAs, promotes joint identification of challenges, and provides a clear mechanism for formatively identifying and evaluating quality and efficiencies within and among CTSAs.

Acknowledgments

This publication was supported by the National Center for Advancing Translational Sciences, National Institutes of Health, through Grant Number 8UL1TR000055. Its Contents are solely the responsibility of the authors and do not necessarily represent the official views of the NIH.

Ancillary