Using a Systems Change Framework to Evaluate Academic Equity & Climate Efforts in a Graduate Program

Statistics on representation in graduate programs show that, while academia is moving forward in terms of generating diverse cohorts of students and faculty, representation still has not reached parity. These discrepancies in representation are seen at the graduate student level and intensify as academic rank increases. While there have been strides to improve representation through more thoughtful recruitment, a new discussion is emerging around inclusion and retention of under-represented minorities. Inclusive programs are that which center and prioritize support for diverse experiences, identities, career goals, and perspectives actively and continuously from recruitment through graduation. An emerging area of focus for inclusion efforts is graduate student programs. Graduate education programs provide significant opportunities for inclusive programming, and conversely, a program that does not take efforts to improve both diversity and inclusion can often contribute to further disparity in representation. While there are many efforts across programs to address inclusion, there is room for improvement on developing cohesive and effective programming that targets the many areas of needed change in order to improve institutional inclusivity. Here, we argue that graduate education programs should utilize a systems change framework to evaluate areas of progress and need in their program as it relates to inclusion. A systems-change approach emphasizes three levels of changes: explicit change (e.g. policies), semi-explicit change (e.g. power dynamics), and implicit change (e.g. biases). We use the Ecology, Evolution, and Behavior (EEB) PhD Program at the University of Texas at Austin in an exercise to (1) identify areas of concern regarding inclusive programming voiced by graduate students and (2) categorize efforts to address these concerns about inclusive programming into a systems change frame, and finish by (3) integrating and evaluating which areas of the systems change framework show the greatest progress and greatest need for the UT EEB graduate program. We acknowledge that the specific examples here are of particular relevance to other EEB programs, as they may see similar patterns in graduate student needs and efforts to address them. But, the exercise itself is certainly not limited to EEB programs. We encourage any graduate program, as well as any departments or even larger institution, to consider undergoing this exercise in order to more effectively address inequity in their own domains.


General Background
Since 2008, women have earned the majority of doctoral degrees in biology and approximately 60% of biology baccalaureate degrees (NSF, 2019). In 2016, 42% of baccalaureate degrees and 32% of doctorates in biology were earned by underrepresented minorities (NSF, 2019). Thus, the elementary demographics of biology doctorate earners is roughly representative of the US population as a whole, which was 51% women and 36% non-white in 2017 (although much work remains beyond these most crude categorizations; NSF, 2019). This representation is dramatically reduced at the faculty level: 35% of biology faculty are women and 25% are people of color (among full professors 15% are people of color; NSF, 2019). If faculty demographics were representative of the theoretical PhD applicant pool, we are living in 1987 (the most recent year when women accounted for 35% of biology doctorate earners; NSF, 2017). So why is biology academia 30 years behind?
During graduate school, PhD students first experience and internalize their future career and lifestyle in academia, and most say choose another career path. That's acceptable--a successful PhD program prepares students for the myriad careers doctorate holders in biology eventually pursue. What's must change is that those leaving academia are disproportionately women (Martinez et al 2007) and underrepresented minorities (Allen-Ramdial & Campbell, 2014) ( Figure 1). These marginalized students use all the intelligence that gained them acceptance into their doctoral program to learn the many ways in which the system is not built for them. This is a product not only of their own experiences, but in the keenly observed experiences of other students and representative faculty. When diverse perspectives are lost from the academic system, the quality and scholarship of the institution is diminished. The loss of competitive colleagues and the dampening of academic aspirations occurs, not for scientific reasons, but because crises arise and the safety net is insufficient, nonexistent, or deployed too late.
As third and fifth year biology PhD candidates ourselves, we lack the direct power to enact changes that would reform the system on an institutional level. Availability of mental health resources, handling of harassment and misconduct cases, selection of administration, and family leave policies are all in dire need of systemic reform by those with administrative power.
While advocating for these institutional reforms, we believe we can counteract negative graduate student experiences by focusing on departmental culture and climate reform via thoughtful and inclusive data collection on the quality of the graduate student experience, followed by implementation of actions based on those data.
We include examples of our efforts in the supplemental materials, our goal here, however, is not to summarize or elaborate on those efforts but to critically evaluate them.
Many graduate programs in biology have already created and implemented spaces, techniques, conversations, and policies to improve graduate student well-being, particularly for under-represented students. But what implemented actions have been particularly effective at instilling long lasting change? How do we evaluate inclusive graduate programming, and identify the areas of greatest need?
We propose utilizing a systems change framework to evaluate areas of need and progress in graduate student programming. A systems change framework is not a new idea (Coffmann, 2007), but here we argue that it is a particularly useful framework for biology programs to critically evaluate the challenges that graduate students face, particularly as it relates to mentorship, diversity, and inclusion, and then develop programs to address them.
Here, we demonstrate the value of a systems change framework in identifying the areas of progress and areas of need in a graduate program by using systems change conditions to categorize the most common concerns brought up by graduate students. We then categorize recent programmatic efforts to address those concerns into the same conditions. We use this paired framework to illuminate areas of progress and areas in need of increased focus for future program development.
The specific examples here are particularly targeted to patterns of need and development in biology programs, but the exercise itself is certainly not limited to a specific type of program. We encourage any program, department, or even larger institution to use this exercise to address areas of progress and need in their own domains.

What is a Systems Change Framework?
The theory of systems change is designed to reform the underlying conditions in a system as they relate to social change, diversity, and inclusion, and was originally conceived in activist pedagogy (Coffmann, 2007). Its early applications centered around access to resources related to physical and mental health in early childhood development, and recently has become more used in corporate management areas (Kania et al 2018). The systems change framework is a construct intended to organize and evaluate the needs and corresponding efforts of a community.
The systems change framework itself is a descriptive set of interconnected spheres or categories of influence of a "system" -for example, a program, department, school, business, organization, or initiative. A common thread in systems is that they operate on many organizational levels (e.g. individual, community, state), often have a variety of funding sources, and must "tackle difficult deep-rooted problems such as gaps in services and outcomes based on race, income, culture, and language" (Coffmann, 2007). We hope to provide a clear format to help graduate programs tackle such problems.
The literature on systems change varies in nomenclature and the number of categories, here we choose to utilize the framework described in Kania, Kramer, and Senge (2018) "The Water of Systems Change." Kania et al. highlight six "conditions" or areas of systems change that fall into three categories: explicit, semi-explicit, and implicit. We adjust the definitions to the six conditions used by Kania et al in terms of specificity to graduate programs (Figure 2).

Description of Systems Change Graduate Programming Exercise
Here we design an exercise that programs can undertake to evaluate their own needs and existing efforts and then identify areas where additional work is needed. The exercise consists of three primary parts: 1. Data collection: Regularly assess the most common concerns or needs expressed by trainees in the program 2. Identify currently existing or proposed efforts developed to address those concerns/needs. Organize those efforts into the categories of the systems change framework 3. Evaluate the areas of overlap and limitations from steps 1 and 2. Prioritize which categories of the systems change framework show the greatest potential for progress in tackling concerns, and conversely which categories show the most urgent need for additional attention In the following sections we walk through this exercise using our program: the Ecology, Evolution, and Behavior (EEB) Graduate Program at the University of Texas at Austin, as an example. We believe this program is an effective example because we are addressing challenges likely present in other similar programs, and our demographics are roughly similar to national averages.
Step 1. Identifying areas of concern through data collection and assigning framework categories In terms of demographics, the UT Austin EEB Graduate Program is not immune to the larger deficits in representation described previously. The graduate students are 52% female and yet the supervising faculty are 34% female overall, with only 22% of the senior faculty being female. This discrepancy in representation is reflected across several pools of data: for example, comparing the gender of the admitted graduate students with that of the faculty applicant pool (Figure 3). These and a host of other data were collected from the College of Natural Sciences Dean's office, the Integrative Biology Departmental administrative staff, and the EEB Program coordinator, all using a request form (See Supplemental Material). We drafted a list of the most common concerns expressed by graduate students in the program based on the previously described demographic data, the personal experiences of the authors, conversations regarding experiences of other students, and a comprehensive climate survey developed in 2018 and administered for the department by one of the authors (supplemental material 2). We acknowledge this list is not exhaustive and may be limited based on our own mental models and relationships. In describing the items on the list we take great care to keep information as anonymous as possible. We then sorted these concerns into the categories of the systems change framework (Figure 4).
Step 2. Identifying Efforts Some graduate students in the UT Austin EEB Program have spent significant time and energy developing and implementing various reforms to address many of the concerns described in step 1. While by no means experts in inclusive policy and program development, we have addressed many of the tenets of systems change ( Figure 5). Here we do not elaborate further on the details and value of each individual effort, more information regarding these efforts can be found in supplemental materials. For the purposes of this exercise we focus on which categories of the systems change framework they might fall under in order to identify broader patterns of need and response.

Step 3. Evaluating Areas of Greatest Progress and Greatest Need
From our list of student concerns, we have identified consistent patterns: in the UT Austin EEB Graduate Program, student concerns primarily fall into the semi-explicit category of the systems change framework (Relationships & Connections / Power Dynamics). Graduate students generally are not expressing strong concern regarding explicit policies--we evaluate this to mean that the program is successfully implementing policies that support a diverse and inclusive climate. Collaboration, infrastructure, variety of expertise, grant and fellowship success, and publication quality were not frequently questioned. A "scientific policy" concern worth noting is the infrequency of course offerings: in response to this concern, the students have conducted a survey to collect data on what specific classes should be offered more regularly and what classes students have found most valuable.
The majority of the explicit concerns fall under "Resource Flows" and refer to the intangible resources that result from a mentor/mentee relationship, rather than more traditional resources such as stipend. Within the semi-explicit and implicit systems, nearly all of the concerns regarding relationships and power dynamics are "vertical" rather than "horizontal." We evaluate this positively as promotion of a positive peer-to-peer graduate student environment. But unfortunately, there is an extensive list of concerns that reflect need for improvement surrounding culture and infrastructure of mentorship. That concerns focus primarily on dynamics within a lab or between a supervisor and mentee is unsurprising, given that constructive or abusive lab environments and supervisors can completely define a graduate student's experience. Our efforts to address these mentor/mentee concerns focus more broadly at the departmental level and are aimed at improving the student's sense of support from the program, including a first year mentorship plan and personnel management training. Further effort towards building accountability for quality of mentorship, such as including mentorship evaluation in promotion and tenure applications, would target student concerns more directly.
Graduate students have little control over resource flows and power dynamics. Project distribution, hours, and authorship agreements are often informal, and there is little standardization of appropriate procedures and boundaries. As a result, students are subject to the supervisor's decisions and changes in those decisions. In severe cases, students often have little recourse other than to switch labs, which can be an exhausting and often unrealistic endeavor. The efforts we describe here that fall under Resource Flows include a Bill of Rights which codifies some basic rights for students as well as collecting regular data on resource distribution (e.g. demographics of award recipients). Formalizing a mechanism for enforcing student rights, clarifying the arbitration process, and adjusting resource distribution to meet program equity goals are all sensible and necessary next steps.
The implicit concerns voiced by students distinctly target women and minorities. The incidents described in the concerns are frequently encountered firsthand or witnessed by students. Graduate programming to prevent or respond decisively to these interactions (jokes, comments, minimization) should be of the utmost importance, in this case. Our efforts to address implicit systems include a monthly discussion group for issues of equity and inclusion as well as trainings for students and faculty in bystander intervention, bias, and personnel management.
However, implicit concerns, such as those regarding mental biases, discrimination, and narratives, are highly intangible and vary immensely from individual to individual. Given this intangibility, and inherent connection to power dynamics, we propose that policies and practices are the primary routes in which students can try to impact the mental models of those with more seniority and power. The authors are certainly not alone in having personally experienced the challenges that arise when engaging in discussions of mental bias and discrimination. Paradoxically, the trainees who are most emphatically expressing these concerns are those most impaired by these mental models. Thus, indirectly influencing mental models through policy change (such as creating a student Bill of Rights or organizing allyship trainings) may be the most attainable way for students to address these implicit concerns.
The evaluation of concerns and efforts we have engaged in was made possible by the trainee-led department Climate Survey. In the survey, researchers were asked to reflect on their experiences and evaluate how included they felt in various aspects related to their identity. The survey was critical in changing mental models regarding the pervasiveness and seriousness of graduate student concerns. This ability to "change the narrative" is a crucial step in long-lasting systemic change; it is a core feature in the systems change framework. Policy change without progress in mental models is not an effective tool for creating a more inclusive climate.

Conclusions
The academic system is touted as a meritocracy, but in reality, it still embraces norms and policies that are inequitable and paternalistic. Pushing back on these inequities by supporting graduate student trainees is vital to generating a more just scientific enterprise. Improving the experiences of graduate students, particularly marginalized students, is vital to our collective publication fecundity, effective teaching, competitive recruitment, prolific grant applications, and high quality research.
In many cases there is an initial energy by graduate programs to improve inclusion, often led by students as they are perceptive to these systemic challenges. But addressing complex issues that ultimately stem from deep societal iniquities and power structures cannot be solved in a single hour-long bias training event. Efforts must be multimodal and consistent to be effective.
Graduate students lack power to generate systemic change not only because of limited financial resources and institutional power, but also because of the loss in institutional knowledge that occurs as students graduate and no mechanism exists for systematically passing knowledge to new students. Thus, concerns and efforts initiated by graduate students must be respected and maintained by more permanent members of a program, in particular faculty members. Additionally, department wide conversations, events, and trainings should be held regularly to allow ample opportunity for communications and interactions between levels of the hierarchy. As we describe here, a critical first step in this process is to evaluate your own programs' diversity and climate efforts and how well they reflect graduate student concerns is a critical step in implementing effective change.
In assessing our efforts, we acknowledge our limitations and hope that our descriptions spur more comprehensive exercises of this nature. For example, future work could involve administering a large anonymous survey or discussion to generate the list of concerns, rather than the well-informed but informal process we have engaged in here. Future exercises should also include regular re-evaluation across trainee cohorts and after policy implementations, to gauge how the program needs and improvements grow and change over time. Comparisons between similar programs could stimulate compelling and useful crossinstitutional discussion.
We hope that the pedagogical exercise constructed here can help programs critically assess their efforts to improve climate and culture, and we hope this in turn positively impacts diversity and inclusivity in both direct and indirect ways. Programs' efforts to thoughtfully and actively enrich trainees' experiences in graduate school promotes a healthier and happier scientific community.