Bringing a Lens of Equity to Geoscience Qualifying Examinations

Qualifying examinations are an important milestone in geoscience graduate programs, but students with marginalized identities are disproportionately lost from graduate programs around the time of these exams. Inequity in qualifying exams can enter at multiple stages throughout the exam design, student mentorship experience, exam administration, and post exam feedback. Therefore, robust assessment is necessary when building an equitable examination. We provide concrete suggestions for graduate programs to evaluate and modify their qualifying examinations. The data‐driven and iterative process encourages graduate programs to outline specific expectations for success, employ best‐practice pedagogy, proactively support students, and use data to measure progress and inform changes in the examination.


Introduction
A majority of geoscience graduate programs have qualifying examinations that seek to assess a student's progress and ability to successfully complete their degree (Harshman, 2021;Leshner & Scherer, 2018).These examinations can be a valuable and important milestone for graduate students (Baker & Pifer, 2011;McAdams III et al., 2013).However, they can additionally be a point at which students, particularly those with identities marginalized in science, are lost from or leave degree-granting programs (Till, 2023;Wilson et al., 2018), contributing to the "hostile obstacle course" of academia (Berhe et al., 2022;Liera et al., 2023) and impacting student health and well-being (DiPietro et al., 2010;Furstenberg & Nichols-Casebolt, 2001;Lopez, 2022).We use "qualifying exam" as a general term to refer to any milestone exam in a graduate program which must be passed to continue toward the degree completion, although these examinations go by different names.
Statistical biases in the pass rates of these qualifying exams highlight their potential discriminatory nature.In several graduate programs in the United States, including geoscience programs, students who identify as women are disproportionately more likely to fail their qualifying examinations than male-identifying students (Kurfess et al., 2020;Singer, 2024;Till, 2023).Other programs have noted differences between the experiences of domestic and international students (Hestenes et al., 2022;Santiago & Nguyen, 2020).Unfortunately, for many programs, limited to no data exists on differences in outcomes by gender identity, race, citizenship status, LGBTQ + identity, first-generation student status, or many other identity axes.This is a result of either a lack of data collection (McLaughlin et al., 2023) and/or statistically small sample sizes of underrepresented groups.As the geosciences are one of the least racially diverse scientific fields (Bernard & Cooperdock, 2018;Dutt, 2020), and women, non-binary people, and disabled people remain underrepresented in certain geoscience subfields (Gonzalez, 2009;Ranganathan et al., 2021), it is vital that geoscience graduate programs assess their qualifying examinations as a potential point of failure of the academe for marginalized groups.To best serve the geoscience community, qualifying examinations must be undertaken and assessed through a lens of equity (McLaughlin et al., 2023).It is vital to consider the messaging and resources provided to students before, during, and after the examination as sources of inequity.While the administration of the exam itself is a place where implicit bias may enter, other activities surrounding the exam can also be sources of inequity-from its design, support and communication leading up to the exam, and feedback after the exam.We present a collection of concrete recommendations aimed at making qualifying examinations more equitable and propose a framework that is transferable across geoscience graduate programs (Figure 1).While suggestions are outlined here, further information around many steps are outlined in Supporting Information S1.

Step 1: Collect and Analyze Data. Gather Data on Examination Outcomes and Experiences and Assess the Need for Changes
As in natural science, the first step in assessing an examination's success is gathering the relevant data (Ali et al., 2021;Avery et al., 2022;Ormand et al., 2022).Graduate programs should collect quantitative data on examination outcomes alongside information on student identities and the intersections thereof (Ali et al., 2021;Núñez et al., 2020).Where possible, historical data should be gathered and synthesized.Data should be made public within the graduate department, following any institutional regulations, anonymizing and redacting data that identifies individuals, and allowing for self-description of identity (Myanna, 2023).In order to maintain Steps to foster more equitable qualifying examinations in geoscience.In this data-driven, iterative process, a graduate program uses data to inform changes in the examination; provides specific and explicit expectations for success; employs best-practice pedagogical standards; proactively provides student support; and uses data to measure progress.anonymity in the case of small numbers of students from specific groups (e.g., racial identity (Bernard & Cooperdock, 2018) or fields of study), data can be aggregated over several years.
In addition to the quantitative data on examination outcomes with respect to identity, graduate programs should also collect qualitative data on students' experiences with the examination (Memarian et al., 2019), through processes like anonymous surveys (see Supporting Information S1 for example questions).Department or campus climate surveys provide the opportunity to collect this data in the context of other aspects of the program.This qualitative feedback from students is vital to assess whether students have productive learning experiences before, during, and after the qualifying examination.While the quantitative data on examination outcomes is necessary, the student experience data provides a more holistic view to ensure that programs are meeting their equity goals and that the exam is functioning as a learning experience for students.
Programs should take time to consider metrics for what constitutes a successful exam when evaluating the data on outcomes and experiences.Examples of such metrics may be a certain pass rate, equity in pass rates between identity groups, achieving certain thresholds of high-quality mentorship as reported by students, or thresholds for actionable feedback.Defining these metrics at the outset will help the program both design and analyze their surveys.Examples of data that can spark concern for inequity in a qualifying exam include statistically significant differences in pass rates for students of different identity groups, for example, men, women, and non-binary individuals, or native English speakers and non-native English speakers.Furthermore, students expressing inadequate preparation or mentorship leading up to the exam or sparse feedback after the exam are causes for concern and highlight potential intervention points.

Step 2: Define Examination Goals. Explicitly Define the Goals of the Examination and Metrics of Success
To achieve evidence-based best practices, graduate programs must consolidate a definition of success that examiners use to decide examination outcomes (see Supporting Information S1 for examples) (Furstenberg & Nichols-Casebolt, 2001;Posselt & Liera, 2022).These criteria for success should rise above individual perceptions (Kostohryz, 2016) and the concept of the qualifying examination as a historically necessary "hurdle" (Posselt et al., 2021) or "rite of passage" (Estrem & Lucas, 2003).Although subjectivity is inherent in their design (Liera et al., 2023), being specific in what outcomes are and are not acceptable for a successful examination can help reduce this subjectivity.
Without a clear and unified statement about the goals of the examination from the degree program, faculty and students can derive different views of the examination's purposes (Guloy et al., 2020) and examinations may carry examiners' implicit biases or unwritten expectations (MacLachlan, 2017).In addition, vagueness in expectations significantly increases student stress and anxiety, which can impact examination performance (Andrade & Du, 2019;DiPietro et al., 2010;Harding-DeKam et al., 2012).By explicitly defining what success looks like and how students can achieve that goal, graduate programs ensure that students can focus on putting their time into building skills that will be assessed, rather than focusing on things that are not being examined.For instance, if a student's oral presentation skills are (or are not) part of the assessment, this can be explicitly stated so students know whether they should put time into practicing their presentations.In a field as broad as the geosciences, which covers topics across the biosphere, hydrosphere, cryosphere, atmosphere, and geosphere, programs must be explicit about what topics students must be versed in for a successful outcome.

Step 3: Assess Examination Structure. Determine the Best Format to Assess the Examination Goals
There are a range of structures of qualifying examinations across graduate programs, including written examinations, oral examinations, or a combination of formats.There are benefits and drawbacks to each of these structures that are typically dependent on the goals of the graduate program and the required coursework.In addition, qualifying examinations can also occur at a range of times in a graduate program.Programs must assess if the structure of the qualifying examination is a remnant of tradition or if it truly fits the goals of the examination outlined in Step 2, particularly as informed by student feedback detailed in Step 1 (Allard et al., 2021).A range of potential structures are described in the Supporting Information S1.

Step 4: Determine Examination Outcomes. Choose the Range of Examination Outcomes and Define How They Align With the Examination Goals
Programs should additionally consider the potential outcomes of a qualifying examination.While a stark pass/fail option may be the most straightforward, offering a "conditional pass" can help the exam serve its purpose as a learning opportunity.The use of a conditional pass result requires examiners to identify the cause of the concern and to specify the conditions to be met for the student to pass (Wilkinson et al., 2011).Under a conditional pass, students may be required to take, retake, or be a teaching assistant for a certain class, or do individual study under faculty guidance.

Step 5: Make a Rubric. Design a Rubric That Reflects the Examination Goals, Structure, and Outcomes
Rubrics are a fundamental tool for providing information to students on areas of assessment and for programs to assess preparation of students along different axes (Cockett & Jackson, 2018;Wolf & Stevens, 2007) (see Supporting Information S1 for examples).The effective use of rubrics provides consistent assessment through reproducible scoring, reducing subjectivity inherent to qualifying examinations (Science Education Resource Center, 2024).It is important to note that rubrics do not inherently imply inflexibility in exam content; rubrics can be adaptable to individual examinations, with examiners filling in topics on the rubric specific to an individual students' examination.In the case of a rubric which is adapted to individual examinations, the rubric may initially be a somewhat sparse document, but should still outline the assessed skills.Well in advance of the examination, examiners should explain the rubric to the students, as students who understand a rubric are more likely to have greater assessment performance outcomes (Francis, 2018).The rubric must be detailed enough to usefully inform the examination outcome (Brookhart & Chen, 2015).

Step 6: Communicate With Students. Provide Students With Clear, Structured Information
Steps 2-5 are incomplete without clearly communicating the information to students.The goals, rubric, and possible outcomes of the exam should be carefully explained, both in writing and verbally, with ample opportunity for students to ask questions provided.Graduate programs may hesitate to provide transparency on the examination in an effort to increase flexibility; however, transparency and flexibility are not mutually exclusive.Even, or especially, in situations where the exam is highly individualized, the program should communicate this flexibility clearly to the students.It is important that students receive official information from the program rather than potentially haphazard information from other sources.Students may also have financial, immigration, or personal concerns related to examination outcome that can be addressed with transparent communication from the graduate program.The production and explanation of a definition of examination success and a rubric (Step 5) is a concrete step to providing this information to students.

Step 7: Prioritize Student Mentorship. Give Students Varied Avenues for Direct Mentorship
Clear communication between students and faculty is a requisite for student success at all stages.Therefore, it is necessary to ensure students receive quality faculty mentorship throughout the time leading up to the examination, as the success and satisfaction of graduate students is closely tied to their mentorship experience (Paglis et al., 2006;Tuma et al., 2021).This practice is especially important for historically excluded groups, as these students are more likely to struggle to find supportive mentors (National Academies of Sciences, Engineering, and Medicine; Policy and Global Affairs; Board on Higher Education and Workforce; Committee on Effective Mentoring in STEMM, 2019a).In particular, peer and cross-institutional mentoring programs have been demonstrated to enhance the success of underrepresented groups in the biomedical sciences (Wilson et al., 2018) and the geosciences (Huntoon et al., 2015).
Advisors and faculty mentors should frequently engage in open communication with their mentees about their needs, including developing a timeline for progress leading up the examination and a discussing the required support to reach the outlined goals, in line with mentorship best practices (National Academies of Sciences, Engineering, and Medicine; Policy and Global Affairs; Board on Higher Education and Workforce; Committee on Effective Mentoring in STEMM, 2019b).Examples of mentorship best practices in the Supporting Information S1, but frequent student surveys are an excellent way to evaluate whether students are receiving quality mentorship.

Step 8: Support Skill Building. Offer Explicit Settings for Students to Practice the Skills That Will Be Assessed
Students must be provided both structured and unstructured environments to practice skills assessed during the exam (examples in the Supporting Information S1).Such intervention was shown to be successful in combating attrition of historically marginalized students in the biomedical sciences (Wilson et al., 2018) and increasing students' self-efficacy in the geosciences (Baber et al., 2010).Formative opportunities for students to demonstrate knowledge under minimal test anxiety was also shown to increase performance on high-stakes exams, particularly for women (Ballen et al., 2017;Cotner & Ballen, 2017;Salehi et al., 2019).
Vitally, students must be provided feedback on their practice, preferably using the rubric developed for the qualifying examination (Johnson et al., 2016;Morris et al., 2021).This provides students the ability to mentally align their actions with their outcomes in a lower-stakes environment (Rose et al., 2006) and also gives examiners practice applying the rubric.Students should have several points of contact from whom they can receive feedback, and care must be taken to ensure that historically marginalized students are not excluded from informal mentoring circles (Liera et al., 2023).

Step 9: Ensure Accessibility. Connect Students With Disability Services and Provide Accommodations
Upwards of 13% of the United States population is disabled, but disabled students are regularly underrepresented in and not served fully by graduate programs (Brown & Leigh, 2018;Dali, 2018).There is an additional discrepancy in geoscience graduate programs, as geoscience has a reputation for being unfriendly for disabled students due to its historical reliance on often inaccessible field data (Atchison & Libarkin, 2016;Lawrence, 2021;Mol & Atchison, 2019).
Qualifying examinations can present an additional barrier for disabled students.For instance, examinations that are several hours long may not be accessible for students with learning or physical disabilities, the examination room may not be appropriate for students with sensory processing disorders, or accessibility technology may need to be adapted for an examination's format.Graduate programs should provide the university's accessibility and disability services program information on the examination format and proactively provide all students with information about how to connect with and attain accommodations through these offices well in advance of the examination date.Graduate programs must follow through to provide necessary accommodations for students.

Step 10: Designate Examination Leadership. Assign a Student-Facing Examination Leader
During the exam it is important to have a designated examination leader who can advocate for the student and answer any questions that arise (more details in Supporting Information S1).This person should be available during the exam, but also ahead of the exam, to answer questions.This role is applicable across multiple exam formats.In particular, this student-facing examiner should monitor examiner behavior and enact bystander intervention steps in the case of questioning that is out of scope or hostile (Haynes-Baratz et al., 2021;Shea et al., 2019) This role is vital, as antagonistic and demeaning environments and interactions are one major reason that students may leave the geosciences (Cabay et al., 2018;Marín-Spiotta et al., 2020).

Step 11: Encourage Critical Thinking. Offer Students Opportunities to Demonstrate Their Knowledge in Multiple Ways
Often, one of the primary goals of qualifying examinations is to assess students' abilities to make connections between topics and engage with questions that require complex thinking.Offering opportunities for students to participate in diverse ways to effectively express their knowledge is a key component of inclusive instructional design (Capp, 2017;Carabajal et al., 2017).For example, the exam can provide both written and oral formats for students to share the same knowledge.Students can be provided with paper or a whiteboard during an oral exam to write down thoughts or explanations.Or students can be prompted to explain their answers in diverse ways using metaphors, pictures, equations, data, or more.Examiners should encourage students to use these alternative methods and provide alternative ways to engage with the questions (Abegglen et al., 2021;Rose et al., 2006) through prompting questions or availability of tools.

Step 12: Provide Feedback. Collect and Return Prompt and Actionable Feedback to Students
A key part of treating qualifying examinations as a learning experience is providing students feedback on their performance so they can reflect on their successes and be made aware of places for improvement (Haughney et al., 2020;Jensen et al., 2012;Poulos & Mahony, 2008).Feedback can focus on both topics where students performed well and topics where students can improve.In all cases, feedback must focus on the student's performance on the examination, not on a student's personal characteristics or the examiner's perception of the student's belonging in academia.Feedback should be actionable, meaning that students should walk away with a specific understanding of where and how they can improve.
In the case where students are asked to retake an examination, prompt and actionable feedback presents students the ability to focus their attention on places where they can improve (Evans, 2013).In the case where students passed an examination, prompt and actionable feedback provides a confidence boost and also gives students goals to focus on for the future.Strategies for providing cohesive feedback are detailed in the Supporting Information S1.

Conclusion
Graduate programs should conduct a regular, data-informed review of their qualifying exam, using a lens of equity to ensure that student outcomes and experiences are equitable across identity groups.The framework we have outlined above must be approached iteratively, using the quantitative and qualitative data collected to inform and assess changes made to the examination.Data analysis and examination updates do not need to necessarily be repeated every year, but should align with the frequency and nature of changes made to the exam.Patterns in outcomes and experiences may take time to emerge, particularly in small programs.However, regularly scheduled evaluations of outcomes are vital to assess progress, achievement, and maintenance of equitable examination outcomes (Ruggs & Avery, 2021) and student support.
Qualifying examinations are an understudied yet potentially important place where students underrepresented in the field are not retained.Geoscience graduate programs can address this challenge by being mindful about their examination practices and approaching their creation, execution, and assessment through a lens of equity.By addressing this issue with evidence-based practices, geoscience graduate programs can align their values with their actions, enhance the graduate student experience, and pursue equity in the scientific endeavor.

•
Qualifying exams in the geosciences are an understudied yet potentially important failure point for retention of underrepresented students • The design and implementation of the qualifying exam should be done in a data-driven and iterative way to ensure equity in outcomes • Graduate programs should be specific in their definition and communication of a successful exam outcome with students Supporting Information: Supporting Information may be found in the online version of this article.

Figure 1 .
Figure 1.Steps to foster more equitable qualifying examinations in geoscience.In this data-driven, iterative process, a graduate program uses data to inform changes in the examination; provides specific and explicit expectations for success; employs best-practice pedagogical standards; proactively provides student support; and uses data to measure progress.