Culturally responsive evaluation teaching and learning in higher education: A higher calling

Culturally responsive evaluation (CRE) education has gained traction over the past 15 years, yet there is a dearth of literature on this topic. In response, we offer guidance on CRE education in higher education. Influenced by Stafford Hood, we engage CRE education as a social responsibility or a higher calling. Accordingly, in this article, we describe the importance of culturally responsive pedagogy (CRP) to fulfill our social responsibility in CRE education. We also highlight signature CRP used in our CRE courses, namely the incorporation of diverse practical field experiences and innovative instructional technology. We conclude with lessons learned.


INTRODUCTION
Make sure you are always well technically trained and grounded heavily in your social responsibility.Also, make sure you are reaching back and helping to bring someone else along.Stafford Hood, Ph.D., 2017 While culturally responsive evaluation (CRE) education has been recognized in the literature (Boyce & Chouinard, 2017;Reid et al., 2023;Thomas & Madison, 2010), there is limited scholarship on how to teach CRE using culturally relevant pedagogies (CRP) (Ladson-Billings, 1995) in higher education.Accordingly, in this article, we offer guidance on CRP for teaching CRE.Our work is grounded in the legacy and higher calling set forth by Stafford Hood as noted in the quote above, the CRE and CRP literature, our experiences teaching CRE, and our commitments to social justice and equity (Avent et al., 2023;Boyce, 2019;Boyce et al., 2022).
To begin, we first describe our positionalities and review the CRP and CRE literature.Then, we describe two culturally relevant pedagogical approaches to teaching CRE and lessons learned from integrating these approaches into our CRE courses.Our work underscores how CRE educators and students are called to carry out their social responsibility, their higher calling.

OUR POSITIONALITIES
We identify as Black evaluators and educators who are influenced by Stafford Hood.We have all taught formal semester-long CRE courses.We are also committed to centering culture and social justice as part of the evaluation process because it is the socially responsible thing to do (Hood, 2004).
Although we share these characteristics and commitments, we have different academic positions, institutional locations, and approaches to CRE training.Jori N. Hall is a multidisciplinary methodologist who has taught social science methods for over 15 years.She is a tenured professor in Chicago, Illinois.Cherie M. Avent is a Black woman, wife, and mother who serves as a faculty member in Champaign, Illinois as an assistant professor.Ayesha S. Boyce is a cisgender Black Latina woman, wife, and mother.She is a tenured associate professor in Tempe, Arizona.Kwame O. Acheampong is a cisgender Black African man.He is a doctoral student at the University of Georgia.
While our CRE pedagogical choices are shaped by our different geographical locations, institutions, and professional stages, our choices are informed by the same CRP literature.

Culturally relevant pedagogy
The roots of CRE began in US educational scholarship from Carol Lee (1998) and Gloria Ladson-Billings (1995) on CRP (Hood 2015).Lee and Ladson-Billings saw gaps in the literature on African-American students' educational success and put forth a theory that affirms students' cultural identity and lived experiences.CRP is described as "a pedagogy that empowers students intellectually, socially, emotionally, and politically by using cultural referents to impart knowledge, skills, and attitudes" (Ladson-Billings, 2009, p. 20).Cultural referents in the context of CRP are elements of a student's culture that are used as a foundation for teaching and learning.Cultural referents can include things such as cultural symbols, customs, literature; students' native language or vernacular; music, art, and other culturally relevant creative expressions; as well as media and technology that showcase students' cultural perspectives.CRP also pushes against the status quo while attending to cultural, historical, and political factors for equitable outcomes in student learning.According to Lee et al. (1990), a cultural lens can "contribute to achieving pride, equity, power, wealth, and cultural continuity," particularly for African Americans (p.47).CRP proposes to develop students who can achieve academically, exhibit cultural competence, and understand and critique the current social order (Ladson-Billings, 1995).

Culturally responsive evaluation education
Despite the increase in evaluation pedagogical literature (LaVelle, 2018), only a modest amount emphasizes CRE pedagogy from a social justice-orientation or CRP lens in higher education.A notable contribution is a New Directions for Evaluation issue that takes a critical lens on evaluation training in academic settings (Lavelle et al., 2023).Authors in the issue discuss culturally and linguistically appropriate evaluator education (Robles-Shrader & Lemos, 2023), course design principles for centering social justice (Robinson et al., 2023), and value-engaged training models (Reid et al., 2023).Other contributions to CRE pedagogy include Boyce's (2021) article on strategies for mentoring and advising graduate students of color; a conceptual framework for teaching graduate-level CRE courses (Boyce & Chouinard, 2017); an article on why and how to integrate a social justice orientation in graduate evaluation training (Thomas & Madison, 2010); a textbook for graduate students and others that covers social justice, diversity, and quality in evaluation practice (Thomas & Campbell, 2021); and an evaluation matrix that explains several evaluation philosophies and approaches that center cultural responsiveness and equity (Bledsoe et al., 2022).
Together, the US CRE education literature argues that culturally responsive and socialjustice orientations are needed in graduate education to produce critical evaluators.Our work responds to this argument by presenting CRP as a pedagogical approach that responds to these needs.While CRP is implemented in K-12 settings (Aronson & Laughter, 2015), we view it as appropriate for CRE graduate education as CRP aligns with CRE.Specifically, CRP affirms students' culture, encourages responsiveness, and demands engagement with issues of power and inequity.Further, applying CRP to higher education contexts is essential as "university-based evaluation programs" are a "mainstay for educating and socializing would-be evaluators" (LaVelle, 2018, p. 2).Therefore, we view employing CRP in our CRE courses as our social responsibility as advanced by Stafford Hood.Over the years, we have developed signature culturally relevant pedagogies in our graduatelevel CRE courses to enhance our students' methodological skills and responsiveness in evaluation practice.We highlight two of these practices below.

CASE EXAMPLES
This section highlights case examples drawn from a CRE course taught by Avent and another co-taught by Hall and Acheampong.Both graduate-level CRE courses employed CRP but in unique ways.The first used diverse field experiences, and the second used innovative instructional technology.Below, we provide the course context, signature pedagogy, and snapshots to illustrate pedagogical activities employed in each course.

Course context
In spring 2021, Avent taught a 16-week graduate evaluation course in-person at a researchintensive, predominately white institution in Illinois, USA.The course aims to provide students with diverse field experiences.Avent's course goals include working with students in teams to discuss and critique readings on CRE; construct an evaluation plan for local programs; offer feedback on how to negotiate evaluation agreements with clients; understand the political dynamics of the evaluation context; meet with clients; and design an evaluation plan.

Signature pedagogy
Avent identified three local programs that worked with diverse communities for students to engage with.She selected the programs to: (a) utilize the course and her teaching as a way to give back to the community; (b) provide field experiences that allow students to apply culturally and contextually appropriate practices (Boyce & Chouinard, 2017); and (c) encourage collaborative learning (Ladson-Billings, 1995).One program affirmed the racial identity of Black youth and promoted social justice while fostering cultural awareness and enhancing academic, civic, and socio-emotional capabilities through a literacy curriculum for K-5th grade students.The second program offered wraparound services for families of Black boys seeking to disrupt the school-to-prison pipeline.The third program created a science, technology, engineering and mathematics (STEM) enrichment afterschool program for Black and Brown youth in 3rd-5th grades who have experienced inequities concerning academic and technological innovation.

Snapshot
Below is a "snapshot" of how Avent used CRP to enhance the capacity of novice evaluators and attend to her institutional context.
Early in the semester, students were told they would apply a CRE lens focused on "demographic, sociopolitical, and contextual dimensions, locations, perspectives, and characteristics of culture" (Hood et al., 2015, p. 283) within the context of program evaluation.To support students' application of a CRE lens, Avent used course readings and in-class discussions to build a foundation for students to situate a program's historical context and have conversations with program members about how research and evaluation have positioned people whom the program is intended to benefit.For example, students working on the evaluation plan for the wraparound services for families of Black boys had early conversations with program leaders to learn about the program and discuss how research often positions Black boys from a deficit perspective.Students connected these conversations with the community's history and examined how the university-a predominantly white institution with high research productivity-possibly perpetuates deficit framing locally and nationally.
Together, the students and Avent reflected on conversations with program members as part of recognizing, understanding, and critiquing how sociopolitical and contextual dimensions influence a program, participants, and the nature of doing evaluation (McBride et al., 2020), as well as the students' responsibilities as evaluators to adopt an antideficit approach.Furthermore, students had to consider how they might address issues of inequity for historically marginalized communities (Hood et al., 2015) in their evaluation plan following conversations and interactions with their local community program.Overall, the diverse field experiences made it possible to engage with CRE principles through community building, collaborative learning, reflection, and collective responsibility.

Course context
In spring 2023, Hall and Acheampong co-taught a 15-week CRE hybrid course, including alternate online and in-person sessions.All of the students identify as Black and the course was conducted in a predominantly white institution in Georgia, USA.Course learning goals included identifying principles of CRE; reflecting on assumptions one brings to evaluation; investigating appropriate theories and practices when applying CRE; and considering challenges in CRE.

Signature pedagogy
As culturally responsive instructors, building a sense of community and establishing meaningful learning experiences are two main pedagogical aims.To accomplish these aims, the Community of Inquiry framework guided the course design (Garrison et al., 2000).This framework emphasizes collaboration and reflection in higher education courses through: (a) exchanging ideas through reflective discourse (cognitive presence); (b) developing relationships with the learning community (social presence); and (c) designing and facilitating course content to support learning goals (teaching presence).While other instructional technologies (Zoom, discussion boards) were used to support the course design, the Metaverse-a 3-D virtual reality (VR) context-is highlighted as it is an innovative instructional technology to support CRE education and extends previous work on the incorporation of VR technology in graduate-level methods courses (Hall & Campbell, 2023).

Snapshot
The Metaverse brought the instructors and students together in a CRE-VR classroom as avatars or electronic images.The following snapshot illustrates how the CRE-VR classroom was used to facilitate CRP focused on establishing a sense of community and supporting meaningful learning experiences.
Early in the semester, a brief orientation session was held during an inperson session, enabling students to access the Metaverse and navigate the CRE-VR classroom.During the semester, students practiced accessing the CRE-VR classroom, using chat/audio functions, and interacting with the e-board.Instructors' and students' relationships with each other deepened while working to navigate the VR environment.Collaborating on VR-related tasks also laid the foundation for open discussions about ethical issues and challenges in the context of CRE like the role of privilege and intersecting forms of oppression (i.e., racism, sexism) (Hood et al., 2015).As part of an hour-long session in the CRE-VR classroom, the instructors and students explored the challenges of CRE, namely global demands for new CRE approaches, via a PowerPoint presentation.To conclude, students connected CRE and its associated challenges to their field guided by questions like, What are the future implications of CRE for your discipline?Although Hall and Acheampong do not suggest that using VR is necessary for CRE education in higher education contexts, they offer some instructive observations.First, the shared racial identity of instructors and students helped to establish rapport and a sense of community in the CRE course.Second, collaboratively engaging the CRE content while simultaneously navigating the VR technology as a class served to establish a trusting environment and deepen relationships.Third, navigating the VR technology facilitated discussions about the value of innovative technology for CRE practice.Overall, the VR-specific collaboration was viewed as a unique experience in that the instructors and students were all positioned as "learners," taking risks and exploring the technology together, which facilitated a sense of community and deeper engagement with the CRE content.

CONCLUSION
Based on our experiences, we have learned a few valuable lessons.First, the context of the CRE course matters.Both CRE courses took place in predominantly white institutions, which had implications for CRE teaching and learning.For example, Avent engaged in conversations about how predominately white institutions and evaluations can examine marginalized groups from a deficit perspective, which, in turn, informed how students engaged the community program members and the development of their evaluation plans.Second, CRP cannot guarantee critical scholars.Admittedly, there is a nuanced relationship between CRP and the context within which it applied; therefore, its contributions to critical teaching and learning will vary.Yet, instructors working collaboratively with students can enhance students' collaboration and communication skills with each other, which are vital to CRE practice (Boyce & Chouinard, 2017).This was apparent in the course taught by Hall and Acheampong.Their collaborative VR tasks (both in person and virtually) with students created a trusting classroom environment that facilitated dialogue on challenging topics such as oppression and privilege.Third, engagement in CRE education is strengthened when CRE principles are relevant to students' lives (Hood et al., 2015).For example, both courses had students apply CRE principles to their interests, disciplines, and projects.
These lessons suggest that the future of CRE education relies on "relational and engaged evaluator education" (Neubauer et al., 2023, p. 158).As illustrated by the example cases, it is important for CRE pedagogy in the future to include examinations of various contexts (i.e., students', institutional, participant-community); opportunities for students and instructors to work and learn collaboratively; and applications of the course content in meaningful ways.As evidenced by our CRP practices and in alignment with other evaluation educators, we believe that future evaluator training would "benefit from continuing to understand the value-added of social learning and communities of practice" (Neubauer et al., 2023, p. 158).Our experiences also suggest the value of CRE courses to explore and leverage emerging technologies for CRE purposes.To conclude, we hope our CRP work in CRE education helps others to consider how to challenge the status quo in innovative and socially responsible ways, which we believe is our higher calling.