African Americans, Hispanic Americans, and American Indians are less likely than European Americans and Asians to pursue science and engineering careers. African Americans, Hispanic Americans, and Native Americans comprise 13.1%, 16.6%, and 0.8%, respectively, of the United States working-age population. However, these groups make up only 5.8%, 5.2%, and 0.4%, respectively, of the scientists and engineers in the United States (National Science Foundation [NSF], 2007). Evidence suggests that attrition of urban students from these underrepresented groups away from science and engineering careers begins in the middle grades (Atwater, 1990; King, Shumow, & Lietz, 2001; NSF, 2007) as inequitable access to the kinds of instructional opportunities necessary for success in science may cause science achievement and attitudes toward science decline (Hill, Atwater, & Wiggins, 1995; Kahle, Meece, & Scantlebury, 2000). That students' reduced attitudes toward science during middle school may eventually lead them away from science careers makes sense as reduced attitudes may lead a student to choose not to take the science courses that can lead to science-related careers. Or reduced science achievement levels may result in students not being given the opportunity to take the science courses that can lead to science careers. Therefore, the extent to which students' science attitudes and achievement decline during middle school can ultimately impact the representation of certain racial and ethnic groups in science careers in the workplace.
We recognize that there are cross-cultural barriers put up by the type of science instruction typical in the urban schools that many minority students attend, wherein the memorization of static facts and theories is emphasized, a “pedagogy of poverty” that works to maintain cross-cultural barriers and disproportionately hurt students' science achievement and attitudes (Barton, Ermer, Burkett, & Osborne, 2003). Our premise is that an alternative mode of instruction, project-based science or PBS (Singer, Marx, Krajcik, & Chambers, 2000), can better leverage what students bring to their science instruction rather than focusing on what they lack. We ask if PBS as an alternate mode of instruction may better promote border crossing into the culture of school science for students marginalized from traditional science schooling (Aikenhead, 1996). If PBS can do this, we expect students' science achievement and attitudes (and ultimately representation in science careers) will improve as a consequence. This is our first research question: Can underrepresented students' science achievement, science attitudes, and science career plans improve with a PBS curriculum? The specific ways that PBS is designed to leverage what students bring to their science instruction are discussed in the next section. As this work is a preliminary study of this potential of a PBS curriculum, we combine urban students from several different ethnic and racial groups who are underrepresented in science careers in our data set. We have every reason to expect individual students from these groups to vary in their initial levels of science achievement or attitudes, or for a PBS curriculum to impact every student from these groups to a varying degree. However, we do believe that a PBS curriculum might better leverage the strengths of students marginalized from traditional science schooling and thus in this study we will look at the capacity of a PBS curriculum to effect changes in student outcomes by aggregating students across ethnic and racial groups underrepresented in science careers. Our second research question aims to isolate the role of teacher knowledge in our findings: Do teachers' levels of science content knowledge and science pedagogical content knowledge influence the extent of the impact of a PBS curriculum on underrepresented minority students? By learning more about the mechanisms by which a PBS curriculum impacts underrepresented minority students' science achievement, attitudes, and career plans, and the ways in which teacher knowledge is implicated, we can ultimately use the results of this work to start a program of design-based research: to generate and later test ideas about how to modify the design of the PBS curriculum as well as teacher professional development to better support underrepresented student outcomes.
ADVANTAGES OF PROJECT-BASED SCIENCE CURRICULA
Project-based science (PBS) is a reform-based pedagogy that emphasizes the students themselves constructing a usable or meaningful understanding (Ausubel, 1968) of the science they are learning, as opposed to memorizing decontextualized scientific facts. PBS aims to do this by emphasizing scientific inquiry in the classroom, but it also aims to motivate students' inquiry by their need to find a solution to a real problem, the resolution of which requires students to apply the ideas they have learned (Edelson, 2001; Singer et al. 2000). PBS is notable for having a question or problem that serves to organize and drive the activities, as opposed to more conventional curricula that may contain inquiry-based activities but lack a driving question. In addressing its driving question, a PBS unit may require several weeks of class time, during which time students will address science standards, often crossing traditional disciplines. PBS curricula have shown the capacity to improve students' meaningful understanding of science content (Kanter, 2009; Kanter & Schreck, 2006; Kolodner et al., 2003; Krajcik, McNeill, & Reiser, 2008; Linn, Bell, & Davis, 2004; Marx et al., 2004; Puntambekar & Kolodner, 2005; Rivet & Krajcik, 2004; Schneider, 2002; ).
At the same time, there is reason to hypothesize that a middle school PBS curriculum might be uniquely well suited, by virtue of its design, to positively impact science achievement and attitudes and ultimately science college and career plans of students from ethnic and racial groups underrepresented in science careers. The project-based aspect of a PBS curriculum emphasizes the real-world utility of science and its relevance to everyday life, where learning concepts is not separated from personally relevant contexts and experiences. In this way, the project-based aspects of a PBS curriculum should promote a degree of border crossing into the culture of school science (Aikenhead, 1996; Bouillion & Gomez, 2001), the lack of which reduces these students' attitudes toward science (Cobern, 1996). Using terminology from Barton et al. (2003), the project-based aspect of a PBS curriculum allows for the reconceptualizing of achievement in the science classroom for students to see that they are now rewarded for using the science they are learning to respond to concerns that are relevant to them. The inquiry-based aspect of a PBS curriculum emphasizes knowledge construction and supports students doing science rather than memorizing static facts and theories—a practice that is typical of the “pedagogy of poverty” minority students often receive in urban districts (Haberman, 1991; King et al., 2001). Such instruction reduces students' attitudes toward science as well as their achievement (Kahle et al., 2000). Opportunities to engage in inquiry should improve science achievement while also improving attitudes toward science. Again, to use terminology from Barton et al. (2003), the inquiry-based aspect of PBS allows for students to see that the resources they are to bring to bear on the construction of new science knowledge have been reconceptualized to include their individual knowledge, skills, and expertise, in addition to that of Western scientific practices. Together, the project- and inquiry-based aspects of a PBS curriculum redesign science instruction to leverage all the real-world interests and resources that students bring to the science classroom and to reduce barriers to their crossing from their real-life worlds into the world of the science classroom. Thus, as an alternative to the science instruction that students from groups underrepresented in science careers frequently receive in urban schools, a PBS curriculum might better leverage the strengths of these students otherwise marginalized from traditional schooling, reducing barriers to border crossing. Improved science achievement and attitudes may result and ultimately increased representation in science careers may follow. Our first research question was to explore the extent to which we would find this to be the case, the extent to which a middle school PBS curriculum could positively influence underrepresented minority middle school students' science attitudes and achievement and their plans to pursue science careers. The specific middle school PBS curriculum we used to conduct this study is described in detail in the Methods section.
Importance of Teacher Knowledge in PBS
PBS curricula are known to be hard to teach (Schneider, Krajcik, & Blumenfeld, 2005). To use them, teachers have to make clear the need to learn the science content to do the project and maintain that connection throughout. Furthermore, teachers have to be able to recognize from the students' comments and work the particular ways students misunderstand the science. And given what the teachers diagnose in that regard, teachers have to be able to use the PBS curriculum activities to support students constructing a more scientific understanding from their initial misunderstandings. Finally, teachers have to determine whether students have made progress toward a more accurate scientific conception (Bransford, Darling-Hammond, & LePage, 2007).
To be able to do all these things, teachers need both a knowledge of the specific science content (content knowledge or CK) and a knowledge of how to make this specific science content accessible to others (pedagogical content knowledge or PCK; Gess-Newsome, 1999; Hammerness et al., 2007; Shulman, 1987). If teachers are lacking in CK or PCK, they may not be able to employ the inquiry-based aspects of a PBS curriculum to notice and diagnose students' science misconceptions, or challenge and change these science ideas. Given the wide variety of ways the PCK construct is interpreted in the literature, the specific interpretation and related measure of PCK that we use in this study is described in detail in the Methods section under Teacher Instruments and Coding. We hypothesized that minority students' science achievement and attitudes would benefit by learning science with a PBS curriculum as discussed above, but our second hypothesis was that the levels of teachers' CK and PCK would impact the extent to which teachers could use a PBS curriculum to benefit minority students. As related to student achievement, this possibility is supported by educational production function studies that connect teacher CK to student achievement (Rowan, Chiang, & Miller, 1997), as well as the findings of the Hill, Rowan, and Ball study (2005) that showed that teachers' “content knowledge for teaching mathematics” (encompassing elements of mathematics CK as well as PCK) was a significant predictor of gains in student mathematics achievement.
Thus, in addition to simply providing teachers with a PBS curriculum, we recognized the need to add professional development, designed to increase teachers' CK and PCK for specific science subject matter before teaching the PBS lessons on that science subject matter. We would do this to raise teacher CK and PCK to a high level that would allow us to observe the maximum possible impact of a PBS curriculum on minority students' science achievement and attitudes. We assumed that, although teachers may not necessarily start with uniformly high levels of CK and PCK for the specific science content emphasized in the PBS curriculum, we could improve teacher knowledge through professional development.
Our approach to this professional development was “practice based,” (Loucks-Horsley, Hewson, Love, & Stiles, 1997) in that we attempted to address a fundamental challenge of professional development: the disconnect between the professional development context and teachers' own daily practice (Putnam & Borko, 2000). The types of constructivist pedagogical approaches for learners advocated in science reforms are critical for teacher learning as well. Often teachers are left alone to figure out how to map or apply what they learn in professional development to their own practice, away from the support of the professional development setting. Instead, practice-based approaches to professional development aim to use any of a variety of learning activities to support teachers in applying what they are learning to their own practice, to resolving problems they are experiencing in their own classrooms. In short, we are situating professional development in the real practices of teaching. In so doing, we make the everyday work of teaching the object of inquiry out of which we can develop teacher CK and PCK, so that teachers can connect this knowledge to their practice. In this way we aim to provide teachers with a transformative professional development learning experience that involves sweeping changes in teachers' knowledge. The intended outcome is to move teachers away from using the PBS curriculum to help students get the right answer to a more informed practice of using their CK and PCK to guide students in productive conversations that will clarify and change their ideas using the PBS curriculum.
We will emphasize practice-based professional development (Loucks-Horsley et al., 1997) examining student thinking as teachers implement a new curriculum, in this case a PBS curriculum, as the practice in which we situate teacher professional development. Examining student thinking as teachers implement a new curriculum requires teachers to draw on their CK and PCK, but is at the same time authentically what teachers are responsible for improving in their classrooms, namely student learning. The specific details of the teacher professional development course we designed are provided below. Thus the second of our two research questions was to assess the degree of correlation between the levels of post-CK and post-PCK that teachers attained and the impact of the PBS curriculum on students. We hypothesized that there was such a correlation, and it is on this basis that we would provide professional development to observe the maximum possible impact of the PBS curriculum on minority students' science achievement and attitudes. We aimed to both confirm and quantify such a correlation between teacher knowledge and student outcomes when using the PBS curriculum. A positive correlation would suggest that attaining higher levels of CK and PCK would allow the teachers' use of the PBS curriculum to have an even greater impact on minority students' achievement and attitudes.
“I, Bio” PBS Curriculum Context
Here we summarize the PBS “I, Bio” curriculum (Kanter, 2009; Kanter, Kemp, & Reiser, 2001) for the reader, including the nature of the project, its inquiries, related use of technology, and related science content, which serves at the context for our investigation. This middle school PBS curriculum takes approximately 10–12 weeks in the classroom and has been previously used in a pilot study (Kanter, 2009). Lesson One asks the driving question: What will it take to redesign our school lunch choices to meet our bodies' needs? Students figure out that to promote better health, they will need to learn how to measure the amount of energy that food adds to their bodies and the amount of energy that activities use up from their bodies. Using these data, students design school lunch and activity choices to balance these two measurements. Lessons Two through Four focus on completing the first piece of the project wherein students devise the means by which to measure the energy in food, and in so doing learn science content about energy forms and interconversions, chemical change (as related to oxidation), and the properties of matter (including topics such as specific heat capacity). Lesson Two asks the question: Where can we find the energy in food? Lesson Three asks the question: How does hidden energy in food compare to our bodies' stored energy? And Lesson Four asks the question: How can we measure heat to measure the energy in food? This first piece of the project culminates with students being guided to devise direct calorimetry. This technique requires burning food inside an oxygen-filled chamber that is surrounded by water. The heat from the burning food increases the temperature of the water, and this temperature change can be used to calculate the number of calories in food.
Lessons Five through Seven focus on the second piece of the project wherein students devise the means by which to measure the energy used up doing activities, and in so doing learn additional target science content about levels of biological organization, the body's organ systems and their integrated function, and cellular respiration. Lesson Five asks the question: Where in our bodies are energy stores used up? Lesson Six asks the question: How do the ingredients for energy get to and into every working cell? And Lesson Seven asks the question: What can we measure, and how would we do so, to measure the energy used up by all working cells? This second piece of the project culminates with students being guided to devise indirect calorimetry. This technique uses a one-way valve to collect a student's expired air while doing a given activity. Using oxygen and volume sensors, the amount of oxygen consumed is calculated (Consolazio, Johnson, & Pecora, 1963), and this measurement indirectly determines the calories used up by all the body's working cells. Lesson Eight completes the project. Students redesign their school lunch and activity choices to balance the measurement of the calories consumed in school lunch choices with the calories used up doing chosen activities, thus promoting better health.
Professional Development Context
We next describe the professional development program that is also part of our investigation. Teachers received their professional development in the form of a for-credit graduate-level course in the M.S.Ed. program (crosslisted in biology), which met in the evenings for 3 hours each week for 10 weeks, concurrent with teachers teaching the PBS “I, Bio” curriculum in their own classrooms. All teachers participating in this study attended all sessions of the professional development. In preparation for a given week's session, teachers would review the upcoming “I, Bio” lesson and complete “content” and “student prior conceptions” readings about the big science idea in that lesson. For that same big science idea, teachers would use the readings to draft a “levels of understanding,” a path through successively more sophisticated ways of thinking about the big science idea, that is anchored at one end by students' prior conceptions about the idea and at the other end by what the science discipline knows to be true about the idea (Committee on Science Learning, Kindergarten Through Eighth Grade, 2007). Teachers would then review video clips. We selected these video clips for presenting puzzling student conceptions about the big science idea. Teachers would argue from this video evidence about where students fell along teachers' draft “levels of understanding.” This preparatory work was designed to build teachers' CK and PCK. Then, during the graduate class, teachers discussed the readings and also received additional content lectures activities or laboratories designed to further teach the science content. They then used what they learned to develop a common classwide “levels of understanding” for the big science idea, which they then used to critique each others' analyses of the video clips. Instructors would also engage the teachers as learners by modeling the upcoming “I, Bio” lesson for them. At this point, teachers would be prepared to go back to their classrooms to teach the next “I, Bio” lesson, but with a vigilant eye for students' ideas about the big science idea. For homework, teachers would record data from their own classroom in the form of notes or video, and analyze these data to determine where their students fell along the “levels of understanding” for the big science idea before and after the lesson. Teachers would argue from this evidence to describe their students' initial, changing, and final ideas. This homework was completed in essay format for two “I, Bio” lessons—one emphasizing calorimetry content and the other body systems content. These essay assignments were the means by which we gathered data on each teacher's PCK. The assigning of quantitative PCK scores to these essays is described below.
Analyses were performed with nine sixth- through eighth-grade urban science teachers who used the middle school PBS curriculum “I, Bio” with the students in their classrooms. Participating teachers were recruited into the study and received graduate credit for the professional development as well as an additional stipend. We believe these incentives combined with the opportunity to use the “I, Bio” curriculum and its related technology kit, and support for doing so, were the reasons why teachers chose to participate in the professional development program and study. Participating teachers came from eight public schools in a large, urban school district in the Midwest. The student populations at five of the participating schools had a majority of underrepresented minority students (100%, 99.8%, 98.9%, 88.0%, and 86.0%) while the percentage of underrepresented minority students was smaller at two of the participating schools (41.5% and 28.9%). All of these schools had majority low socioeconomic status (SES) student populations (77.0% on average). The eighth school had a smaller fraction of both underrepresented minority students (19.1%) and low SES students (14.3%). Three of the participating teachers majored in science education, of whom two had masters degrees. Two of the participating teachers majored in science, of whom one had a masters and one had a doctoral degree. Three of the participating teachers majored in the liberal arts, all of whom had masters degrees. The background of one teacher was unavailable. Participating teachers ranged from 0 to 13 years experience teaching science, with an average of 6.6 years. A total of 301 of these teachers' students had complete data. Of these, 197 students were minority, i.e., from racial and ethnic groups underrepresented in science careers, and were included in our analyses. For these minority students in our sample, 79% were low SES. We also present the percentage of these students meeting or exceeding the fifth-grade Illinois Standards Achievement Test (ISAT) reading test standard as a proxy for demographics about the general science acumen of these students. Because ISAT science test scores were not uniformly available for these students, and performance on state standardized reading tests has been shown to correlate strongly with state standardized science test scores (Nolen, 2003), we note as a proxy for general science acumen of these students that approximately 22% of the minority students in our sample met or exceeded the ISAT reading test standard.
Student Instruments and Coding
To measure student achievement, we administered student pre- and posttests that included items from three levels of cognitive difficulty, constructed using a revised Bloom's taxonomy (Anderson et al., 2001). Low cognitive difficulty items were those that asked students to remember basic facts and vocabulary. Medium cognitive difficulty items were those that asked students to apply the content in circumstances near to those in which they learned these ideas in the curriculum. High cognitive difficulty items asked students to apply the content in a new context further from that in which they learned the idea in the curriculum or to draw new connections between concepts. The content areas covered included all the science content areas taught in the “I, Bio” PBS curriculum, which were as follows: body systems, levels of organization, energy stores in the body, cells as the site of energy transformation, the nature of energy, measuring energy, and diffusion. (See Appendix A for a sample of student achievement items; please contact corresponding author for complete item set.)
Written criteria for what was meant by the content area concepts and how one might employ these concepts as various levels of cognitive difficulty were used to develop items. Three members of the research team used these criteria to review the categorization of all items by content area and cognitive difficulty. Items' categorization or the items themselves were changed until consensus was reached. For this reason, we believe there to be conceptual validity of the items by content area and level of cognitive difficulty. While our analysis is carried out on the basis of this conceptual validity, we also report reliability using the Gutmann split-half measures of reliability since we mixed multiple-choice and constructed response items, using different scoring for each type as described in the note to Appendix A. Student achievement on calorimetry and body systems item subsets are particularly relevant to this study, for which the reliabilities on the pre- and posttest were λ = 0.56 and λ = 0.56, respectively, for calorimetry, and λ = 0.67 and λ = 0.71, respectively, for body systems.
A five-level rubric was used to score the items, which were either multiple choice or open-ended (see Appendix A note). The nature of these levels is described and illustrated in the Results section on students' achievement. Summative scores were calculated for the total test and for content area subsets by taking the mean of the item scores. Coders were three undergraduate biology students, all of whom were trained to code with adequate interrater reliability in the following manner. Coders used training data from a separate cohort to gain familiarity with the scoring rubric and to establish interrater reliability. During training, coders worked in groups using the rubric to score the achievement tests, and disagreements were discussed and resolved by the coders and the research team. Coders then scored a subset of tests on their own, and the correlation between coders for each item was calculated. This process was repeated using the training data until the degree of correlation between the coders for each item reached r = .85. As coders scored the actual achievement tests, the level of agreement for each item was repeatedly verified using random subsets of tests to make sure that scores were assigned as consistently as possible.
We anticipated that the opportunities the PBS curriculum provided to engage in inquiry in the context of a motivating project might positively impact students' attitudes toward science. Therefore, we also collected student pre- and postsurveys covering five attitude constructs (Cronbach's α is provided for each measure): perception of the value and relevance of science (pre-α = .76, post-α = .74), interest in science (pre-α = .84, post-α = .84), sense of efficacy doing general science tasks (pre-α = .75, post-α = .80), sense of efficacy doing science tasks related to the curriculum (pre-α = .71, post-α = .78), and science self-concept (pre-α = .83, post-α = .84). Items from the perception of the value and relevance of science scale determined students' perception of how important science is for what they want to do in the world. These items were adapted from scales used during the Michigan Study of Adolescent and Adult Life Transitions (MSALT, n.d.). The interest in science scale measured students' engagement in and enjoyment of classroom science, and the sense of efficacy doing general science tasks scale assessed students' perceived capability to meet the expectations of science class. The source of these two scales was the Patterns of Adaptive Learning Scales (Midgley et al., 2000). The scale measuring sense of efficacy doing science tasks related to the curriculum was developed by the researchers to assess the students' perceived capability to meet the expectations of the PBS curriculum. The science self-concept items sought to measure students' perception of themselves doing science and were taken from Marsh (1990). Students responded to statements by selecting an answer on a five-point scale (1–5), or six-point scale for science self-concept (1–6), ranging from “not at all true” to “very true.” Reverse coding of negatively presented statements was used. (See Appendix B for a sample of student attitudes survey items; please contact corresponding author for complete item set.)
To measure the ultimate success of using the “I, Bio” PBS curriculum, we used middle school minority students' educational and occupational plans as reported in a pre- and postsurvey. This survey of students' plans was a tenable alternative to an extended longitudinal study that would track middle school students into high school, college, and careers. This use of self-reported plans is supported by Eccles, Vida, and Barber (2004), who found that middle school college plans were predictive of actual college attendance at age 20. Middle school students are already developing their career preferences (Vondracek, Silbereisen, Reitzle, & Wiesner, 1999). We used five plans constructs that asked students to rate how likely they would be to attend college (Eccles et al., 2004), to take science classes in college, to major in science in college, to want a job like science class, and to seek a job that uses science. (See Appendix C for student plans survey items.) These were also measured on a five-point scale (1–5).
Teacher Instruments and Coding
To determine the degree to which teachers' CK and PCK levels initially varied and improved in our professional development course, as well as the extent to which higher levels of teachers' post-CK and post-PCK were associated with greater impact of the “I, Bio” PBS curriculum on minority students' science achievement and attitudes, we designed instruments to measure teacher CK and PCK. These instruments focused in on two subsets of all the science content taught in “I, Bio”: science content related to calorimetry (the nature of the potential energy in food and how to measure that energy as heat) and science content related to body systems (how the body's cardiovascular, respiratory, and digestive systems work together to use up the body's energy doing the body's work). In these specific science content areas, we measured teachers' CK and PCK prior to their beginning our study and then again after teachers had received instruction on each topic during the professional development.
We measured teacher CK as the percentage correct on a content assessment related to the calorimetry and body systems content. Teachers completed the post-CK assessment immediately before teaching the “I, Bio” lesson that focused on that specific content. (See Appendix D for teacher content knowledge items.) A five-level rubric was used for scoring teacher content items. Each teacher received a calorimetry pre- and post-CK score and a body systems pre- and post-CK score based on the percentage of possible points achieved on that assessment. An average post-CK score was also created by taking the mean of the calorimetry post-CK and body systems post-CK scores.
We assessed teacher PCK using a coding technique designed to yield numerical scores that could be used in quantitative analyses (Kanter et al., 2006). The two post-PCK measurements required teachers to complete a detailed written analysis of a videotape of their teaching of the “I, Bio” lessons related to calorimetry and body systems content. Teachers' were instructed to emphasize in their essays “episodes” with particular features that had been defined at the beginning of the course and that had since been practiced in all homework assignments and classwork. On average, teachers' post-PCK essays contained 34 episodes. Episodes were defined as sequences wherein the teacher wrote about three particular elements. First, they recognized something specific a student said or wrote or did in class that had the potential to be analyzed for evidence of how the student was thinking about a science concept. Second, they described how the student was thinking about that science concept, i.e., describing the student's network of ideas interrelated around that science concept and ways the student could use those ideas to solve problems, that is to say locating that student's ideas about the science along the “levels of understanding” and sharing why that placement made sense given the classroom evidence. Third, they generated an appropriate follow-up response that would confirm the location of the student on the “levels of understanding” and/or move the student along the “levels of understanding,” in addition to justifying why the follow-up made sense. By the time these essay data were collected, teachers were well practiced in writing episodes of this sort that included elements from each part of this idealized sequence. They had been using of a guiding “organizer” during the first 8 weeks of homework and classwork that supported them writing such episodes. For the PCK essays, the teachers expanded their organizer entries into full sentences, the better for us to understand their thinking. We specifically tasked them to frame their essays using episodes. As the first step in our coding technique, we used the sequence of episode elements, described above, as criteria for dividing teachers' essays back into episodes. Sometimes teachers' episodes only employed some of the sequence elements, which is one aspect of how the episodes were scored, described next.
Each episode was given a score between 1 and 7 based on the extent to which, in that episode, the teacher completed the three-element sequence and in so doing focused on student ideas about the science, correctly interpreting and responding to the way that the student was thinking about the science concept, and reasonably justifying this interpretation and response. While this PCK scoring rubric is quite involved, enough for it alone to be the sole subject of a manuscript we have in preparation, we aim in this paper to present enough information about the PCK scoring rubric that the reader is comfortable that our findings about teachers' PCK are valid. In this limited space, we aim to illustrate the PCK scoring rubric with selected examples of teachers' episodes along with explanations of their scores.
A score of 7, which defines the high end of our PCK scale, requires the following components. The episode must focus on students' ideas about science content from the curriculum as opposed to any other pedagogical issue. It must also have elements in its sequence that are logically connected one to the other such that either an interpretation of the student's idea about the science concept or a follow-up response is based on specific classroom evidence. Furthermore, these episode elements must be deemed correct, insofar as the coders decide that the teacher's conclusion regarding the student's idea about the science concept does seem reasonable given the classroom evidence, or in the case of a follow-up response, that the teacher's suggestion of a way to clarify or change the student's stated idea would be effective. Finally, the teacher's reasoning must be provided to justify why both the interpretation of the students' idea made sense and why the follow-up was appropriate. The following episode received a score of 7 as part of a teacher's post-PCK essay on calorimetry. The teacher focuses on classroom evidence from the energy detection activity that she relates to her students' ideas about energy transformation:
On the first day of this initial part of Spiral 2A, Question 3, we looked over some examples from the energy stations where potential energy had been observed. Students brainstormed ideas for measuring the energy hidden in each object. Their ideas (“snap the glowstick,” “turn on the robot,” “turn on the flashlight and see how long it goes,” etc.) reflected the general concept that they needed to transform the energy in order to measure it. My knowledge of energy transformation and potential energy helped to me classify these comments in this category. Potential energy is energy that is stored, waiting to do something. To measure it, we must see what it can do. Students have the understanding they need in a fragmented way. They can come up with specific tests but have yet to verbalize an all-encompassing idea about how to determine how much potential energy an object contains. They must understand this overall idea to develop a plan for transforming the potential energy in food. In order to clarify whether or not students could generalize this concept, I asked what each of the examples had in common with each other. This question was designed to have students search for words to describe the commonalities between the actions they had suggested. I felt this was an important idea for conceptual change because students needed to step back to see that we would be transforming the energy from the chip to a different form to measure its potential. My understanding of the concept of an energy transformation making the potential energy easier to sense helped me create a question that would encourage students to search for things the objects had in common.
In this case the teacher has chosen some student statements as classroom evidence in response to a question about measuring the energy hidden in each object. These statements are relevant to considering students' ideas about the science content of energy transformation as related to measuring the energy in food. She interprets these classroom data to mean that students' ideas of energy transformation are fragmented. Her interpretation was connected to what the students said and is correct in that she does not overestimate or underestimate the students' understanding given the evidence. She justifies her interpretation with her own understanding that potential energy is energy that is “waiting to do something,” and we must see what it can do to measure it. Her reasoning is connected insofar as she provides a follow-up based on this interpretation. Her follow-up, to ask her students what the stations had in common with each other, was deemed reasonable and correct as it would feasibly have the intended impact. In this way, she could determine whether her students really do not yet see the general principle of energy transformation and also change her students' ideas about energy transformation by allowing the students to discover that all the stations have a form of potential energy that can be transformed to be more easily measured. She further justifies why this follow-up makes sense for changing ideas by saying that it will help her students “step back.” Given the information the teacher provides, this episode receives a score of 7. If she had in this episode justified only her interpretation of the students' idea or only her follow-up, this episode would have received a score of 6.
At the other end of the scale, we assign an episode a score of 1 when the classroom data are not relevant to a science concept from the curriculum in which case the episode also cannot be said to be connected, correct, or justified. The following episode received a score of 1 as part of a teacher's post-PCK essay on calorimetry:
But then Karmyn said “(this activity) made us feel like we were there!” She's thinking hard and actively engaged; as I think they all were today. I'm reminded of my earlier reference to Ross and his emphasis on the need for students to have plenty of opportunities to interpret their ideas into words. Using an inquiry-based project like this had definitely engaged my students and brought them to actively question their beliefs and what is happening.
In this episode, while the teacher has presented classroom data describing what one of her students said, this particular student's statement is not relevant to figuring out what the student thinks about any science content related to measuring the energy in food. The evidence provided is relevant only to the student's feelings rather than the student's ideas about science content. At the same time, it is not at all clear what the student means by “there.” While the student might be intending to say that the class has nearly reached a conclusion about measuring the energy in food, the teacher's interpretation does not address this possibility or consider a clarifying follow-up. Rather, her emphasis is on the affective nature of the student's comment instead of the student's science idea. This episode has many of the required sequence elements, but it lacks an overall focus on a relevant science content idea and as such received a score of 1. An episode that may have no explicitly connected sequence elements, but that contains elements that are focused on relevant science content, will receive a score of 2. Such episodes might include the teacher's conjectures about what the students are thinking about certain science content, but provide no classroom data on which to base that conclusion.
We have located the ends of the scale. Returning to the high end of the scale, we give an episode a score of 5 when the following conditions are met: It contains sequenced elements focusing on the science content, at least two elements are connected, and all of the elements are found to be correct. However, for episodes receiving a score of 5, no justifications are provided for either the interpretation of the student idea or the follow-up. The following episode received a score of 5 as part of a teacher's post-PCK essay on calorimetry:
Jaclyn said, “We should look in the “up” column” and then added “I have evidence!” She went on to ARGUE from evidence. Jaclyn asked the kids to look at the data. She said “All of the “up” columns are around 30 or 32” then saw a 40 and changed to say “most of the “up” columns were around 30–32.” She then went on to note that her group had the lowest at 24 in the up column and she said the reason was that her group used more water. She went on to convince most of the class that we should look at the “up” column and that the amount of water really changed the data but the start and end temperatures didn't really matter. I see this as an incredible leap in the area of experimental design and even in area of measuring energy. She took the class from looking at a variety of numbers to really focusing on the change in temperature and she clearly has the idea that the amount of water must be factored into the end “Amount of Energy” equation. Later at 43:49 Katie answered the question do we want to use the up amount or the end temperature amount with “Honestly, I don't think it really matters what temp you started at because we measured how much the temperature went up. What really matters is up. We started at all different temperatures but all got an up of about 30 degrees.” I see this as really changing ideas concerning measuring energy. These two girls and most of the class agreed that we should be looking at the change.
The teacher picks out a series of student statements that she interprets together to mean that the student not only knows that change in temperature is necessary to measure the energy in food using direct calorimetry, but also that a change in the amount of water will impact the change in temperature and that this must somehow be addressed in the measurement as well. This interpretation of the students' statements is judged to be correct, although the teacher does not explain her logic for how she arrived at this conclusion. Her follow-up asks whether the student wants to use the “up” temperature (change in temperature) or the end temperature to measure the energy in food. This follow-up is correct for this student's idea in that it helps to clarify that this student is indeed moving away from confusing temperature with a measure of energy to a more scientifically correct understanding. Again, the teacher does not explain what makes this an appropriate follow-up. Without justification, the episode receives a score of 5.
An episode with the same sequence of elements would receive a score of 4 if, given the classroom data presented in the essay, the teacher appears to be overestimating or underestimating a student's understanding. These interpretations would still be correct but would be less accurate in light of the classroom data. Finally, an episode that is focused on science content and has connected sequence elements, but ones considered incorrect, either because the interpretation of the student's idea does not follow logically from the classroom data or because the follow-up is not appropriate for clarifying or changing the student idea, would receive a score of 3.
We calculated a calorimetry post-PCK score and a body systems post-PCK score by taking the mean of the episode scores for each content area. We also computed an average PCK score by calculating the mean of the two content specific scores, to act as a more general measure of the level of PCK reached. We obtained pre-PCK measurements for each teacher by having teachers complete similar analyses based on a classroom video of another teacher teaching lessons on the calorimetry and body systems content. Pre-PCK write-ups contained an average of 11 episodes.
PCK as a construct is conceptualized in different ways in the literature. And as a result of being conceptualized in different ways, PCK can be found to be measured in very different ways. Given this, we wanted to emphasize the conceptualization of PCK that we use in this study, which is embodied in the way we measure PCK, which we have just explained. We conceptualize PCK as the usable amalgam of various knowledge types named in the PCK framework of Magnusson, Krajcik, and Borko (1999): knowledge of science curricula, knowledge of students' understanding of science, knowledge of science instructional strategies, and knowledge of science assessments. However, it is the extent to which teachers can use these knowledge types together to engage in various pedagogical reasoning tasks and actions that we aim to measure when we set out to measure PCK, pedagogical reasoning tasks such as confirming how students are thinking about a particular science concept or pedagogical actions such as tailoring instruction to change students' ideas. While measuring as PCK the extent to which teachers can use these knowledge types is clearly based on teachers having the underlying knowledge types, it is not our aim to measure how much of this underlying knowledge teachers have, recognizing the possibility that teachers may have this knowledge but it could be inert or not usable for pedagogical reasoning and actions. Thus, focusing on measuring as PCK the extent to which the aforementioned teacher knowledge types are usable in combination, we derived our measure from ones focused on teachers' PCK as embodied in content-specific pedagogical reasoning tasks and actions (Hill et al., 2005; Loughran, Mulhall, & Berry, 2008), as opposed to those who have conceptualized PCK as the knowings that teachers have and measuring PCK accordingly (Henze, van Driel, & Verloop, 2007; Lee & Luft, 2008).
Finally, as a check on teachers' use of the PBS curriculum, we included questions on the student survey that asked students to select one of the following choices in response to the question “How often did you do this activity in your science class THIS YEAR?”: 1, Never; 2, Rarely; 3, Sometimes; 4, Often; 5, Very Often. The listed activities were inquiry based and built into the “I, Bio” curriculum. (See Appendix E for frequency of inquiry activities teacher survey.) For each teacher, we computed the mean of their students' ratings and used this score as a measure of how often teachers led the class in performing different inquiry activities.
To establish the effects of the “I, Bio” PBS curriculum on minority students, we conducted dependent samples t-tests (n = 197). Specifically, we looked at changes in students' science achievement, attitudes, and plans over the course of the PBS curriculum. We also calculated the effect size shift in student measures from pre to post using Hedges' g. To determine the extent to which teachers' levels of CK and PCK, given their initial variation, approached higher levels with our professional development, we calculated the effect size shift of these scores (n = 9) from pre to post. For PCK, we also calculated the effect size shift of the percentage of episodes receiving a given PCK score, as these percentages shifted from pre to post.
Then we used regression (n = 197) to test empirically the hypothesis that the level of post-PCK and post-CK attained by teachers correlated with greater degrees of impact of the “I, Bio” PBS curriculum on minority students' science achievement and attitudes. In the teacher data set (n = 9), the correlation between post-CK and post-PCK was high and thus we needed to use these variables separately as predictors in our regression models. Average post-PCK and average post-CK scores were significantly correlated [r(6) = .72, p<.05]. Given the small sample of teachers, it is notable that PCK and CK measures were correlated to this extent. We will return to this finding in the Discussion.
To determine the extent to which teacher CK and PCK predicted minority students' achievement, we performed separate analyses regressing the teacher calorimetry post-CK and then post-PCK variables onto students' scores on just that part of the student science achievement test that focused on calorimetry content. We then performed the same analyses using the teacher body systems post-CK and post-PCK variables onto that portion of the student science achievement test that focused on the body systems content. In addition, we regressed the teacher average post-CK and post-PCK variables onto students' average achievement test scores just for the subset of items in both of these content areas.
To determine the extent to which teacher CK and PCK predicted minority students' attitudes and plans, we performed separate analyses regressing the teacher average post-CK score and then the teacher average post-PCK score onto each student attitude and plans variable. We did not further subdivide the analysis and use the calorimetry- or body systems-specific teacher CK and PCK variables to try to predict general minority student attitudes and plans because we had no expectation that teacher knowledge in a particular content area would be any more predictive of students' general attitudes and plans. To predict change in student variables, we used the value of the postvariable as the outcome and the value of both the prevariable and the percentage of minorities in each classroom as covariates.
Impact of PBS Curriculum on Minority Students' Achievement, Attitudes, and Plans
Our results showed that minority student achievement increased with the use of the PBS curriculum (Table 1). On the five-level rubric, Level 1 is completely incorrect, Level 2 is partially correct, Level 3 is correct, Level 4 is firm understanding, and Level 5 is sophisticated understanding. For this reason, a statistically significant mean shift of half a level or more is believed to be consequential in our study. This can be illustrated by considering a mean score shifting from being more often completely incorrect on average (e.g., Level 1.8) on the pretest to being more often partially correct (Level 2.3) on the posttest. We would judge such a shift of half a level or more to be consequential. For example, as per Appendix A: Student Achievement Items, for open-ended calorimetry achievement item 15a on measuring energy, students were asked, “You burn some food and collect the heat from the fire. The 50 grams of water you were using to collect the heat changes in temperature from 27°C to 34°C. How much energy was in the food you burned? Explain why you made the calculations that you made.” At Levels 4 and 5, firm and sophisticated understanding, a student's explanation would include a correct calculation of energy that accounts for both temperature change multiplied by the amount of water in addition to a less or more extensive explanation of why this procedure makes sense. A typical Level 5 response might be as follows: “The change in temperature (34°–27°) tells how much heat energy each gram of water collected. Multiplying by the amount of water in grams tells how much heat energy all of the water together collected (the energy released when the food burned).” For the same item, for a Level 2 response, partially correct, a student would not use the correct calculation, even without an explanation (as one would find for a Level 3, or correct, response). Instead, a student would use an incorrect calculation, most often one that would rely solely on the change in temperature of the water and fail to take into account the amount of water. Still, there would be some connection in the explanation between the food energy and the energy of the heat of the flame. A typical Level 2 response might be as follows: “I subtracted 27° from 34° because the energy makes the temperature rise.” In contrast, a Level 1 response would have neither element. The levels for each open-ended item were treated similarly in the rubric we developed. Thus, we aim to illustrate by way of this example that there is a significant difference in understanding even between Level 1 and Level 2 responses and as such we treated a mean shift of half a level as consequential. Using this criterion, we found consequential improvement in student achievement on the test as a whole and on all items of low cognitive difficulty, medium cognitive difficulty, and high cognitive difficulty on the test as a whole.
Table 1. Change in Minority Student Science Achievement, Science Attitudes, and Science Plans
|Student Science Achievement|
|All items*||1.24 (0.44)||1.84 (0.52)||1.26|
|Low cognitive difficulty*||1.82 (0.67)||2.39 (0.59)||0.89|
|Medium cognitive difficulty*||1.37 (0.50)||2.09 (0.60)||1.30|
|High cognitive difficulty*||0.53 (0.39)||1.05 (0.56)||1.09|
|Calorimetry items*||1.28 (0.56)||1.88 (0.55)||1.07|
|Low cognitive difficulty*||2.25 (0.97)||2.58 (0.78)||0.37|
|Medium cognitive difficulty*||1.03 (0.62)||1.78 (0.72)||1.12|
|High cognitive difficulty*||0.64 (0.63)||1.31 (0.70)||1.00|
|Body systems items*||1.41 (0.61)||2.15 (0.80)||1.05|
|Low cognitive difficulty*||1.69 (0.96)||2.48 (1.09)||0.77|
|Medium cognitive difficulty*||1.90 (0.86)||2.39 (0.83)||0.58|
|High cognitive difficulty*||0.63 (0.53)||1.54 (1.12)||1.02|
|Student Attitudes Toward Science|
|Perception of the value and relevance of science*||3.73 (0.69)||3.55 (0.70)||−0.26|
|Interest in science*||3.69 (0.85)||3.49 (0.88)||−0.23|
|Sense of efficacy doing general science tasks*||3.92 (0.71)||3.77 (0.81)||−0.20|
|Sense of efficacy doing science tasks related to the curriculum||3.52 (0.83)||3.47 (0.88)||−0.06|
|Science self-concept||4.28 (1.05)||4.25 (1.04)||−0.03|
|Student Science Plans|
|Plans to attend college†||4.56 (0.79)||4.45 (0.96)||−0.13|
|Plans to take science classes in college*||3.54 (1.21)||3.24 (1.27)||−0.24|
|Plans to major in science in college*||3.17 (1.26)||2.79 (1.23)||−0.30|
|Desire for a job like science class||2.60 (1.35)||2.64 (1.31)||0.04|
|Plans to seek a job that uses science||2.87 (1.37)||2.75 (1.33)||−0.09|
The normal effect size science achievement gains children make from one year to the next can provide another frame of reference for interpreting the effect size shifts we observed in student achievement. The effect size gain on nationally normed science tests as students transition between any two middle grades is on average ES = 0.23 (Bloom, Hill, Black, & Lipsey, 2009). In our study, students' achievement gain on the test as a whole was ES = 1.26, 5.5 times greater than this benchmark. Students' gains on items of low cognitive difficulty, medium cognitive difficulty, and high cognitive difficulty were ES = 0.89, 1.30, and 1.09, respectively, or 3.9, 5.6, and 4.3 times greater than this benchmark, respectively.
Changes in the two specific content areas we chose to focus on for the purposes of measuring teacher CK and PCK were similar to changes in student achievement across all content topics. Using the benchmark of a half a level mean shift, minority students' calorimetry achievement increased consequentially overall and on items of medium and high cognitive difficulty, but not for items of low cognitive difficulty. Against the effect size benchmark, overall and high cognitive difficulty calorimetry achievement was 4.6 and 4.3 times greater, respectively. In addition, using the criteria of half a level mean shift, minority students' body systems achievement increased consequentially overall as well as on items of all cognitive difficulty levels. Against the effect size benchmark, overall and high cognitive difficulty body systems achievement was 4.6 and 4.4 times greater, respectively. However, while there was improvement in minority student achievement overall, given that none of the posttest means reached a level of 3 or higher, there was still room for significant improvement in achievement.
There were also some statistically significant declines in minority student attitudes toward science and plans to pursue science (Table 1). Specifically, there were decreases in perception of the value and relevance of science, interest in science, and sense of efficacy doing general science tasks. There were also statistically significant declines in minority students' plans to take science classes in college and plans to major in science in college. Although the changes in these attitudes and plans were unexpectedly negative, the magnitude of these effect size shifts was smaller than the larger effect size increases in student achievement.
In both the calorimetry and body systems content areas, the teachers who taught the “I, Bio” curriculum with the accompanying professional development course (n = 9) ended up with levels of post-PCK dramatically higher than those with which they began. On the PCK scale of 1–7 described above, teachers' calorimetry post-PCK reached, on average, 2.93 (SD = 0.43) from a pre-PCK score of 2.15 (SD = 0.63), a shift of 1.45 effect size units. On the same scale, teachers' body systems post-PCK reached, on average, 3.10 (SD = 0.76) from a pre-PCK score of 2.23 (SD = 0.57), a shift of 1.3 effect size units. Overall, teachers' average post-PCK increased to 3.01 (SD = 0.52) from 2.22 (SD = 0.55), a shift in PCK of 1.48 effect size units over the course of the professional development, in spite of initial variation in teachers' PCK. For average post-PCK to have reached a score of approximately 3 is important because it is at this level that teachers have developed the level of PCK where they are basing interpretations of students' ideas about the science content on classroom observations. We expect this might be the minimal level of PCK required for teachers to use the PBS curriculum as designed. Another way to see the difference in the level of PCK teachers attained with the professional development, compared to prior, is to look at the effect size shifts in the average percentage of episodes receiving a particular score. In both content areas, there were effect size decreases in the percentage of episodes receiving scores of 1 and 2 and increases in the percentage of episodes receiving scores of 4, 5, 6, and 7 (Table 2). This is another way to see that the post-PCK level teachers attained showed far more of the high-quality episodes that one would expect to correlate with better use of the PBS curriculum and that teachers' post-PCK level is different in substantive ways from teachers' initial PCK level.
Table 2. Change in Teacher PCK: Percentage of Episodes Receiving Each Score
|1||20.85 (23.07)||13.49 (10.13)||−0.41||22.47 (28.97)||13.09 (22.94)||−0.36||22.50 (24.45)||13.29 (14.33)||−0.46|
|2||50.75 (18.20)||34.90 (7.97)||−1.13||49.76 (22.53)||32.62 (13.98)||−0.91||49.34 (16.30)||33.76 (7.08)||−1.24|
|3||14.09 (16.24)||15.30 (5.58)||0.10||14.23 (8.55)||14.39 (9.89)||0.02||14.51 (11.67)||14.84 (7.17)||0.03|
|4||8.51 (9.87)||20.31 (7.24)||1.36||9.81 (13.58)||18.87 (8.08)||0.81||9.12 (11.23)||19.59 (5.06)||1.20|
|5||4.95 (8.57)||14.28 (11.27)||0.93||3.74 (7.26)||12.97 (9.10)||1.12||4.05 (5.29)||13.63 (8.60)||1.34|
|6||0.78 (2.21)||1.19 (1.90)||0.20||0 (0)||8.02 (7.57)||1.50||0.45 (1.18)||4.60 (4.19)||1.35|
|7||0 (0)||0.54 (1.54)||0.50a||0 (0)||0 (0)||0||0 (0)||0.27 (0.77)||0.50a|
These teachers also achieved high levels of post-CK in both the calorimetry and body systems content areas. Measured from 0 to 100% correct, teachers' calorimetry post-CK reached, on average, 57.19 (SD = 7.25) from a pre-CK score of 42.81 (SD = 5.58), a shift of 2.22 effect size units. Measured the same way, teachers' body systems post-CK reached, on average, 71.25 (SD = 11.11) from a pre-CK score of 40.81 (SD = 12.48), an increase of 2.58 effect size units. Overall, teachers' average post-CK increased to 64.22 (SD = 8.44) from 41.81 (SD = 7.81), a shift of 2.76 effect size units. Although teachers varied at the outset, there were substantial changes in their CK with the professional development. In addition, the average score per content item across all teachers was approximately 3, which is the first level that represents a correct answer on the scoring rubric, using the same five-level rubric used for the student achievement test, described above. Thus, this level of content knowledge that teachers attained is believed to be consistent with capable use of the PBS curriculum.
Connecting Teacher Knowledge to Minority Students' Achievement
We found that, although teachers' CK and PCK began at a variety of levels, teacher knowledge could be improved with professional development to levels expected to be consistent with good use of the PBS curriculum and to support anticipated increases in minority student achievement, attitudes, and plans. To account for nesting of students within teachers' classrooms, we used mixed linear model analyses (SPSS 13.0, 2006) to test the hypothesis that the level of post-PCK and post-CK attained by teachers correlated with greater degrees of impact of the “I, Bio” PBS curriculum. We ran our models using the MIXED procedure that takes into account the clustering of students within teachers. The results were identical to those we obtained when we ran our models using ordinary least squares (OLS) regression analyses, which are the results we report throughout. As anticipated, analyses using the student data (n = 197) showed that both teacher average post-CK and average post-PCK were predictors of minority student achievement in mastering the related calorimetry and body systems content at all levels of cognitive difficulty (Table 3). Both average post-CK and average post-PCK were especially predictive of minority student success in mastering medium and high difficulty content items.
Table 3. Effect of Teacher Average Post-CK and Post-PCK on Science Achievement of Minority Students Doing the PBS Curriculum (Calorimetry and Body Systems Items Only)
|Teacher average post-CK||.448*||.41||.367*||.17||.345*||.32||.450*||.42|
|Teacher average post-PCK||.443*||.39||.335*||.14||.349*||.32||.455*||.41|
We next performed analyses linking teacher knowledge in a specific content area to student achievement in that same content area. As anticipated, we found that in analyses specific to calorimetry content (Table 4) as well as in analyses specific to body systems content (Table 5) that teachers' post-CK and post-PCK in a specific content area were statistically significant predictors of minority student achievement in mastering the same content at all levels of cognitive difficulty. Within calorimetry, teacher CK and PCK in calorimetry were best able to predict change in student achievement on the medium difficulty items as indicated by the percent of variance accounted for (R2); within body systems, it was change in student achievement in the high difficulty items for which teacher CK and PCK in body systems had the most predictive power. This suggests that teacher CK and PCK are important for students learning to apply concepts, getting students beyond recalling basic facts.
Table 4. Effect of Teacher Calorimetry Post-CK and Post-PCK on Science Achievement of Minority Students Doing the PBS curriculum (Calorimetry Items Only)
|Teacher calorimetry post-CK||.331*||.25||.223*||.05||.276*||.27||.243*||.15|
|Teacher calorimetry post-PCK||.366*||.27||.176*||.03||.399*||.34||.258*||.16|
Table 5. Effect of Teacher Body Systems Post-CK and Post-PCK on Achievement of Minority Students Doing the PBS Curriculum (Body Systems Items Only)
|Teacher body systems post-CK||.468*||.37||.359*||.19||.268*||.19||.489*||.39|
|Teacher body systems post-PCK||.417*||.31||.216*||.11||.243*||.17||.512*||.38|
Connecting Teacher Knowledge to Minority Students' Plans and Attitudes
While we saw improvements in teacher knowledge correlated with improvements in minority students' achievement, especially so for those science content items that emphasized more advanced reasoning, we did not see on average the anticipated positive impact of the PBS curriculum on minority students' attitudes or plans. In light of our initial assumption that teachers' higher knowledge levels would correlate with improved minority student outcomes with a PBS curriculum, we considered the possibility that the variations in individual teacher's post-CK and post-PCK might effect teacher-specific variations in these teachers' students' outcomes, such that that the impact of the PBS curriculum on minority students' attitudes and plans might indeed depend on teacher knowledge as assumed, but this might not be apparent when looking at mean student outcomes. Therefore, we used the student data (n = 197) to perform regressions predicting change in minority students' attitudes and plans using the teacher average post-CK and post-PCK variables.
Contrary to our expectations, teacher average post-CK and average post-PCK were statistically significantly correlated to declines in minority students' perception of the value and relevance of science, interest in science, and sense of efficacy doing science tasks related to the curriculum. Teacher average post-CK was also related to declines in minority students' sense of efficacy doing general science tasks (Table 6). We found significant positive relationships between changes in the student science attitude variables and changes in several student science plans variables (Table 7). Given the finding that as attitudes go, so go plans, increased teacher CK or PCK, which were correlated with decreased student attitudes, would be indirectly correlated with decreased students' plans as well. This would be the case even though we found no significant direct relationships between teacher average post-CK or average post-PCK and minority students' science plans.
Table 6. Effect of Teacher Average Post-CK and Average Post-PCK on Science Attitudes of Minority Students Doing the PBS Curriculum
|Teacher average post-CK||−.140*||.29||−.196*||.37||−.119*||.40||−.128*||.30|
|Teacher average post-PCK||−.133*||.29||−.207*||.37||−.098||.40||−.148*||.31|
Table 7. Effect of Change in Science Attitudes on Science Plans of Minority Students Doing the PBS Curriculum
|Change in perception of the value and relevance of science||.317*||.20||.335*||.29||.304*||.22||.256*||.28|
|Change in interest in science||.193*||.14||.250*||.24||.240*||.18||.098||.23|
|Change in sense of efficacy doing general science tasks||.191*||.14||.202*||.23||.268*||.20||.128||.23|
|Change in sense of efficacy doing science tasks related to the curriculum||.142*||.12||.168*||.21||.172*||.16||.098||.23|
Connecting Teacher Inquiry Practices to Minority Students' Plans and Attitudes
While higher levels of teachers' post-CK and post-PCK correlated with improvements in minority students' achievement, especially for those items that target more advanced reasoning, these high levels of teacher knowledge showed some negative associations with attitudes, which would also be expected to negatively impact students' plans. Might there be another teacher variable that the literature suggests should correlate to improving student attitudes and plans that we are not measuring when we focus on teachers' CK and PCK? In some studies, teachers' science teaching practices, including doing hands-on activities and involving students in cooperative learning, have been shown to positively influence student attitudes (see Haladyna & Shaughnessy, 1982; Druva & Anderson, 1983). In addition, Koballa (1988) found evidence that activity-oriented science instruction can help develop students' favorable attitudes toward science. Might this be the case here? We considered the possibility that the frequency with which teachers do various types of inquiry activities designed into the PBS curriculum might positively correlate with minority student attitudes and plans. Student survey-based reports of their teachers' inquiry activity frequency had been collected. For all the teachers in our sample, we computed an average frequency score for each inquiry activity using the responses from their students. We found no correlation between the frequency with which teachers did these inquiry activities and their post-CK or post-PCK, suggesting that inquiry activity frequency had the potential for a different relationship to minority students' attitudes and plans than that for the teacher knowledge variables, which would speak more to the teachers' ability to draw out or change students' ideas. To test this hypothesis, we used the inquiry activity frequency scores in regressions predicting change in minority student achievement, attitudes, and plans.
Where we had found CK and PCK to be related to decreases in minority students' attitudes, analyses using student data (n = 197) showed several statistically significant positive relationships between the frequency with which teachers did certain inquiry activities and students' attitudes. The frequency of teachers supporting minority students explaining concepts to one another predicted increases in students' perception of the value and relevance of science, interest in science, and sense of efficacy doing general science tasks (Table 8). All three of these attitude variables predicted positive changes in three minority students' plans variables: plans to take science classes in college, plans to major in science in college, and the desire to have a job like science class. Increases in minority students' perception of the value and relevance of science also correlated with increased plans to seek a job that uses science (Table 7). We also found that frequency of teachers supporting minority students analyzing data was related to students' increased sense of efficacy doing general science tasks (Table 8), which in addition to its positive correlation to the plans variables listed above (see Table 7), also predicted students' increased plans to attend college [β = .151, p < .05, R2 = .28]. Through these avenues, the frequency with which teachers did certain inquiry activities in the PBS curriculum ultimately had a positive relationship with minority students' science college and career plans as well as science attitudes, where teacher knowledge types (CK and PCK) did not. Additional analyses showed several direct positive associations between inquiry activity frequencies and minority students' plans themselves. Regressions using inquiry activity frequency demonstrated a statistically significant positive relationship of higher frequencies of teachers supporting students explaining concepts to one another with students' plans to major in science in college [β = .151, p < .05, R2 = .23]. The relationship between higher frequencies of teachers supporting minority students doing hands-on science investigations and students' plans to major in science in college was borderline positive [β = .126, p < .057, R2 = .23]. We also found that frequency of teachers supporting minority students engaging in designing and doing science investigations was significantly related to increases in students' plans to major in science in college [β = .150, p < .05, R2 = .23]. The other two of the six frequency of inquiry activity variables (working in groups and drawing conclusions based on data) were not found to have statistically significant relationships to any attitudes or plans variable. Why we might have seen these correlations for frequency of particular inquiry activities but not others is considered in the Discussion.
Table 8. Effect of Frequency of Teachers Supporting Students “Explaining Concepts to One Another” and “Analyzing Data” on Science Attitudes of Minority Students Doing the PBS Curriculum
|Supporting students explaining concepts to one another||.180*||.30||.139*||.35||.155*||.39|
|Supporting students analyzing data||.118†||.26||.09||.31||.134*||.39|
Previous studies as cited above suggested that a PBS curriculum might better leverage the strengths of middle school students from ethnic and racial groups underrepresented in science careers, improving their science attitudes and achievement in such a way as to increase the number who go on to pursue science in college and careers. Our aim was to assess the extent to which this assertion was true. We took into account the effect of teacher knowledge, since previous studies suggested that teacher CK and PCK levels would impact how well teachers could use a PBS curriculum to these ends. For this reason, we provided professional development to raise teachers' CK and PCK before they used each PBS lesson to assess the greatest possible impact the PBS curriculum could have on minority students. We found that although teachers began with a range of levels of CK and PCK, this knowledge was tractable with our professional development. Teachers achieved high levels of both CK and PCK, and the knowledge levels teacher reached were believed to be sufficient to support teachers' best use of the PBS curriculum.
While the PBS curriculum was of relatively short duration, and as such one might expect that the extent of its impact would be circumscribed, PBS also represents a significant departure from the instructional status quo in the urban science classroom and as such we might anticipate an impact even from this short-duration PBS intervention. We documented some improvement in minority students' science achievement with the PBS curriculum. At the same time, we documented some declines in minority student attitudes toward science and plans to pursue science careers, in particular students' plans to pursue further science studies in college. That students' attitudes unexpectedly declined may speak to just how different the nature of the PBS instruction was perceived to be by the students. Students may have been made uncomfortable by the significant changes to their science class norms as instituted with the PBS curriculum. Making this shift in norms apparent and palatable to students has implications for the design of the PBS curriculum and professional development. We might ultimately find that students' attitudes increase as PBS becomes the norm in the science classroom. For students' plans, we recognize that there are many factors over the years after middle school that might prevent students' plans from being realized. Were we to follow students over time, we might ultimately find the decline in students' self-reported science plans plays out differently when we examine the relationship between PBS and students' actual entry into college and/or careers in science.
Teacher CK and PCK were found to be predictive of minority student success mastering the related science content. These findings are consistent with a growing body of research linking teacher subject-specific CK and PCK to student achievement in those subjects. As related to teacher CK, our findings are consistent with studies from the educational production function literature (Mullens, Murnane, & Willett, 1996; Rowan et al., 1997) that measure teachers' subject-specific knowledge more directly (as opposed to proxies like number of postsecondary subject-matter courses) and show the importance of this knowledge for student learning. We extend these findings by showing that this same connection between teacher content knowledge and student achievement exists not just for science in general, or even for a specific disciplinary area of science, but also for specific science concepts like calorimetry and body systems. As for PCK, our measure of PCK is similar to the measure of effective teaching used by Johnson, Kahle, and Fargo (2006) who found a correlation between teachers' proficiency in working with students and students' content knowledge growth. They also found this relationship to hold for just the minority students in their sample, further corroborating our findings. Our findings of the connection between teachers' subject-specific PCK and student achievement in those subjects are also consistent with those of Hill et al. (2005). Using a measure of teachers' “content knowledge for teaching mathematics” (CKT-M) that encompasses attributes of our teacher PCK as well as CK measures, these researchers showed that CKT-M was a significant predictor of gains in students' mathematics achievement. In our case, we can distinguish the independent contributions of teacher CK and PCK, and our measure of PCK is based on teachers' analysis of their own instructional practices rather than on a multiple-choice assessment such as that used by Hill et al. While in some ways this is a methodological improvement, teachers' ability to communicate their ideas clearly in essay writing becomes a confounding factor in our measure of PCK. Nevertheless, our results show a significant connection between teacher learning (both CK and PCK) and student achievement, a relationship that has been considered difficult to establish. At the same time, it must be recognized that our findings are specific to a PBS curriculum context. So, we found not only that a PBS curriculum could improve minority student achievement, but that carefully designing teachers training on the PBS curriculum to bolster CK and PCK in the science content areas specific to the PBS curriculum was an effective way to improve minority student achievement even more. Additional research would be warranted to determine whether improving these teacher knowledge types via professional development could have a similar impact beyond just PBS curricula to the use of a variety of types of science curricula. It would be important to know whether there is a general impact on underrepresented student achievement across science curricula of bolstering teachers' CK and PCK, which we suspect might be the case, but this question cannot be addressed by this initial study. Additional research would be required, conducting a similar study with teachers teaching the same content using different types of curricula, with a program of practice-based professional development tailored to each type of curriculum.
It is also important to note that about 60% of the variance in minority student science achievement on the calorimetry and body systems items together was not accounted for by teacher CK or PCK. In addition, the frequency of teachers' use of inquiry activities was not found to be predictive of minority student science achievement in this study. Thus, it will be necessary to conduct additional research to determine other important teacher variables that explain minority student achievement.
Furthermore, while we used different instruments to measure teachers' CK and PCK, the fact that we found these two teacher knowledge variables to be highly correlated suggests the interconnectedness of these two knowledge types. Teachers' content knowledge of certain science subject matter and teachers' subject matter-specific pedagogical knowledge used in teaching that science subject matter appear to be mutually reinforcing both in general and also at the level of specific science subjects. It will be important to consider the interconnectedness of these two types of teacher knowledge in designing a teacher professional development program.
However, we did not see the anticipated increases in attitudes toward science and plans to pursue science. In fact, we saw decreases in some of these measures. These decreases in attitudes toward science are consistent with the results of studies by George (2006) and Simpson and Oliver (1990). They are also are consistent with conclusions by Koballa (1988) that students' attitudes drop as they progress through middle school. In addition, our results are quite similar to those from Cannon and Simpson (1985) who showed that students' attitudes toward science decline across middle and high school, even as students' science achievement rises. While Cannon and Simpson (1985) do not report findings for the minority population overall, they report declines in attitudes for minorities by ability and gender subgroups. The effect size shifts for these subgroups, as we have calculated them from data presented in that paper, range from −0.37 to −0.06. Across the five attitude variables in our study, minority student attitude effect sizes range from −0.26 to −0.03 (Table 1). Our findings as related to attitudes in general are similar to those of the Cannon and Simpson (1985) study. For specific attitudes about the usefulness of school science, our finding is inconsistent with the general results of the George (2006) study. However, our results for this attitude are consistent with the findings of Yager, Simmons, and Penick (1989), whose research found decreasing attitudes about the usefulness of school science across middle school. This may have been due to students gaining more realistic perceptions of the utility of science, leading to a decline in their perception of the value and relevance of science from what was initially an exaggerated perception, which might explain our findings as well. We had hoped that the PBS curriculum would have reversed this trend.
Contrary to our expectations, we found that higher teacher CK and PCK were correlated with decreases in minority student attitudes toward science, and indirectly to decreases in plans to pursue science. While it appeared that knowledgeable teachers could use a PBS curriculum to promote minority student achievement, the same was not necessarily the case for minority students' science attitudes and plans to pursue science careers. One reason for this finding might be that our assumption that teachers achieving an average PCK level of 3 (able to base interpretations of students' ideas about the science content on classroom observations) was simply not enough PCK to impact student attitudes. However, we do not believe that this is the case as we found the same relationship between PCK and student attitudes when we ran the same analyses using only that proportion of teachers' PCK responses at the higher 5, 6, or 7 levels.
However, we found that teachers' frequent use of specific PBS inquiry-based activities in the classroom did positively impact minority student plans directly as well as indirectly through student attitudes. This is consistent with the studies in which teachers' science teaching practices influenced student attitudes (Druva & Anderson, 1983; Haladyna & Shaughnessy, 1982). While in our study, students designing and doing their own science investigations, analyzing data, and explaining concepts to one another more often was correlated to increases in minority students' attitudes and plans, we do not expect that these types of activities need only be conducted in the context of a PBS curriculum, simply that a PBS curriculum may better support these activities, whereas other curricular types may inhibit them. Teachers supporting their students doing these inquiry-based activities may be similarly important for curricula other than PBS. That said, the teacher having students do these inquiry-based activities in her classroom appears to be the unanticipated key to improving minority students' science attitudes and plans with the PBS curriculum. The teacher supporting the frequent doing of these inquiry-based activities, which are an integral part of the PBS curriculum, as opposed to skipping over them, appears to be important, which has implications for improving the design of a PBS curriculum and its professional development to ensure this. Our findings suggest that it would improve the professional development if we were to further train and encourage teachers to use these specific inquiry-based activities as often as possible. With this alteration, the PBS curriculum might have a more positive impact on minority students' attitudes toward science and plans to pursue science. PBS curricula are intentionally designed to support many inquiry activities, and the professional development allowed teachers to experience the modelling of each lesson and its related inquiry-based activities before trying the lesson in the classroom. However, it is possible that teachers did not fully benefit from this aspect of the professional development, as this modeling was not its primary focus. Furthermore, teachers are often pressed for time in the classroom by events outside of their control and may rush through or skip students designing and doing investigations, analyzing data, and explaining concepts to one another, thus vitiating the intended effect of the inquiry and making the PBS curriculum more like traditional classroom seatwork.
It is an important implication of our study that teacher knowledge and aspects of teacher practice are both integral to maximizing the impact of a PBS curriculum on minority students. We see in our results that knowledge on the part of the teacher as necessary to think through conceptual change teaching elicits the desired cognition and learning on the part of minority students, but we did not find this same teacher knowledge to be correlated to other desired outcomes, namely improvement in minority students' science attitudes and career plans. In our findings, it was specific aspects of teacher practice, having students designing and doing their own science investigations, analyzing data, and explaining concepts to one another that were correlated to improved minority students' attitudes and career plans. These findings suggest that the social constructivism (and related conceptual change teaching) on which PBS is based may be insufficient to help students from diverse backgrounds cross from their real-life worlds into the worlds of the science classroom and science in general. What we may be seeing in our findings is the necessity, as discussed in the literature, to attend to culturally relevant pedagogy (CRP) practices (Ladson-Billings, 1995; Patchen & Cox-Petersen, 2008), going beyond just the social constructivist pedagogical practices on which we have focused. CRP goes beyond the social constructivism embedded in PBS to explicitly recognize power relations, using this discovery to improve educational access and opportunities for minority students. What we might be seeing in our findings is the important impact of teachers' CRP pedagogical practices on minority student science attitudes and plans, while social constructivist pedagogical practices only impact minority student science knowledge. The importance in our study of the frequency of minority students designing and doing their own science investigations may be indicative of the importance of CRP practices that redistribute authority. The importance in our study of the frequency of minority students explaining concepts to one another may be indicative of the importance of CRP practices that support the use of students' native language for comprehension.
In addition, there may be ways for teachers to use their CK and PCK to facilitate conceptual change that are also consistent with CRP pedagogical practices. Much of the focus of the professional development was on developing teacher CK and PCK, and we were successful in increasing these types of teacher knowledge. However, we have seen that while strong CK and PCK are invaluable in effecting minority student achievement, we must also be sensitive to the fact that such knowledge appeared to be a liability for minority students' attitudes and plans. The CK and PCK knowledge supports teachers' social constructivist pedagogical practices, but our findings suggest that we need to attend to the specific ways we train and support teachers to use this knowledge to elicit and challenge students' ideas. It might be the case that if this elicitation and challenging of student ideas employed CRP pedagogical practices that recognized the power relations in this interplay, we might ultimately observe a positive correlation between teacher knowledge and student attitudes and plans. To the extent to which this might be found to be the case, it would be important to support CRP pedagogical practices in using CK and PCK to draw out and challenge students' ideas both in the design of the PBS curriculum itself, as well as in the professional development.
While all teachers used the PBS curriculum in their classrooms and also had the same amount and type of support from the professional development program in learning to use the PBS curriculum as intended, we recognize that the credibility of our findings for this study would be improved with direct classroom observations. Such observations would help to corroborate the findings gleaned from teachers' reflective PCK essays. It might be that the PCK revealed in the context of teachers planning for or reflecting on their own practice is different from the PCK revealed in the midst of engaging in that teaching practice. Such PCK as revealed during the practice of teaching might be found to have a different relationship to student outcomes than that which we report herein. Direct classroom observations would also help corroborate students' subjective survey responses about how frequently their teachers had them doing various inquiry-based activities. Again, a more objective measure of teachers' frequency of inquiry-based activities might be found to have a different relationship to student outcomes than that which we have reported. In future studies, classroom observation data would help to validate our findings. Such data would also allow us to see whether the specific ways students are conducting investigations, analyzing data, and explaining concepts to one another, or the ways teachers are eliciting and challenging students' ideas are consistent with CRP pedagogical practices. It would be important to determine if these teacher practices have to be done in a manner consistent with CRP pedagogical practices to observe the better student outcomes.
While we have argued the theoretical basis for why a PBS curriculum and teacher professional development program might reasonably impact minority student science achievement, attitudes, and career plans, and reported our findings, to make progress in proving that such a curriculum and professional development program causes these outcomes, we recognize that we would need to continue our program of research by pursuing a rigorous experimental design that employs a control group. In spite of this limitation, our study design has allowed us to document the extent to which a PBS curriculum could improve minority student achievement, recognizing the degree to which teacher knowledge drives this result. And to the extent to which we did not see the PBS curriculum improve minority student attitudes and plans, we were able to discover the important role that specific teacher-supported inquiry practices played in this relationship. Finally, we speculate that if additional attention were paid in curriculum design and professional development to supporting the frequent use of the specific inquiry-based activities that are consistent with culturally relevant pedagogical practices, as well as helping teachers use CRP practices in how they draw out and challenge students' ideas, we might see a similar positive impact of the PBS curriculum and professional development on minority student attitudes and plans as we did on minority student achievement.