The results of this study showed a statistically significant gain in first-semester mathematics course grades for students who participated in the modeling-based mathematics course compared with a previous cohort of students who had taken a traditional summer course in college algebra or pre-calculus. The modeling-based course was effective in closing the previous letter grade gap in the first mathematics course between participants and non-participants in the bridge program, when controlling for the effect of SAT mathematics score. In each year of the modeling-based course, we found statistically significant gains in student understanding of average rate of change, as measured by the Rate of Change Concept Inventory. By examining the inventory items on which the students showed the greatest improvement, we gain some insight into the ways in which the design and implementation of the modeling-based course may have influenced students' first-semester mathematics performance. We report these results in the following sections.
First Mathematics Course Grade
Participants across cohorts comparison The bridge program students who participated in the modeling-based course (the 2010–2012 cohort) performed significantly better in their first-semester mathematics course compared with the 2007–2009 cohort of students who had taken traditional college algebra and pre-calculus courses, as shown in Table 5.
Table 5. First-Semester Mathematics Grades for Bridge Program Participants
| ||2007–2009 cohort||2010–2012 cohort|| || || |
| ||n||M (SD)||n||M (SD)||t||df||d|
|Pre-calculus||38||1.75 (1.27)||14||3.17 (0.69)||5.11ba||42.5||1.39|
|Calculus I||41||1.28 (1.20)||51||2.20 (1.15)||3.74b||90||0.78|
However, as shown in Table 6, the 2010–2012 cohort had a statistically significant higher SAT mathematics score than the 2007–2009 cohort. Hence we conducted an analysis of covariance (Table 7), with SAT mathematics score as the covariate, which showed that participation in the modeling-based course was a significant factor, but that SAT mathematics score was not a significant factor. Cohen's effect size values, d = 1.39 and d = 0.78, for pre-calculus and Calculus I, respectively, suggest a high practical significance of these results and that the modeling-based course may be even more effective for pre-calculus students than for calculus students.
Table 6. SAT Mathematics Scores for Bridge Program Participants
| ||2007–2009 cohort||2010–2012 cohort|| || |
| ||n||M (SD)||n||M (SD)||t||df|
|Pre-calculus||37||500.8 (53.9)||13||534.6 (33.8)||2.62ba||33.9|
|Calculus I||37||562.7 (70.5)||48||588.8 (47.8)||2.03b||83|
Table 7. Effect of Bridge Program Participation by Cohort with SAT Mathematics Score as a Covariate
|Factor||df||MS||F||Partial eta squared|
|SAT math||1, 50||0.03||0.027||0.00|
|SAT math||1, 85||0.26||0.20||0.00|
Participants versus non-participants comparison We examined the effect that the modeling-based course had on the letter grade gap between participants and non-participants in the bridge program. The fall course grades for these two groups for the 2010–12 cohort are shown in Table 8. However, as noted earlier in Table 2, the SAT mathematics scores for the participants versus the non-participants are significantly different for this cohort. Hence, we performed an analysis of covariance for each course to account for the SAT mathematics scores.
Table 8. First-Semester Mathematics Grades by Bridge Program Participation for the 2010–2012 Cohort
| ||Non-participants||Participants|| || |
| ||n||M (SD)||n||M (SD)||t||df|
|Pre-calculus||131||2.90 (1.11)||14||3.17 (0.69)||0.86||143|
|Calculus I||641||2.55 (1.13)||51||2.20 (1.15)||−2.13a||690|
For the pre-calculus course, the analysis showed that the effect of SAT mathematics scores was not significant. Our results showed that the modeling-based course was effective in closing the previous letter grade gap (shown in Table 3 for the 2007–2009 cohort), even though there was still a significant difference in SAT mathematics scores between the participants and the non-participants in this cohort. Fisher's exact test (used due to small sample size) showed that there was no longer a significant difference in the grade distributions between the participants and the non-participants (p = 0.923).
For the Calculus I course, the analysis of covariance (Table 9) showed that there was no significant difference in the fall Calculus I grade by participation in the modeling-base course, but rather the difference in course grades was explained by the differences in SAT mathematics scores between the two groups (p < 0.001).
Table 9. Effect of Bridge Program Participation on Calculus I Grades with SAT Mathematics Score as a Covariate
|Factor||df||MS||F||Partial eta squared|
|SAT math||1, 618||54.11||45.41a||0.07|
The mean Calculus I course grades, when adjusted for the SAT mathematics score, were nearly identical. The adjusted mean course grade for the non-participants was 2.51 (n = 571, SE = 0.05) and for the participants was 2.51 (n = 48, SE = 0.16). This result means that the modeling-based course enabled the bridge program participants to perform on a par with the non-participants who had comparable SAT mathematics scores; thus the course closed the previous letter grade gap between the participants who in the earlier cohort (2007–2009) had performed below the non-participants (as shown in Table 3). A chi-square test showed that there was no longer a significant difference in the grade distributions between the participants and the non-participants, χ2 (4, 692) = 5.89, p = 0.21.
Understanding of Average Rate of Change
As noted above, we systematically changed the items on the inventory: we added three items from the first to the second year of the course implementation and modified items from the second year to the third year of the implementation. The pre- and post-test results shown in Table 10 indicate that there was a significant improvement (p < 0.001) in the student understanding of the concept of average rate of change for each year of the modeling-based course, with large effect sizes in all three years.
Table 10. Pre- and Post-test Results for the Rate of Change Concept Inventory
| ||Pre-test||Post-test|| || || |
|Year||M (SD)||M (SD)||t||df||d|
|2010||8.86 (3.47)||12.69 (2.81)||10.65a||33||1.21|
|2011||8.05 (3.10)||12.34 (3.16)||7.92a||16||1.37|
|2012||6.92 (3.25)||12.62 (3.30)||12.37a||34||1.74|
As shown in Table 11, there was a significant improvement on the nine common items on the Rate of Change Concept Inventory for the 2010–2012 cohort in the three subscore areas of algebraic expressions, graphical interpretation, and symbolic interpretation, with a maximum score of three points in each area. Because these subscores were not normally distributed, we compared the pre- and post-test results using related-samples Wilcoxon Signed Rank Test and found statistically significant improvements on all three subscores when aggregating the data across all three years of the modeling-based course.
Table 11. Pre- and Post-test Results for the Subscores of the Common Items on the Rate of Change Concept Inventory for 2010–2012 Cohort
| ||Pre-test||Post-test|| |
|Subscore||M (SD)||M (SD)||z|
|Algebraic||1.14 (.99)||2.27 (1.08)||6.503a|
|Symbolic||0.79 (.82)||1.53 (.97)||5.261a|
|Graphical||1.33 (.87)||1.98 (.80)||5.850a|
In the third year of the modeling-based course, the overall scores improved from 6.92 (35%) to 12.62 (63%), as shown in Table 10. There were nine items on the concept inventory for which the improvement was greater than 30%. Four of these were algebraic expression items, four were graphical interpretation items, and one was a symbolic interpretation item. We report the analysis of student gains on these items in relationship to the model development sequence.
Table 12. Algebraic Expression Items with High Percentage Gain
| ||Correct on pre-test||Correct on post-test||Gain|
Item Q19 asked students to find an average speed over a round trip between two cities when given speed (45 miles per hour) and time (two hours) information for the first part of the trip and only speed (30 miles per hour) information for the return trip. The significant improvement on this item was largely accounted for by the number of students who shifted from incorrectly seeing the average speed as a simple average of the two speeds (37.5 miles per hour) to correctly reasoning about this as a weighted average that has to account for the amount of time traveled at each speed, resulting in an average speed of 32.5 miles per hour.
The gains on items Q2 and Q16 (simplifying algebraic expressions for a quadratic function and for a complex fraction) likely reflect the emphasis in the course on developing students' algebraic skills. The gain on item Q12 (finding an average rate of change between two points on a parabola) is likely due to the limits of students' prior experiences with the meaning of the term average rate of change in a mathematical context. This was an open-ended question; without knowing the mathematical meaning of the term, many students simply left this question blank on the pre-test. We attribute the gain on item Q19 (interpreting two speeds over different time intervals) in part to the emphasis in the model development sequence on analyzing and interpreting velocity in the context of motion along a straight path.
Symbolic interpretation items The symbolic interpretation items required the students to create appropriate symbolic expressions when given a problem context or to interpret the meaning of the parameters in symbolic expression. There was a substantial gain on one of these three items, which is taken from Carlson et al. (2010). The students had to choose correct interpretations of the parameter m in a linear growth function:
A baseball card increases in value according to the function, , where b gives the value of the card in dollars and t is the time (in years) since the card was purchased. Which of the following describe what conveys about the situation?
There was a substantial gain (37%) on this question from the pre-test (n = 7, 20%) to the post-test (n = 20, 57%). This gain likely reflects the emphasis in the model development sequence on making meaningful interpretations of data and on giving descriptions of the average rate of change in various contexts.
Graphical interpretation items There were substantial gains on four items that measured student proficiency at reasoning about and interpreting rates of change when given graphical information, as shown in Table 13. We take these forms of graphical reasoning to be central to the work and the learning of engineers and scientists.
Table 13. Graphical Interpretation Items with High Percentage Gain
| ||Correct on pre-test||Correct on post-test||Gain|
Two of these items (Q4 and Q8) addressed the interrelated interpretation of velocity and position graphs. Question Q4 addressed interpreting information about velocity when students were given a position graph. Question Q8, on the other hand, involved interpreting position information when students were given a velocity graph. Students often confuse the interpretation of these graphs; they experience difficulty in inferring velocity information from a position graph and misread a velocity graph as if it were a position graph (Beichner, 1994). This coordination of representational systems (shifting between the velocity or rate graph and its associated position graph) was the main focus of the model exploration tasks described earlier.
Item Q4A asked for the value of the velocity of the person whose position graph is shown in Figure 2 at seven seconds, and item Q4B asked for the person's average speed over the entire time interval. On the post-test, 24 (68%) of the students correctly reasoned about the velocity at a particular point in time when the velocity is constant and the starting point for the motion is not at the origin. There were two common errors on the pre-test. The most common error was to incorrectly assume that the speed would equal position divided by time, or 35 meters divided by seven seconds, which produced an incorrect answer of 5 meters per second. The second most common error on the pre-test was to read the position graph as if it were a velocity graph, which produced an incorrect answer of 35 meters per second. On the post-test, only five students made this first error and none made the second error. The prevalence and persistence of both errors are well established in both the mathematics education and physics education literatures; these results compare favorably with results on an equivalent item by Beichner (1994), where only 21% of students were able to reason correctly about the velocity after college-level instruction in kinematics. On the post-test, 27 (77%) of the students were able to reason correctly on item Q4B about the average of two different constant velocities over two different intervals of time. We note that this item is the graphical equivalent of algebraic item Q19, on which there was also a substantial improvement from the pre- to post-test. The student achievement on the post-test was greater on the graphical form of the item (with 77% correct) than on the algebraic form of the item (with 54% correct).
Item Q8 represents an important reversal of the above problem and one that is a well-known source of difficulty for physics and calculus students (Beichner, 1994; Monk, 1992). The item asks students to identify which of a set of position versus time graphs would best represent the object's motion during the same time interval for the motion shown in the velocity graph in Figure 3. A successful answer to this item requires an understanding of how to reason about position when given a velocity graph.
On the post-test, we found that 25 (71%) students were able to correctly identify an appropriate position graph. On an analogous item reported by Beichner (1994), only 29% of students were able to correctly use areas to reason about velocity from an acceleration graph after instruction in kinematics. The substantial 43% gain on this question suggests that the model exploration tasks within the model development sequence helped the students understand how to use areas to reason graphically about nonconstant and negative velocities (or rates of change), such as shown in Figure 3.
The two other graphical items (Q5 and Q17) for which there were substantial gains addressed student proficiency at choosing a text description of motion when given a position graph (Q5) and in comparing the average rates of change by reasoning about the outputs of three functions over differing input intervals (Q17). In question Q5, the students were asked to choose a description of the motion of an object whose position is shown in Figure 4.
The post-test results showed that 80% of the students were able to select a correct description; results reported by Beichner (1994) on this item showed that only 37% of students were able to describe the motion correctly after kinematics instruction. We attribute the substantial gain in achievement on this item to student experiences with motion detectors in the model eliciting activities and to the emphasis on descriptions of motion in the model exploration activities in the model development sequence.