The results of this study showed a statistically significant gain in firstsemester mathematics course grades for students who participated in the modelingbased mathematics course compared with a previous cohort of students who had taken a traditional summer course in college algebra or precalculus. The modelingbased course was effective in closing the previous letter grade gap in the first mathematics course between participants and nonparticipants in the bridge program, when controlling for the effect of SAT mathematics score. In each year of the modelingbased course, we found statistically significant gains in student understanding of average rate of change, as measured by the Rate of Change Concept Inventory. By examining the inventory items on which the students showed the greatest improvement, we gain some insight into the ways in which the design and implementation of the modelingbased course may have influenced students' firstsemester mathematics performance. We report these results in the following sections.
First Mathematics Course Grade
Participants across cohorts comparison The bridge program students who participated in the modelingbased course (the 2010–2012 cohort) performed significantly better in their firstsemester mathematics course compared with the 2007–2009 cohort of students who had taken traditional college algebra and precalculus courses, as shown in Table 5.
Table 5. FirstSemester Mathematics Grades for Bridge Program Participants  2007–2009 cohort  2010–2012 cohort    

 n  M (SD)  n  M (SD)  t  df  d 


Precalculus  38  1.75 (1.27)  14  3.17 (0.69)  5.11ba  42.5  1.39 
Calculus I  41  1.28 (1.20)  51  2.20 (1.15)  3.74b  90  0.78 
However, as shown in Table 6, the 2010–2012 cohort had a statistically significant higher SAT mathematics score than the 2007–2009 cohort. Hence we conducted an analysis of covariance (Table 7), with SAT mathematics score as the covariate, which showed that participation in the modelingbased course was a significant factor, but that SAT mathematics score was not a significant factor. Cohen's effect size values, d = 1.39 and d = 0.78, for precalculus and Calculus I, respectively, suggest a high practical significance of these results and that the modelingbased course may be even more effective for precalculus students than for calculus students.
Table 6. SAT Mathematics Scores for Bridge Program Participants  2007–2009 cohort  2010–2012 cohort   

 n  M (SD)  n  M (SD)  t  df 


Precalculus  37  500.8 (53.9)  13  534.6 (33.8)  2.62ba  33.9 
Calculus I  37  562.7 (70.5)  48  588.8 (47.8)  2.03b  83 
Table 7. Effect of Bridge Program Participation by Cohort with SAT Mathematics Score as a CovariateFactor  df  MS  F  Partial eta squared 


Precalculusa 
Cohort  1, 50  16.97  12.17b  0.21 
SAT math  1, 50  0.03  0.027  0.00 
Calculus 
Cohort  1, 85  19.00  14.71b  0.15 
SAT math  1, 85  0.26  0.20  0.00 
Participants versus nonparticipants comparison We examined the effect that the modelingbased course had on the letter grade gap between participants and nonparticipants in the bridge program. The fall course grades for these two groups for the 2010–12 cohort are shown in Table 8. However, as noted earlier in Table 2, the SAT mathematics scores for the participants versus the nonparticipants are significantly different for this cohort. Hence, we performed an analysis of covariance for each course to account for the SAT mathematics scores.
Table 8. FirstSemester Mathematics Grades by Bridge Program Participation for the 2010–2012 Cohort  Nonparticipants  Participants   

 n  M (SD)  n  M (SD)  t  df 


Precalculus  131  2.90 (1.11)  14  3.17 (0.69)  0.86  143 
Calculus I  641  2.55 (1.13)  51  2.20 (1.15)  −2.13a  690 
For the precalculus course, the analysis showed that the effect of SAT mathematics scores was not significant. Our results showed that the modelingbased course was effective in closing the previous letter grade gap (shown in Table 3 for the 2007–2009 cohort), even though there was still a significant difference in SAT mathematics scores between the participants and the nonparticipants in this cohort. Fisher's exact test (used due to small sample size) showed that there was no longer a significant difference in the grade distributions between the participants and the nonparticipants (p = 0.923).
For the Calculus I course, the analysis of covariance (Table 9) showed that there was no significant difference in the fall Calculus I grade by participation in the modelingbase course, but rather the difference in course grades was explained by the differences in SAT mathematics scores between the two groups (p < 0.001).
Table 9. Effect of Bridge Program Participation on Calculus I Grades with SAT Mathematics Score as a CovariateFactor  df  MS  F  Partial eta squared 


Participation  1, 618  0.00  0.00  0.00 
SAT math  1, 618  54.11  45.41a  0.07 
The mean Calculus I course grades, when adjusted for the SAT mathematics score, were nearly identical. The adjusted mean course grade for the nonparticipants was 2.51 (n = 571, SE = 0.05) and for the participants was 2.51 (n = 48, SE = 0.16). This result means that the modelingbased course enabled the bridge program participants to perform on a par with the nonparticipants who had comparable SAT mathematics scores; thus the course closed the previous letter grade gap between the participants who in the earlier cohort (2007–2009) had performed below the nonparticipants (as shown in Table 3). A chisquare test showed that there was no longer a significant difference in the grade distributions between the participants and the nonparticipants, χ^{2} (4, 692) = 5.89, p = 0.21.
Understanding of Average Rate of Change
As noted above, we systematically changed the items on the inventory: we added three items from the first to the second year of the course implementation and modified items from the second year to the third year of the implementation. The pre and posttest results shown in Table 10 indicate that there was a significant improvement (p < 0.001) in the student understanding of the concept of average rate of change for each year of the modelingbased course, with large effect sizes in all three years.
Table 10. Pre and Posttest Results for the Rate of Change Concept Inventory  Pretest  Posttest    

Year  M (SD)  M (SD)  t  df  d 


2010  8.86 (3.47)  12.69 (2.81)  10.65a  33  1.21 
2011  8.05 (3.10)  12.34 (3.16)  7.92a  16  1.37 
2012  6.92 (3.25)  12.62 (3.30)  12.37a  34  1.74 
As shown in Table 11, there was a significant improvement on the nine common items on the Rate of Change Concept Inventory for the 2010–2012 cohort in the three subscore areas of algebraic expressions, graphical interpretation, and symbolic interpretation, with a maximum score of three points in each area. Because these subscores were not normally distributed, we compared the pre and posttest results using relatedsamples Wilcoxon Signed Rank Test and found statistically significant improvements on all three subscores when aggregating the data across all three years of the modelingbased course.
Table 11. Pre and Posttest Results for the Subscores of the Common Items on the Rate of Change Concept Inventory for 2010–2012 Cohort  Pretest  Posttest  

Subscore  M (SD)  M (SD)  z 


Algebraic  1.14 (.99)  2.27 (1.08)  6.503a 
Symbolic  0.79 (.82)  1.53 (.97)  5.261a 
Graphical  1.33 (.87)  1.98 (.80)  5.850a 
In the third year of the modelingbased course, the overall scores improved from 6.92 (35%) to 12.62 (63%), as shown in Table 10. There were nine items on the concept inventory for which the improvement was greater than 30%. Four of these were algebraic expression items, four were graphical interpretation items, and one was a symbolic interpretation item. We report the analysis of student gains on these items in relationship to the model development sequence.
Table 12. Algebraic Expression Items with High Percentage Gain  Correct on pretest  Correct on posttest  Gain 

Item  n  %  n  %  Δn  Δ% 

Q2  17  49  30  86  13  37 
Q12  14  40  32  91  18  51 
Q16  4  11  28  80  24  69 
Q19  8  23  19  54  11  31 
Item Q19 asked students to find an average speed over a round trip between two cities when given speed (45 miles per hour) and time (two hours) information for the first part of the trip and only speed (30 miles per hour) information for the return trip. The significant improvement on this item was largely accounted for by the number of students who shifted from incorrectly seeing the average speed as a simple average of the two speeds (37.5 miles per hour) to correctly reasoning about this as a weighted average that has to account for the amount of time traveled at each speed, resulting in an average speed of 32.5 miles per hour.
The gains on items Q2 and Q16 (simplifying algebraic expressions for a quadratic function and for a complex fraction) likely reflect the emphasis in the course on developing students' algebraic skills. The gain on item Q12 (finding an average rate of change between two points on a parabola) is likely due to the limits of students' prior experiences with the meaning of the term average rate of change in a mathematical context. This was an openended question; without knowing the mathematical meaning of the term, many students simply left this question blank on the pretest. We attribute the gain on item Q19 (interpreting two speeds over different time intervals) in part to the emphasis in the model development sequence on analyzing and interpreting velocity in the context of motion along a straight path.
Symbolic interpretation items The symbolic interpretation items required the students to create appropriate symbolic expressions when given a problem context or to interpret the meaning of the parameters in symbolic expression. There was a substantial gain on one of these three items, which is taken from Carlson et al. (2010). The students had to choose correct interpretations of the parameter m in a linear growth function:
A baseball card increases in value according to the function, , where b gives the value of the card in dollars and t is the time (in years) since the card was purchased. Which of the following describe what conveys about the situation?
There was a substantial gain (37%) on this question from the pretest (n = 7, 20%) to the posttest (n = 20, 57%). This gain likely reflects the emphasis in the model development sequence on making meaningful interpretations of data and on giving descriptions of the average rate of change in various contexts.
Graphical interpretation items There were substantial gains on four items that measured student proficiency at reasoning about and interpreting rates of change when given graphical information, as shown in Table 13. We take these forms of graphical reasoning to be central to the work and the learning of engineers and scientists.
Table 13. Graphical Interpretation Items with High Percentage Gain  Correct on pretest  Correct on posttest  Gain 

Item  n  %  n  %  Δn  Δ% 

Q4 A  9  26  24  68  15  42 
Q4 B  14  40  27  77  13  37 
Q5  12  34  28  80  16  46 
Q8  10  29  25  71  15  43 
Q17  6  17  19  54  13  37 
Two of these items (Q4 and Q8) addressed the interrelated interpretation of velocity and position graphs. Question Q4 addressed interpreting information about velocity when students were given a position graph. Question Q8, on the other hand, involved interpreting position information when students were given a velocity graph. Students often confuse the interpretation of these graphs; they experience difficulty in inferring velocity information from a position graph and misread a velocity graph as if it were a position graph (Beichner, 1994). This coordination of representational systems (shifting between the velocity or rate graph and its associated position graph) was the main focus of the model exploration tasks described earlier.
Item Q4A asked for the value of the velocity of the person whose position graph is shown in Figure 2 at seven seconds, and item Q4B asked for the person's average speed over the entire time interval. On the posttest, 24 (68%) of the students correctly reasoned about the velocity at a particular point in time when the velocity is constant and the starting point for the motion is not at the origin. There were two common errors on the pretest. The most common error was to incorrectly assume that the speed would equal position divided by time, or 35 meters divided by seven seconds, which produced an incorrect answer of 5 meters per second. The second most common error on the pretest was to read the position graph as if it were a velocity graph, which produced an incorrect answer of 35 meters per second. On the posttest, only five students made this first error and none made the second error. The prevalence and persistence of both errors are well established in both the mathematics education and physics education literatures; these results compare favorably with results on an equivalent item by Beichner (1994), where only 21% of students were able to reason correctly about the velocity after collegelevel instruction in kinematics. On the posttest, 27 (77%) of the students were able to reason correctly on item Q4B about the average of two different constant velocities over two different intervals of time. We note that this item is the graphical equivalent of algebraic item Q19, on which there was also a substantial improvement from the pre to posttest. The student achievement on the posttest was greater on the graphical form of the item (with 77% correct) than on the algebraic form of the item (with 54% correct).
Item Q8 represents an important reversal of the above problem and one that is a wellknown source of difficulty for physics and calculus students (Beichner, 1994; Monk, 1992). The item asks students to identify which of a set of position versus time graphs would best represent the object's motion during the same time interval for the motion shown in the velocity graph in Figure 3. A successful answer to this item requires an understanding of how to reason about position when given a velocity graph.
On the posttest, we found that 25 (71%) students were able to correctly identify an appropriate position graph. On an analogous item reported by Beichner (1994), only 29% of students were able to correctly use areas to reason about velocity from an acceleration graph after instruction in kinematics. The substantial 43% gain on this question suggests that the model exploration tasks within the model development sequence helped the students understand how to use areas to reason graphically about nonconstant and negative velocities (or rates of change), such as shown in Figure 3.
The two other graphical items (Q5 and Q17) for which there were substantial gains addressed student proficiency at choosing a text description of motion when given a position graph (Q5) and in comparing the average rates of change by reasoning about the outputs of three functions over differing input intervals (Q17). In question Q5, the students were asked to choose a description of the motion of an object whose position is shown in Figure 4.
The posttest results showed that 80% of the students were able to select a correct description; results reported by Beichner (1994) on this item showed that only 37% of students were able to describe the motion correctly after kinematics instruction. We attribute the substantial gain in achievement on this item to student experiences with motion detectors in the model eliciting activities and to the emphasis on descriptions of motion in the model exploration activities in the model development sequence.