Journal of Computer Assisted Learning

2.1 Computer‐aided problem solving

Over the last years, there has been greater research on the influence of computer‐assisted environments on mathematics instruction (see Harskamp & Suhre, 2006; Huang & Ke, 2009; Schoppek & Tulis, 2010; Li & Ma, 2010; Gunbas, 2015). There is empirical evidence that computer‐assisted learning environments play a role in students' learning enhancement and motivation (Lopez‐Morteo & Lopez, 2007). For example, Panaoura (2012) investigated the improvement of students' mathematical performance with a mathematical model via a computerized approach. He showed that students bolster their problem‐solving abilities. Xin et al. (2016) explored the potential effects of the Please Go Bring Me Conceptual Model‐Based Problem Solving (PGBM‐COMPS) intelligent tutor system compared with traditional teacher‐delivered intervention when applied to multiplicative problem solving. According to the findings, the PGBM‐COMPS intelligent tutor seemed to yield better outcomes in enhancing participating students' multiplicative problem solving. However, the number of participants involved in this study was limited. Conducting future studies with larger sample sizes may perhaps enhance the validity of the PGBM‐COMPS program. Comparison of the effectiveness between computer‐ and teacher‐mediated instructions on the word problem‐solving performance of students has also been studied (Leh & Jitendra, 2013). However, there was no statistical significance in between‐condition differences at posttest and with a 4‐week retention test of word problem solving. One of the most popular software programs in word problem solving is “GO Solve Word Problems” (Tom Snyder Productions, 2005). This tool introduces students to the most common types of arithmetical situations reflected in word problems. It makes use of graphic organizers to help students construct concrete mental models of the situations and relationships through the supplied information in each problem. This program has the advantage of providing a different diagram for each problem situation. However, it does not focus on problem‐solving episodes, making it difficult to diagnose at which stage students make misconceptions.

2.2 Polya's problem‐solving strategy

Polya's (1945) four‐step process has provided a model for the teaching and assessing problem solving in mathematics classrooms: understanding the problem, devising a plan, carrying out the plan, and looking back. In order to help students better cope with difficulties encountered in solving problems, many researchers have developed computer‐assisted mathematical problem‐solving tools based on Polya's problem‐solving strategy. For example, Ma and Wu (2000) designed a set of interesting active learning materials for teaching. The system was designed according to Polya's problem‐solving strategy. Research outcomes indicated both students' learning interest and achievement had improved. Huang et al. (2012) formulated a computer‐assisted mathematical problem‐solving system to help low‐achieving second and third graders in mathematics with word‐based questions. They found that the mathematical problem‐solving abilities of experiment group students were significantly superior to that of controls. Combination of Polya's strategy with schema representation and solution trees has also been studied (Chang et al., 2006). In this study, the authors developed a computer‐assisted problem‐solving system and tested it with elementary school mathematical problems that involve the operations of addition, subtraction, multiplication, and division. The system was empirically demonstrated to be effective in improving the performance of students with lesser problem‐solving capabilities. Recently, Yang, Chang, Cheng, and Chan (2016) examined how to foster pupils' mathematical communication abilities by using tablet PCs. A reciprocal peer‐tutoring‐enhanced mathematical communication system was designed for supporting students' math creations (including mathematical representation, solution, and solution explanation of word problems) and reciprocal peer‐tutoring activities. The system activity flow involved four steps: understanding the problem, drawing a representation, writing a solution, and explaining the solution. These steps were designed according to Polya's findings (1973) about problem solving. The results showed that students' mathematical representations and solution explanations became more accurate after the learning activity.

Taking the cited examples together evinces the notion that computer‐assisted mathematics problem‐solving systems have a positive impact on children's problem‐solving abilities. Yet it is worth underscoring certain issues warranting investigation. Herein, we first explore whether students fail to solve problems because they do not understand the specific relations among the problem quantities, or in choosing a correct solution strategy, or in carrying out the solution strategy correctly. We also explore whether providing students with computer‐based assistance at each stage of the problem‐solving process helps students overcome their difficulties. Therefore, we focus predominantly on supporting students at the various stages to allow them to solve one‐step mathematical word problems on the basis of (a) understanding of the problem, (b) planning of the solution, (c) execution of the plan, and (d) reviewing of the solution.

3.1 Students' problem‐solving process

Students' problem‐solving guidance process is portrayed in Figure 1. First, the adequate problem is provided by the system among other problems stored in the problem‐solving information database. At Stage 1, the assessing module for understanding the problem proposes responses to be selected by the student and other techniques for comprehension of the problem. The plan‐making stage enables the student to build a schema representation of their solution. Stage 3 offers, according to the operation type, a calculation interface followed by a multiquestionnaire for the student to validate their solution. Problem information is provided at each stage of the problem‐solving process, and assessment of each stage is recorded in the student‐tracking database. The system displays feedback messages subsequent to problem‐solving completion.

3.1.1 Understanding the problem stage

During this stage, the system offers students the possibility to circle important words in the problem. Students also have to distinguish between what is known and what is requested in the problem by selecting appropriate responses. For illustration, Figure 2 displays the problem in the problem frame and check boxes for the needed answers. As the teaching language is in Arabic, the text must be read from right to left, thus adapting the learning system to the student‐specific cultural context.

3.1.2 Making a plan stage

The guiding process of plan elaboration is divided into four steps as illustrated in Figure 3 so as to help students express graphically problem solving. The first step consists of identifying the first operand and its value among a list of operands; the student selects the appropriate one and enters its value. The second step proposes an operator choice of either addition or subtraction. At the third step, the student selects the second operand and its value. Finally, during the fourth step, all that is required is a result label. According to the problem missing part, a result label may either be requested in the first or third steps.

In the example of Figure 3, the student has opted for the second label for the first operand and enters a value of 13, whereas for the second operand, they also chose a second label with a value of 8 and attempt subtraction between these two operands. Labels are presented so as to gauge if the student understands the meaning of the operator.

After the student has pressed the “next” button, the system compares the solution plan created by the student with that built into the system, and the suggestions regarding the student's problem solving are stored in the student tracking database and displayed after the student completes the problem.

3.1.3 Executing the plan stage

At this stage, the system evaluates the student's procedural knowledge. As shown in Figure 4, the system provides a graphical preview of all the addition and subtraction worksheets in a vertical problem format. Large boxes are reserved for digits of operands, and little boxes are used if regrouping is required for subtraction (exchanging one of the tens for 10 units or one of the hundreds for 10 tens) or for addition when the process involves a carry‐over number. Three steps are required at most to complete this stage, each aimed at manipulating ones, tens, and hundreds. The student has to fill in the large cases (ones, tens, and hundreds) of results and little cases if regrouping is needed for subtraction or if addition necessitates carrying over. After finishing the calculations of all the columns, the system assesses the student's answer, and feedback is stored. A last stage is needed to validate the student's problem solution.

3.1.4 Reviewing the solution stage

During this stage, the student answers questions as depicted in Figure 5. In order to validate the solution from the previous stage, the system proposes questions that are related to the problem, and the student must answer with true or false. After completing this stage, the student presses the evaluation button that triggers the system to evaluate the results, and messages appear to indicate whether any mistakes were made. Additionally, the correct problem‐solving steps are displayed simultaneously next to the student's answers.

One of the most common strategies for studying the problem‐solving process involves the use of an analytic scoring scale. Analytic scoring is an evaluation method that assigns point values to various dimensions of the problem‐solving episode (Charles, Lester, & O'Daffer, 1987).

The grading rubric assesses the solutions in four specific areas. According to experts, 10 maximum possible points are awarded to each problem by assigning two maximum possible points for understanding the problem, three maximum possible points for making a plan stage, three maximum possible points for procedural skills, and finally, two maximum possible points for reviewing the solution. Details of the stages' scoring are available in Table 1.

Table 1. Assessment grading rubric

Stage	Understanding the problem	Making a plan	Executing the plan	Reviewing the solution
Criteria	Complete understanding of the problem with recognition of what is given and what is requested in the problem with correct identification of important words.	Accuracy of setting the plan and degree of describing and interpreting the operands and their values, the operation, and the missing value.	Accuracy of computation with correct manipulation of numbers (adding/subtracting of units, tens, and hundreds) and the right use of regrouping and carrying over concepts.	Correct validation of problem solution by selecting the correct problem–solution variants. Variants are obtained by combining addition and subtraction of problem operands and missing parts.
Rubrics and scores	•What is known: 0.7 pts. •What is requested: 0.7 pts. •Identifying important words: 0.6 pts.	•First operand label identification: 0.5 pts. •Second operand label identification: 0.5 pts. •Identification of missing label: 0.5 pts. •Correct number sentence: 1.5 pts.	•Units manipulation: 1 pt. •Tens manipulation: 1 pt. •Hundreds manipulation: 1 pt. In case of manipulating numbers without hundreds, the rating score of this category is affected equally in terms of units and tens manipulation.	•First problem–solution variant: 0.5 pts. •Second problem–solution variant: 0.5 pts. •Third problem–solution variant: 0.5 pts. •Fourth problem–solution variant: 0.5 pts.

4.1 Participants and experimental design

The participants of the study were 60 Grade 2 students attending four classes in two elementary schools in Tiaret Province in the west Algeria. Students were selected on the basis of scores from the average mathematics course score of the first and second trimesters. All Grade 2 students scoring at or below the 50th percentile were chosen to participate in the study, but the sample was reduced to 52 students on the basis of the provision of informed parental consent. We selected the 50th percentile as our cut‐off score to include children that had basic learning abilities and needed remedial instruction.

This study consisted of a precontrol–postcontrol design with a 2 × 2 mixed‐factorial design where the between‐subject variable was the treatment and the within‐subject variable was tested. Participants were divided into two groups. One group was the focus of the experiment (i.e., experimental group where 26 randomly assigned students used the developed system to solve word problems), and the other group was the baseline (i.e., control group where the other 26 randomly assigned students used a general strategy to solve word problems without the developed system). The levels of the between‐ and within‐subjects variables are found in Table 2.

4.2 Materials

This study concentrated on students' solving of one‐step mathematical word problems. We utilized several one‐step word problems derived from various Grade 2 mathematics textbooks to teach word problem solving using the four steps of Polya's strategy instruction. In addition, situations involving the four problem types that did not include unknown information were developed for use during the training (treatment) phase. The related problems were collected, revised, and compiled into a database of 96 problems (25 of put‐together, 21 of change‐get‐more, 22 of change‐get‐less, and 28 of compare) for practising with the developed system. Each problem type involved three quantities, and the position of the unknown in these problems may be any one of the three quantities.

4.3 Procedure

We organized 10 meetings of the students with our learning environment so they became familiar with and practised the word problem‐solving program based on Polya's four‐stage model. Training sessions where spread out over 5 weeks (frequency being twice a week for 90 min each). Before the first session began, we proceeded to pretest both groups (i.e., experimental and control), with the test taking 60 min, and the collected answers were graded according to the four categories of problems (see Section 3.1). After carrying out the pretesting, the experimental sessions began. The experimental group students received instruction for solving one‐step problems that involved the different four stages of the problem‐solving model using our developed learning environment. During each session, experimental group students solved eight problems (two of each type), and the control group students solved the same word problems on a paper worksheet. For this group, the problems were scored to allow for credit for the correct number sentence, computation, and labels in answers. Finally, we organized 1 hr posttesting for both groups.

5.1 Analysis of group differences

We used Box's (1949) M test of homogeneity of covariance matrices to test the multivariate homogeneity of variance–covariance matrix assumptions. An insignificant value from Box's M test indicates that those groups (i.e., experimental and control) do not differ from each other and would support the assumption. The Box's M value of 1.459 was associated with a p‐value of 0.707, which was interpreted as nonsignificant. Therefore, the variances were homogeneous.

5.2 Analysis of students' performance

A two‐way mixed analysis of variance with repeated measure for one factor was conducted to determine whether there was statistical significance between students using the computer‐assisted mathematical problem‐solving system and students making use of general instruction. The independent variable included a between‐subjects variable, the group (experimental and control), and within‐subjects variable, the test, with repeated measures of pretesting and posttesting. The dependent variable was the students' scores mathematical problem‐solving pretests and posttests. An α level of 0.05 was utilized for this analysis. Results for model assumptions of normality, homogeneity of covariance, and linearity were deemed satisfactory.

The results from the main effect of pretesting and posttesting was significant as depicted in Table 3, F(1, 50) = 177.96, p < 0.001. A large effect size was evident, indicating that scores during the pretest may have been different from those of the posttest for the two groups (ƞ_p² = 0.781 > 0.14). There was also a significant main effect in the group factor as presented in Table 4, F(1, 50) = 5.39, p < 0.05, which was indicative of a moderate to large effect size, uncovering differences in the scores between the two groups. In addition, there was a significant interaction in the scores gain between the experimental and control groups, F(1, 50) = 47.04, p < 0.001, providing evidence that there was a change from the pretest to the posttest that was greater in one group than the other. The interaction was interpreted with simple main effects analysis, focusing on the effects of time within each treatment group.

The simple main effect analysis revealed a statistically significant difference between the two groups during posttesting (F(1, 50) = 14.88, p < 0.05) but not for the pretesting (F(1, 50) = 0.012, p = 0.915). We could consequently deduce that the capacity of problem solving in the experimental group (M = 7.38, SD = 0.32) was significantly greater than that in the control group (M = 5.59, SD = 0.32). With this, the simple main effect comparison between the pretests and posttests showed there to be a significant difference in the experimental group, but not one that was considerable in the control group, indicating that the training had resulted in significant progress only in the experimental group. In reviewing Table 5, it can be seen that the scores for participants in the experimental group increased from the pretest (M = 4.80, SE = 0.25) to the posttest (M = 7.38, SE = 0.33), F(1, 50) = 203.99, p < 0.005, whereas the scores for participants in the control group exhibited a slight change from the pretest (M = 4.76, SE = 0.25) to the posttest (M = 5.59, SE = 0.33), F(1, 50) = 21.00, p < 0.005.

5.3 Analysis of stages at which students encounter difficulties in the problem‐solving model

By exploring log files for the problem‐solving stages for all students recorded by the system in the student tracking database, the results summarized in Figures 6 and 7, we learned the following:

First, the results of the study for the experimental group were analysed by averaging the scores of each stage of the solved problems for all students and then comparing the average scores over three points of time (pretest–training–posttest). In general, the graph in Figure 6 shows there to be a net improvement in scores during the first three stages over time, but the scores from the last stage (i.e., reviewing the solution) remained nearly constant, probably because the students were not able to relate to the different forms of a unique problem. A deep investigation is merited to determine the real reasons for this failure. Detailed review of our study findings shows that for most of the problems, a high score in “understanding the problem” means high scores in both “making a plan” process and “executing the plan” skills. Although the overall correlation is robust, certain students did not adhere to this trend.

In addition, as claimed in Section 1, our study investigated the stages of the problem‐solving model that are problematic for most students. The goal of this analysis consisted of identifying the stages at which low achievers encounter difficulties in order to remedy these weaknesses. A visual examination of the word problem‐solving progress data depicted in Figure 7 uncovered that students realized about 70% of the first stage's reference score, 93.33% of the second stage's reference score, 76% of the third stage's reference score, and only roughly 44% of the last stage's reference score. However, the score gain between pretesting and posttesting was the highest for understanding the problem stage (35%) and was 33%, 26.6%, and 4%, respectively, for making a plan, executing the plan, and reviewing the solution stages.

5.4 Analysis of students' opinions regarding using the computer‐assisted system and the resolution model

After the posttest was completed, students were requested to answer a questionnaire surrounding their opinions on the computer‐assisted system, the staged problem–resolution model, and difficulties encountered during resolution. We also desired to know more about reviewing the solution stage failures. The organizing team assisted students in understanding and answering all the questionnaire questions. Table 6 features the collected information.

Concerning the system use viewpoint, Table 6 shows that the proportion of students who thought they acquired more skills in mathematical problem solving was high (92.3%), suggesting that practising problem solving through the computer was quite valuable. On the other hand, regarding the provided assistance to help students with their problem solving, namely, the use of multiple choice questions, the schema representation strategy, and worksheet use, the majority of students (88.4%, 100%, and 100%, respectively) thought that the provided techniques aided their performance during all stages. When asked which step was the most simple, 46.1% of the students responded that it was making of the plan stage, whereas 30.7% and 23%, respectively, considered executing the plan stage and understanding the problem stage the simplest. However, no students thought reviewing the solution stage was the simplest (0%). According to results, the outcomes indicated that students had encountered problems when reviewing their solutions and had failed to correspond the problem with its variants (the same problem with a different missing part). Therefore, we note that 96.1% of students believed that reviewing the solution stage was less important than other resolution stages.

6.1 Influence on students' problem‐solving ability

The results of this study support the findings of existing studies proposing that employing Polya's methods can significantly improve the overall performance of students (Craig, 2016; Özsoy & Ataman, 2017; Romiszowski, 2016). The statistical results show that the students' problem‐solving performance as well as their problem‐solving ability was improved via the proposed learning approach. It seems that the program with the use of the model created a powerful learning environment in which students overcame their difficulties. Scores for participants in the experimental group increased from the pretest (M = 4.80, SE = 0.25) to the posttest (M = 7.38, SE = 0.33). These findings converge with prior findings that providing of assistance at each episode of the solving process is very helpful to enhance students' problem‐solving ability (Chang et al., 2006; Huang et al., 2012; Lee, 2017).

There have been several studies that were conducted to investigate the effectiveness of visualization support on problem‐solving achievement (Jitendra, 2007; Jitendra et al., 2007; Reinhard, Hesse, Hron, & Picard, 1997; Reusser, 1996). This study continued the convention of using such assistance but was also marked by a few major differences from previous studies. The first was with respect to the use of multiple‐choice questions at understanding the problem stage, where student had to identify what was known and what was requested in the problem. The second was employing of schema representation strategy during making a plan stage to enable students to describe the solution steps in detail by ordering operands of the problem. The third difference was the vertical graphical presentation of addition and subtraction worksheets so as to facilitate students performing regrouping if subtraction was needed to solve the problem or carrying over if addition was required.

6.2 Diagnosis of difficult steps of the problem‐solving model

On the basis of the results of this study, students achieved a successful outcome during the first three stages, but they failed most of the time at the last stage (students realized about 70% of the first stage's reference score, 93.33% of the second stage's reference score, 76% of the third stage's reference score, and only roughly 44% of the last stage's reference score). This finding indicates that students have high levels of performance during making the plan stage and a strong knowledge in understanding the problem and executing the plan stages. In contrast, students encountered problems when reviewing their solutions; this signifying that students did not recognize the variants of the problem or perhaps they felt they accomplished objective of their work by the end of their calculations. These results are in agreement with prior findings from the literature (Huang et al., 2012), where only 5.9% of students considered the double‐checking step the easiest. In addition, Yang et al. (2016) stated that students became more capable of using mathematical representations and make equations (making a plan stage and executing the plan stage), but they still need more practice in using mathematical language for explaining their solutions (reviewing their solutions). Nevertheless, Chang et al. (2006) showed that only 16.7% of students highlighted important information (understanding the problem stage) and only 42% of the students constructed complete solution trees (making a plan stage). It may seem confused; however, authors claimed that the training had resulted in significant progress only in the experimental group.

This means that although the computer‐assisted system improved the problem‐solving performance of the experimental group, further research is necessary in order to find ways to better design reviewing the solution stage. One possible answer is to teach students to recognize variants of a given problem or to translate a problem into requested variant‐type problems (put‐together, change‐get‐more, change‐get‐less, and compare).

6.3 Influence on students' opinions about using the resolution model

The majority of students thought that the resolution model helped them acquire more skills in mathematical problem solving, suggesting that practising problem solving through the computer was quite valuable. This result was consistent with the findings of other researchers (Huang et al., 2012; Yang et al., 2016). However, no students thought reviewing the solution stage was the simplest.