Journal of Computer Assisted Learning

1 INTRODUCTION

The STEM (science, technology, engineering and math) skills are essential elements for the implementation of the European Agenda for Growth and Jobs, a part of the European Union's overall strategy promoting smart, sustainable, and inclusive growth (European Commission, 2010). However, even if investment in STEM areas is seen as a way to intensify innovation, STEM learning in schools appears to be fragile (J. Osborne & Dillon, 2008). Furthermore, in Italy and other European countries, the choice to invest in STEM courses seems to be inconvenient, as the high costs may outweigh the benefits (Lehouelleur, Beblavý, & Maselli, 2015). This seems especially true for women, for whom entering STEM courses is less financially beneficial, yielding low returns, which partly explains the low female participation in STEM (Lehouelleur et al., 2015).

In the recent years, robots have been used to teach STEM concepts (Altin & Pedaste, 2013; Barker, Nugent, & Grandgenett, 2014). Educational robotics (ER) differs from the other types of educational technologies as it provides a “technological fluency or literacy,” meaning to be aware of mastering knowledge and abilities (Papert, 1987) rather than simply “technical competence,” or specialized knowledge (Alimisis, 2013). Indeed, robotics offers the possibility of transforming STEM concepts into real problems to solve, applying constructionism principles. Papert's (1980) constructionism theory, based on constructivism principles, underscores the importance of the active manipulation of objects (“cognitive artefact”) in the learning process and in children's creation of mental representations of the world. Through active experimentation and the ability to receive online feedback, robotics can enhance students' learning process and self‐efficacy (McGill, 2012; R. B. Osborne, Thomas, & Forbes, 2010), offering a fun and playful way of learning. According to Apiola, Lattu, and Pasanen (2010), “robots seemed to work as a powerful trigger of the initial curiosity and motivation of students.” In particular, robotics can influence attitudes towards STEM, nurturing interest and engagement in math and science careers (Nugent, Barker, Grandgenett, & Adamchuk, 2010).

In the literature, robotics activities have been found to create interesting and positive results in improving STEM knowledge. One study (Hussain, Lindh, & Shukur, 2006) tested the influence of 1 year of “LEGO” training, based on constructivist theory, on pupils' school performance. Children worked in smaller working groups and did not follow a specific programme for working with the LEGO material, adapting their work to the usual school activities. The sample was composed of boys and girls (12–13 and 15–16 years old): 322 in the experimental group and 374 in the control group. The training was held for 2 hr a week over 12 months. The results were analysed by means of quantitative and qualitative methods and showed improvement in mathematics, but only for the youngest trained group. For problem solving, there was no significant improvement. An interesting finding was that children with higher mathematics ability tended to be more engaged in the activity. A similar study using a different statistical approach is reported in Lindh and Holgersson (2007). A pilot study (Barker & Ansorge, 2007) examined the efficacy of an informal experiential science intervention based on robotics in an afterschool programme on students' learning in science, engineering, and technology utilizing the LEGO Mindstorms robots. A total of 32 children ages 9 to 11 years were involved, divided into a control group and an experimental group. The experimental group took part in the robotics programme twice a week for 1 hr over 6 weeks. Results were measured by means of a 24‐item paper‐and‐pencil test created by the researchers. The increase of mean scores from the pretest to the posttest for the experimental group indicated the learning effects of ER. Another study investigated the effect of robotics education in a group of 16 students between 10 and 15 years old (Karahoca, Karahoca, & Uzunboylub, 2011). Utilizing a collaborative learning approach, students, teachers, and assistants worked together on robotics lab activities, learning principles and methods to design and develop line‐tracking robots. Here, science learning included introducing equipment for designing electronic circuits and teaching fundamental principles for robot making. Observing the conclusion of every step of the robot design and development process, the authors concluded that robotics can impact students' lives by affecting science performance. They also found a positive effect of improving relationships with friends in the classroom. Nugent et al. (2010) led a 40‐hr intensive robotics/GPS/GIS summer camp for children to improve computer programming, mathematics, geospatial technologies, and engineering (“such as gears and sensors”). The researchers wanted to analyse the impact of an intensive week‐long robotics summer camp (full intervention) compared with a control group and a 3‐hr course (short intervention). The impact of the short‐term intervention was also analysed. The treatment group consisted of 147 students (mean age of 12.28 years), and 141 students formed the short‐term intervention/control group. The researchers developed a content learning instrument (a questionnaire of 37 multiple‐choice items) covering mathematics, geospatial concepts, engineering, and computer programming, along with an attitude instrument, consisting of a 33‐item Likert scale. The results showed positive effects in the experimental group compared with the control group. In particular, students participating in the full intervention increased their STEM learning, and self‐efficacy measurement was significantly higher than that for the control group and short‐term intervention. On the other hand, the short‐term intervention had no impact on student learning. Another study used a different psycho‐pedagogical background, the cognitive apprenticeship, as a framework for developing a 2‐week summer robotics camp (3–4 hr of activities with LEGO Mindstorms each day) for 27 middle school children in order to develop engineering design skills articulation, rapid prototyping, design evaluation, as well as science and computational thinking skills (Larkins, Moore, Rubbo, & Covington, 2013). Results were analysed by both a quantitative method (utilizing a STEM Semantic Survey) and a qualitative one (evaluation of engineering notebooks in which students recorded their designs and problem‐solving processes). The work of Magnenat and colleagues reported the outcome of a programming workshop where children were able to acquire some robotics concepts because, according to the authors, they practised these enough. On the contrary, more theoretical and less practised concepts were not acquired, leaving the authors reflecting on the importance of concepts explanation and exercise, also through the creation of better robotics edutainment materials (Magnenat, Riedo, Bonani, & Mondada, 2012).

ER competitions have also been reported to affect participating students. One example is the international RoboCup. Eguchi (2016) reported the effects of such competitions on student learning, particularly related to STEM concepts and creativity.

The cited articles are examples of how robots have been used to strengthen STEM concepts, along with other skills. As can be seen from such examples and from reviews of the topic (Benitti, 2012; Karim, Lemaignan, & Mondada, 2015), even though some studies support the use of ER to increase academic achievement in STEM areas, not all studies reported significant results and not all the results are comparable. In particular, the variables that contributed to the studies cannot be isolated, as many variables are associated with such studies: the pedagogical background (if present) of the research, the duration and frequency of the robotics activities, the presence of other technologies associated with the robots, the robotic kits used and the number of robots for each children, the number and the background of the participating adults, and many more. Moreover, evaluation methods and samples are not always comparable in terms of age, curriculum, and gender distribution.

Above all, the role of gender in ER studies has varied. In particular, the literature has explored whether the potential learning power of ER differs for boys and girls. Some studies indicate a minor engagement of girls in STEM subjects, reflecting in a minority participation in STEM careers (Lehouelleur et al., 2015). For many years, ER has been proposed as a tool to promote girls' participation in STEM activities. The work of Sullivan and Bers (2013) reported an equally successful achievement in building and programming tasks in 53 kindergarten boys and girls. Atmatzidou and Demetriadis (2016) investigated the effects of ER on students' computational thinking in 164 students ages 15 and 18 years. The results suggest that computational thinking skills improved independently from gender, but girls needed a longer time to reach the same learning effects than did boys. An interesting approach can be found in a study applying expectancy‐value theory to robotics education (Wigfield & Eccles, 2000) in an attempt to investigate the role of gender differences in achievements in STEM areas, perception, and how these translate into long‐term career goals (Weinberg, Pettibone, Thomas, Stephen, & Stein, 2007). The results of the investigation showed positive changes after robotics activities in girls' abilities, perception, and career interests. Other studies reported different results. Nugent et al. (2010) showed that both females and males had significant improvement after robotics activities, even if males scored significantly higher than did females on both the precontent and postcontent assessments. A different study (Nourbakhsh, Hamner, Crowley, & Wilkinson, 2004) explored the effect of a robotics course for high school students, exploring gender differences and finding better results in boys' programming performance and level of confidence in their ability to work with technology. However, girls' confidence in technology increased during the course more quickly than did the boys'. Similarly, Milto, Rogers, and Portsmore (2002) found no gender differences in proficiency in robotics activities, although males were found to be more confident in their abilities.

In conclusion, the reported studies, which are not fully comparable due to important differences in their research methods, do not offer a clear answer about gender differences in STEM learning through ER activities.

2 AIM

This study was developed in the RobEd project framework during the Edu.Ro.Co. training course (Castro et al., 2018). The project was conducted in Italy and represents the first attempt to develop an ER curriculum able to explore the ER potential in different areas contemporaneously (cognitive, metacognitive, emotional and motivational, STEM areas). The aim of the RobEd project was to develop an ER protocol that, if its efficacy was demonstrated, could be used in scholastic and extrascholastic settings with students of varying ages, working on different areas of the learning process (cognition, metacognition, motivation and emotion, STEM). The protocol could provide a shareable methodological background for ER studies in order to facilitate comparisons.

In this article, the effects of the ER protocol developed in the RobEd project on STEM learning were analysed. In the STEM framework, the “E” and the “T” are considered to be missing letters (Sullivan & Bers, 2016) because few studies have concentrated on teaching robotics as part of the E and T framework using ER activities, especially for younger children. As illustrated by the studies reported in the literature review, the engineering and technological knowledge assessed during ER activities often does not employ an explicit knowledge of robotics concepts. Such knowledge, which may appear to be an implicit potential learning power of ER, represents a fundamental basis for introducing children to new technologies in awareness rather than as passive users, even in early childhood. According to one review on the topic (Mubin, Stevens, Shahid, Al Mahmud, & Dong, 2013), when ER is used to teach technical education, particularly knowledge of robots, it is often done by introducing computer science and programming and leaving the students to apply their knowledge practically, by making their robots work. An explicit evaluation of robotics knowledge, which differs from a practical one, is not frequently used.

The aims of this study are twofold:

• to test the hypothesis that an ER curriculum can be used to introduce children of different age ranges to robotics, as a way to expand STEM knowledge, and
• to test for gender differences in curriculum efficacy.

3 MATERIALS AND METHODS

3.2 Protocol overview and experimental design

The RobEd protocol was composed of eight activities of 2 hr each, for a total of 16 programme hours, conducted over 8 weeks. During the activities (see Figure 1), attention was paid to empowering four areas of interest: cognitive, metacognitive, emotional and motivational, and STEM areas. Activities involved the construction and/or programming of robots in order to solve given problems or perform challenging tasks. Each activity stressed different skills and aspects of the four areas, with increasing difficulty, and was embedded in a different narrative context in order to keep students' motivation high.

In regard to STEM areas, a group of experts consisting of researchers and professors in the field of robotics defined key aspects that represented, in their opinion, the basis of the discipline and that could be taught during the ER activities in order to efficaciously introduce students to the field. These key aspects included knowledge of the main components of a robot, understanding of how a robot works, and the comprehension of robots' potentialities and limits.

Such aspects were introduced during the activities and adapted to the different age ranges. To this end, the activities were differentiated by level of difficulty and complexity and robots utilized on the basis of the different three age ranges considered (II–III and IV–V classes from primary school, lower secondary school classes). The goal of each activity was similar but adapted to the age of students. As an example, in each of the three age ranges considered, an ER activity was dedicated to sensors as one of the main components of the robots. In order to explain to the youngest children what sensors are and how they work, the Bee‐Bot robot was utilized on a carpet containing obstacles that obstructed the path of the robot. Such obstacles were utilized to encourage children to reflect on why Bee‐Bot was not able to detect their presence (Bee‐Bot has no sensors) while animals or other robots would be able to do so. A Goose Game utilizing Bee‐Bot (the robot was programmed to move on the boxes; in each box, a different sensor from the natural or the artificial world was represented) was utilized to illustrate different typologies of sensors. In contrast, in lower secondary schools, students were asked to construct LEGO Mindstorms keeping in mind the importance of the position of the different types of sensors. They gained practical experience utilizing the sensors, starting from the classical following‐line activity (utilizing a sensor of light) to a more complex one utilizing multiple kinds of sensors.

In addition to goals, the pedagogical approach applied was the same for all age ranges. In particular, the RobEd protocol used an evidence‐based didactic method (Hattie, 2009, 2012) based on the efficacy of constructivism, avoiding a naïve application of the theory (Tobias & Duffy, 2009) by alternating moments guided by the teachers with moments in which children were left free to experiment. The model is largely in line with Merrill's five principles (problem, activation, demonstration, application, and integration; Merrill, 2002), stressing self‐efficacy perception and metacognitive aspects.

Before starting the activities, informed consent forms signed by students' parents were collected.

To answer the two research questions, a quasi‐experimental design was created using a single‐group pre–post design: immediately before the start and at the end of the RobEd protocol, teachers assessed children in order to evaluate their improvement. Some information about single‐group pre–post designs can be found in Marsden and Torgerson (2012).

3.3 Robotic kits

The commercial robotic kits were chosen for the RobEd project (see Figure 2) depending on age group. The Bee‐Bot (TTS), used for primary school II and III classes, is a bee‐shaped robot able to move forward and backwards 15 cm and rotate 90° right or left (http://www.tts‐group.co.uk/bee‐botrechargeable‐floor‐robot/1001794.html¹). Buttons on the back of the robot enable its programming. Bee‐Bot has a child‐friendly design and is easy to use, inexpensive, and capable of different applications, showing a large versatility.

Primary school IV and V classes and lower secondary I classes used the Pro‐Bot (TTS Group), which has a car design and is a more complex version of the Bee‐Bot (http://www.tts‐group.co.uk/pro‐bot‐rechargeable‐floor‐robot/1009825.html²). With this robot, it is possible to change the centimetres and the degree of rotation that the car will travel. The robot is equipped with touch, light, and sound sensors. Additionally, putting a pen in its back allows for programming the robot to draw geometric figures.

Lower secondary school students were given LEGO Mindstorms. This kit contains robotic LEGO components (http://www.lego.com/en‐gb/mindstorms³). It is widely used in education because it is robust and reconfigurable. Several different sensors (contact, colour/light, ultrasound), three actuators, and a control unit are included in the kit.

4 RESULTS

The final sample (see Table 1) was composed of 389 students (178 from primary school II–III classes, 62 from primary school IV–V classes, and 149 from lower secondary school). The distribution of females and males in the classes was equal, with the exception of lower secondary classes where males were predominant (χ² = 6.450; sig = 0.013).

The nonparametric Wilcoxon test (one‐tailed Monte Carlo distribution) was utilized in order to answer the first research question and to test if there was a significant improvement between preevaluation and postevaluation, of children participating in the study, in the Robotics Questionnaire scores after the ER activities (significance level set at p ≤ 0.05). The Wilcoxon signed‐rank test is a nonparametric statistical hypothesis test used to compare two related samples when the population cannot be presumed to be normally distributed (Siegel, 1956). Results showed significant improvement in each group assessed (II–III Prim. Z = −4.923, p = 0.000; IV–V Prim. Z = −5.104, p = 0.000; Lower Sec. Z = −6.826, p = 0.000).

The nonparametric Mann–Whitney test (two‐tailed Monte Carlo distribution) was utilized in order to answer the second research question and to verify if any gender differences in the improvement between preevaluation and postevaluation can be found, comparing the increments in performance of boys and girls participating in the study(significance level set at p ≤ 0.05). The Mann–Whitney test is a nonparametric test used on independent samples to determine whether two nonrelated samples were selected from populations having the same distribution (Fay & Proschan, 2010).

No significant difference existed between males and females in any of the groups assessed (II–III Prim. Z = −1.003, p = 0.307; IV–V Prim. Z = −0.201, p = 0.842; Lower Sec. Z = −1.231, p = 0.219).

5 DISCUSSION

Robotics has recently become popular in educational settings, and many authors are questioning if ER can be considered only as a fashion or something more (Alimisis, 2013; Johnson, 2003). The time and efforts that many authors have spent on ER projects seem to suggest the potential of ER, even if the real educational possibilities of ER have yet to be clarified (Benitti, 2012).

The present article describes the effectiveness of the RobEd project. The RobEd project aimed to evaluate the effects of ER on different learning areas. To fulfil this goal, a protocol for ER activities was created, reflecting best practices in pedagogy, and was adapted for various age ranges. The construction and the validation of an ER protocol adapted for students of different ages (from 7 to 14 years old) may provide a shareable methodological background for ER studies to enable comparison, expanding the scientific knowledge of ER effectiveness.

This study focused on the RobEd protocol potentialities for STEM‐areas learning. A protocol of 16 hr of ER activities was conducted with a sample of 389 students in order to introduce children of different ages to STEM topics. Primarily, the RobEd project aimed to allow children, even the youngest ones, become familiar with robotics as part of the T and E in STEM. During ER activities, a small dose of robotics was taught in order to demonstrate an implicit, but possibly effective and not fully demonstrated, potential of ER.

Our results suggest that ER can be used as a tool to learn robotics: children in all the age ranges considered in this study reported improved scores in the Robotics Questionnaire after the ER activities. These results suggest how ER can be utilized to teach robotics, although the intervention was short (16 hr) and although utilizing simple robots as the Bee‐Bot one, suggesting the importance of the theoretical approach. Such ER potential, even if intrinsic, has been but not yet fully demonstrated in the ER field (Benitti, 2012). Studies often neglect engineering and technological aspects when designing STEM‐based ER activities, especially in early education (Sullivan & Bers, 2016). When such aspects have been taken into consideration, the evidence of the acquired robotic knowledge is showed practically, by making robots work (Mubin et al., 2013). A direct evaluation by means of explicit questions about robotics has rarely been proposed (for an interesting example of such approach, see Magnenat et al., 2012), making this study an innovation in the evaluation methods of ER activities. We decided to utilize such evaluation method, in contrast to the most utilized one (measured by the functioning of the robot built by the student) in order to ensure greater objectiveness in the evaluation and to research an explicit and conscious knowledge of robotics. It reflects the already cited difference between “technological fluency or literacy,” which means to be aware of mastering knowledge and abilities (Papert, 1987) and “technical competence,” intended as specialized knowledge (Alimisis, 2013).

The ER potentialities of introducing children to robotics rely on the possibilities of presenting engineering and technological concepts to children, even at an early age, in a playful and motivating, but effective, context. Such an experience can bring technology into school settings, which traditionally have often been far from technology‐enhanced learning opportunities (Alimisis, 2013) and can influence students' future interests and careers (Weinberg et al., 2007). This is particularly important for girls who engage less often in STEM‐area studies and careers (Clark Blickenstaff, 2005; Sadler, Sonnert, Hazari, & Tai, 2012). This study, in contrast to others (Milto et al., 2002; Nourbakhsh et al., 2004; Nugent et al., 2010), reports no gender differences in robotics knowledge after ER activities. For this reason, ER could be considered a tool able to contribute to girls' involvement in STEM, reducing girls' negative attitudes towards STEM areas that may begin forming in early childhood and that may tend to get reinforced over time (Weinberg et al., 2007).

One limit of this study is the nonrandomized selection of the sample. Motivated teachers interested in ER chose their experimental class, so the teachers could have influenced the results in multiple ways. Nevertheless, we believe that the motivation of teachers is crucial for the effectiveness, in terms of students' learning results, of every didactic project, including ER activities. Moreover, in the future, the presence of an active control sample could be used in order to compare ER with other technological activities able to introduce children to STEM areas. Finally, the Robotics Questionnaire is not a standardized measurement, even if it demonstrated good face validity. Future work will include the collection of data for statistical validation of the test and the creation of a normative sample in order to enhance the utility of the questionnaire for the scientific community.

In conclusion, ER could be integrated into a STEM curriculum in order to foster positive learning improvements. The RobEd project highlights the importance of an appropriate educational and pedagogical context that also contains ER lessons in order to make the activities effective. The robotic kits offer significant learning possibilities that can enhance an appropriate pedagogical background. In our study, the didactic method is taken from evidence‐based data (Hattie, 2009, 2012) and based on the efficacy of constructivism in hopes of avoiding a naïve application of the theory (Tobias & Duffy, 2009) by alternating moments guided by the teachers with unstructured moments in which children were allowed to experiment independently.

ACKNOWLEDGEMENTS

The authors would like to thank the groups of expert from the Biorobotics Institute of the Scuola Superiore Sant'Anna who helped in the development of the Robotics Questionnaire. Moreover, the authors thank Antonio Calvani and Laura Menichetti from the University of Florence for the design of the pedagogical background of the study. Finally, the authors thank all the teachers and students who participated in the Robed project, allowing the development of the current study.

This study was partially funded by the Tuscany Region. The authors confirm that all co‐authors had complete access to data supporting the manuscript. The Scuola Superiore Sant'Anna Ethical Committee approved the study (resolution 24/02/2016).

CONFLICT OF INTEREST

The authors declare that they have no conflicts of interest.