Flipping learning not just content: A 4‐year action research study investigating the appropriate level of flipped learning
Abstract
This action research study follows a between‐subject design strategy and attempts to identify whether a departure from a direct instructional teaching strategy towards a flipped learning pedagogy results in increases in student performance over time. In particular, the study considers the effects of integrating flipped learning pedagogic instruction into a Year 1, second‐semester undergraduate Computer Architecture module. The first year of the study represented a baseline year in which a traditional direct instructional teaching method was used. The three subsequent years of study involved the inclusion of increased proportions of flipped learning instruction. When removing the baseline year from the study and focusing on the years that included a flipped proportion of instruction only, the analysis showed statistically significant increases in learner performance for mature students as the module migrated towards a fully flipped delivery model. Positive increases associated with continuous assessment components of the modules were also observed across the population as the module migrated towards a flipped learning model. However, this apparent increase in learner performance showed no impact on the terminal examination scores across years, indicating that improved performance in continuous assessments was probably due to shallow learning.
Lay Description
What is already known about this topic:
- The 21st century learner is expected to be able to use technology as a tool to research, organize, evaluate, and communicate information effectively and seamlessly within a knowledge based society.
- Resources vary significantly in quality, presenting educational providers a major challenge in guiding their learners to suitable and relevant content for learning.
- Flipped learning has been applied to many specific domains with mixed results.
What this paper adds:
- If flipped learning is being considered, it should be introduced as early as possible.
- Migration to a flipped learning environment has no practical effect on student performance in an early undergraduate technical module.
- Mature students performed better when incorporating flipped learning as a delivery model.
- Flipped learning reduces the necessity for note taking, with negative effects on student performance.
1 INTRODUCTION AND BACKGROUND
This action research study was established to investigate the role digital resources may play in enhancing student experience and performance within early undergraduate technical modules. In particular, this study investigates the effects of a stepped flipped learning delivery migration of a Computer Architecture module over a 4‐year period. The sections below discuss the major challenges associated with technology and education in modern society. This is followed by a presentation of the theoretical framework relied upon in this research. The results are accompanied by a discussion of the findings and concluded with recommendations for further research.
1.1 Understanding the challenges of the 21st century
According to Thomas and Brown (2011, p. 19), the modern culture of learning consists of an unlimited network of information which is available to anyone with an internet connection. This new culture of learning takes a step towards Marx's (Marx & Engels, 1976) ambitious “right to education for all” matching the opportunity of a bourgeois education with proletarian realities (Montefiore, 1901). However, this march is tempered by the fact that access to resources does not ensure an excellent education on its own. On the one hand, the quality of information or educational resources available varies significantly in the information age. Whereas on the other hand, the 21st‐century learner is now expected to use technology as a tool [to research, systematize, evaluate, and communicate information effectively and seamlessly] in addition to knowledge creation.
This multiplicity of demands creates a complicated tapestry of challenges for both instructors and students alike. The skills required in the use of technology to access educational material change almost daily. Owen (2009) observes that, for educators, “the sheer inevitability and momentum of global adoption of all forms of technology has engendered a range of responses” including anger, denial, wholehearted welcome, and exploitation (Schneckenberg, 2004). According to the Gates Foundation, not all faculty members embrace technology in their classrooms (FTI, 2015). U.S.‐based research found that 40% of the professors surveyed, use, or are interested in using innovative techniques and technologies within their teaching. Only half of these professors integrated technology into their teaching, despite recognizing a cultural shift in students engaging with an archaic educational system (Prensky, 2001). Therefore, there is a need for a new digital‐driven pedagogy that leverages the emerging technological tools “as a way to bridge the gap between formal” academic environments and the lifestyle of today's learners (Danciu & Grosseck, 2011).
For the born‐digital students, the divide between time‐bound formal and informal learning spaces has been breached. Cultivating a society of on‐demand learning, the evolving cultural shift to a dynamic learning ecosystems is more often now controlled and owned by the learner. This sometimes leaves instructors playing catch‐up with their students or, in some instances, being removed from the learning process altogether. For example, learners now use Facebook chat instead of a college's learning management system (LMS)/content management system. This shift represents how some learners are emotionally connected to their self‐created learning spaces (Beetham, McGill, & Littlejohn, 2009: p. 24). In addition to this shift in control, there is also evidence that modern learners consume information in ways that conflict with notions of traditional pedagogy. The didactic idea of “I will tell you what I know” is favoured or replaced by “I will find out what I need to know” mindset (McLoughlin & Lee, 2008; Moore, Fowler, Jesiek, Moore, & Watson, 2008).
However, problems still exist for the born‐digital student. Recent studies show that students lack the skills to find and process suitable information (Aesaert, Van Nijlen, Vanderlinde, & van Braak, 2014; Calvani, Fini, Ranieri, & Picci, 2012; Kuiper, Volman, & Terwel, 2005) and often becoming lost in an education or information hyperspace. Some theorists believe the lack of notetaking or notetaking through digital means is detrimental to learning. Peper and Mayer (1978) claim that note taking increases attention and results in greater concentration on the material to be learned (Bligh, 2000, p. 131). Titsworth (2001) believed “note taking is a crucial aspect of academic success,” whereas Williams and Worth (2002) found that “notetaking was the strongest predictor (among several variables) of students' overall performance in a college human development class” (in Anderson & Armbruster, 1986). However, Mueller and Oppenheimer (2014, p. 1) found that “those who took notes on laptops performed worse on conceptual questions than students who took notes longhand.” They found that “whereas taking notes can be beneficial, laptop note takers' tendency to transcribe lectures verbatim rather than processing information and reframing it in their own words is detrimental to learning” (Mueller & Oppenheimer (2014), p. 1).
The vertiginous pace of technology use which accompanies education instruction today is daunting for both the born digital and nonborn digital students. With an increased commitment to lifelong learning, nonborn digital students struggle with the terminology and use of multiple tools used in the classroom. However, according to an EDUCAUSE Center for Analysis and Research (2013) study, age alone does not determine who will struggle as they cite young learners who find learning with technological tools difficult:
I don't like all this digital stuff. I don't like all the problems that come along with computers. I don't really understand most of it, and there's always something new to learn right after you get used to one thing. (Dahlstrom, Walker, & Dziuban, 2013).
Developing skills early in their educational journey to deal with digital information and to be able to source and evaluate suitable content is essential for student success (Anmarkrud, Bråten, & Strømsø, 2014; Ferrari, 2013). Progress is considered to be hampered due to the lack of a coherent digital pedagogic strategy and multiple roadblocks, such as academic self‐efficacy, academics perceived usefulness of information and communication technology (ICT) integration, perceived ease‐of‐use of technology, teaching beliefs, and general ICT anxiety (Anmarkrud et al., 2014; Igbaria, Parasuraman, & Baroudi, 1996; Mac Callum & Jeffrey, 2014; Sang, Valcke, van Braak, & Tondeur, 2010; Siddiq, Scherer, & Tondeur, 2016). Blending this type of learning within an existing paradigm of education that is underpinned by a management system anchored in a structured institutional environment is a difficult task.
1.2 Using technology to complement the management of the educational process
Teaching is challenging. Koehler and Mishra (2009) describe teaching as a multifaceted, complicated practice that requires specialist domain knowledge, understanding of the learners engaged in the process, and an increased understanding and ability to integrate technology into the learning ecosystem. Educational institutions have responded to technology integration by investing in dedicated LMSs to help educators manage learning and provide learners with digital access to content. LMSs such as Moodle, Sakai, Blackboard, and Desire2Learn act as a framework for educational providers to organize and deliver their instructional content. They also offer blended learning facilities to promote a constructivist approach to learning. However, many problems exist when using LMSs.
According to Bingimlas (2009), integrating ICT into teaching and learning is a complex process. Most practitioners would agree that the knowledge and use of ICT within education should complement the experience, not act as a barrier which is sometimes the case. Two primary issues have been identified in the literature to suggest challenges in the current LMS model. First, retention rates for courses delivered through a massively open online course have mixed results with a completion rate of less than 14% for 90% of massively open online courses (Breslow et al., 2013; Liyanagunawardena, Adams, & Williams, 2013). Second, the rigidity of the systems, characterized by specificity, stability, and transparency of function (Koehler & Mishra, 2009), results in these platforms acting as a transfer agent or reinforcing the information transfer didactic style of content delivery as research suggests that learners tend to use these platforms as content viewers (Benneaser, Thavavel, Jayaraj, Muthukumar, & Jeevanandam, 2016).
It has been found that not every student profits from the assumed learning opportunities of LMSs (Lust, Collazo, Elen, & Clarebout, 2012). Learners can become autonomous and isolated which can lead to procrastination and drop out when introduced to LMS. This disconnect can be exacerbated by lack of author experience with creating instruction content on LMS platforms (Mazman & Usluel, 2010). Koehler and Mishra (2009) observe digital technologies that promote content adaptation; knowledge creation and active learning are protean (Papert, 1980), unstable, and opaque (Turkle, 1995).
One cannot incorporate ICT into higher education in a blink of an eye, academics warn (Saeed, Nasim, Hussain, & Azeem, 2015). It is described as a gradual, time‐consuming, and ubiquitous process (Stensaker, Massen, Borgan, Oftebro, & Karseth, 2007). Attwell (2007) calls for “new approaches for learning” and not just in the form of a “software application.” Scholars' main areas of concern regarding existing technologies, such as an LMS platforms, are the lack of suitable adaptation embedded into the system, the distance between the author and dynamic content, and lack consideration for the individual learner's learning environment and the specific context. Creating a pedagogical approach bridging effective learning with new technologies for students is difficult when each learner is unique; no two will achieve the same learning outcomes (Benneaser et al., 2016). Utilizing technology in the learning space without embedding an element of suitable adaptation for individual learners could disadvantage learners by constraining their cognitive ability. Without appropriate interventions, the pedagogic preference of the author of instructional content compounded by the limitations of the LMS could act as an interference to with the learning experience.
1.3 An approach to enhancing digital skills and improving performance
One possible solution to enhancing student performance within early undergraduate technical modules would be to migrate towards a flipped learning environment, whereby learners engage with educational resources and arrive at face‐time sessions with domain experience for the upcoming learning event. In an attempt to address the stochastic, non‐linear digital world with the Industrial Age model of education, a flipped learning environment may provide both freedom and structure to support learning. Research suggests that a flipped learning approach enhances the potential for active learning which has been shown to stimulate higher order thinking, problem solving, critical analysis (Lage & Platt, 2000), and student collaboration (Strayer, 2012), while providing feedback to both learners and instructors (Bonwell & Eison, 1991; Bransford, Brown, & Cocking, 1999; McKeachie, 1987; Wenderoth, Freeman, & O'Connor, 2007). This represents a move towards a model that students have become accustomed to through social media platforms (Frand, 2000; Gannod, Burge, & Helmick, 2008; Jonas‐Dwyer & Pospisil, 2004).
Investigations concerning flipped learning have resulted in publications that can be separated into two broad categories. The first set of scholars focuses on the learners' perceptions of and satisfaction with engaging in a flipped pedagogic environment and includes educator perceptions of their flipped teaching practices. These studies primarily involve the administration of validated scales (Hao, 2016a; Hao, 2016b; Hao & Lee, 2016; Koo et al., 2016; Sohrabi & Iraj, 2016) to measure levels of satisfaction, general surveys to gauge qualitative experiences (McCarthy, 2015; Rodriguez, 2016; Schmidt & Ralph, 2016, Evseeva & Solozhenko, 2015), and interviews (Hall & DuFrene, 2016; Nguyen, Yu, Japutra, & Chen, 2015). The second category of research publications focuses on actual student performance within their flipped learning environment and explores whether or not flipped learning pedagogy results in significant increases in learners' academic grades. Of the 15 published journal articles that document quantitative results in 2016, ranging from disciplines such as elementary mathematics, statistics, economics, medicine, radiology, pharmacy, and nursing; mixed results have been reported with regard to positive effects from the implementation of a flipped learning pedagogic strategy. Eight cases report significant performance increases due to flipped learning (Olitsky & Cosgrove, 2016; Lai, & Hwang, 2016; O'Connor et al., 2016; Peterson, 2016; Harris, Harris, Reed, & Zelihic, 2016; Giuliano & Moser, 2016; Cotta, Shah, Almgren, Macías‐Moriarity, & Mody, 2016; Rose et al., 2016). Mixed results were reported in four cases (Yestrebsky, 2015; Liebert, Lin, Mazer, Bereknyei, & Lau, 2016; Heyborne & Perret, 2016; Betihavas, Bridgman, Kornhaber, & Cross, 2016). A small number of cases reported no effect (Bossaer, Panus, Stewart, Hagemeier, & George, 2016; Nishigawa et al., 2016).
The literature above is suggesting that a flipped learning approach is an enhancement to pedagogy. However, questions remain through the mixed results and no effect studies. Therefore, this study was conducted to consider these findings. The overall aim of the research was to investigate the effects of increasing the number of custom, context‐sensitive digital resources and migrating the learning ecosystem towards a flipped learning environment for a first year Computer Architecture module on student performance. Our research was conducted over a period of 4 years at a higher education college in Ireland. The next section of this paper discusses the theoretical framework, followed by the methodology used and results generated from this investigation.
2 THEORETICAL FRAMEWORK
Flipped learning supports the theoretical framework that underpins this study. It should be noted that this framework is executed in an action research environment of the classroom. Flipped learning is different than just flipping content or a classroom. The Flipped Learning Network (FLN, 2014) makes a clear differentiation between flipped learning and a flipped classroom. Although flipped learning is mapped against the four pillars of F.L.I.P, a flipped classroom may lead to flipped learning. A flipped classroom primarily involves supplying learners with additional resources without changing the learning ecosystem. However, the FLN defines flipped learning as follows:
Flipped Learning is a pedagogical approach in which direct instruction moves towards “the group learning space to the individual learning space, and the resulting group space is transformed into a dynamic, interactive learning environment where the educator guides students as they apply concepts and engage creatively in the subject matter.” (FLN, 2014).
The FLN describes the four pillars of the educational ecosystem required for successful delivery of flipped learning as follows:
- Flexible learning environment: a fundamental, physical change is required when organizing the educational ecosystem to facilitate group‐based work.
- Learning culture: moves away from the typical didactic style of traditional lecturing to foster a culture of active learning. Discussion work, peer evaluations, and game‐based assessment are examples of the many active learning instruments that can be used to enhance the learning experience.
- Intentional content: the educational environment should extend the in‐class experience. Instructors should consider developing custom, context‐aware videos to maximize time spent outside of the classroom.
- Professional educator: the role of the traditional lecturer migrates towards a professional educator, whereby the lecturer directs and monitors the learning experience.
Sufficient quantitative evidence to support the mass implementation of a flipped learning strategy across all domains does not exist. This strategy has been recognized to improve student academic performance and support active learning, dependent on the motivation of the students (Chaplin, 2009; Hake, 1998; Knight & Wood, 2005; Wenderoth et al., 2007). The qualitative analysis offers mixed feedback, suggesting that the flipped learning approach depends on both the module content, the learner population (Bishop & Verleger, 2013; Strayer, 2007), and the designer of the learning experience. Therefore, gaps in the literature exist.
2.1 The present study
This study followed an action research cyclical approach as seen in Figure 1 and a four‐step cycle each year.
The module and module components suitable for migration to a flipped learning environment were evaluated before each semester began. Following the selection process, custom videos were developed for the identified content, and learning units were designed to complement the delivery of the content. These learning units differed based on the content being delivered but typically consisted of an active learning component based on students viewing the video for that learning unit prior to meeting in class. Students in Ireland viewed the videos approximately 12,000 times (over 200,000 views worldwide) throughout the life cycle of the study.
3 METHODOLOGY
The methodology associated with the incremental migration of a Computer Architecture module towards a flipped learning approach is presented in this section. First, an overview of the structure of the Computer Architecture module from an assessment perspective is provided. Second, a breakdown of the sample sizes across each of the 4 years of the study is presented. In addition, an overview of the statistical tests used to determine whether or not increased inclusion of flipped learning content over 4 years resulted in increased student performance is discussed. Finally, an overview of the changes implemented across each of the 4 years as the module migrated to a fully flipped approach as defined by the FLN is presented, and the limitations of the study are highlighted.
3.1 Computer architecture module
Because students traditionally find the Computer Architecture module challenging, this module was chosen as a suitable module for the study. The module is typically delivered in the first year of an undergraduate degree on Level 6 of the National Framework of Qualifications in Ireland. The module consists of both continuous assessments (CAs) and a terminal examination (TE) with a 50:50 split between the TE and the CA components. There were two CA components: CA1 (with 40% of the CA mark being awarded for ongoing, biweekly assessments) and CA2 (with 60% of the CA mark being awarded for a CA examination during the semester time). The TE is the final assessment within the module and measures all the learning outcomes from the module. This assessment follows a problem‐based approach that assesses the skills being built throughout the semester. The module is usually scheduled with a 2‐hr lecture slot and a 1‐hr tutorial time slot delivered in a lab environment. Throughout the tutorial, the learners were randomly assigned to group sizes of approximately 15 students.
3.2 Sample participants and statistical procedures
The study considered a total of 185 participants (n = 185) of which 22 participants took assessments in 2010, 49 participants in 2011, 50 participants in 2012, and 64 participants in 2013. Only participants who submitted CAs or completed a TE were included in the analysis for each of the three assessment types.
The study follows a between‐subjects design strategy and attempts to identify whether or not increased exposure to flipped learning pedagogic content results in increases in student performance across years. In particular, the study explores whether or not greater exposure results in monotonic increases in student grades year‐on‐year. The analysis relies on the Jonckheere–Terpstra test statistic (Jonckheere, 1954; Terpstra, 1952), a nonparametric and distribution‐free test for ordered alternatives to determine if greater exposure to flipped content does, in fact, result in increases in student performance across years. Hollander, Wolfe, & Chicken (2013) provide a detailed example of applying the Jonckheere–Terpstra to the measurement of the “Motivational Effect of Knowledge of Performance.” In addition, we rely upon the Kendall Tau‐b test statistic (Kendall, 1938) as a measure of effect size and interpret our reported effect sizes based on Ferguson's (2009) classification scale. In particular, Ferguson's threshold for a practical effect size being those Tau‐b statistic scores of at least 0.2, with moderate effects identified by Tau‐b scores of at least 0.5 but less than 0.8, and strong effects being associated with Tau‐b scores of at least 0.8.
3.3 Year 1 of the study: Traditional lecture
Throughout the first year of the study, the module followed the typical life cycle for the Computer Architecture module which consisted of typical didactic style presentations followed by self‐directed learning in lab environments. This year represented a baseline against which all subsequent years were measured.
3.4 Year 2 of the study: 20% of the module flipped
Twenty percent of the module was migrated to a flipped learning approach, supported by additional YouTube videos in the second year of the study. Lecture time moved towards whiteboard work where learners lead in‐class problem‐solving activities for the concepts that were flipped. The lecturer's role moved from the typical didactic style of traditional lecturing to fostering a culture of active learning, whereby students are central to the direction and construction of the solutions to in‐class problem sets. Throughout the delivery of the module, 80% of the content was delivered through the traditional didactic lecture style.
3.5 Year 3 of the study: 50% of the module flipped
Fifty percent of the module content was delivered through the use of custom digital resources hosted on YouTube. These resources were embedded into the modules' Moodle page a number of weeks before the delivery of the associated topics. Students were requested to watch these videos before attending class. The lecture time was divided into active learning sessions and traditional lectures with approximately 50% of the content being delivered through the use of flipped learning, whereby the delivery adhered to the four pillars of F.L.I.P. as described by FLN.
3.6 Year 4 of the study: Module fully flipped
In the final year of the study, 50% of the module content was delivered through the use of custom digital resources hosted on YouTube (over 6,000 views for the resources in Ireland throughout the delivery of the Module in the final year). Resources were embedded into the modules' Moodle page a number of weeks before the delivery of the associated topic. Students were requested to watch these videos before attending class. The lecture time was fully flipped throughout the delivery of the module. In addition to the custom digital resources, each of the pillars of flipped learning was integrated into the learning ecosystem as follows:
- The learning space was modified to ensure that students could discuss problems in groups during both the tutorial and lecture time slots.
- CAs throughout the delivery of the module were flexible, and each student was allowed a 1‐week period to retake assessments following an initial assessment score. If a student participated in a re‐examination of a minor CA component, the student was required to sit a viva examination (typically a 5‐min oral defence) of the updated assessment submission.
- The learning culture migrated towards an active learning environment as students were directed using student‐centred approaches, including peer reviewing, game‐based assessments, solving real‐world problems in groups, and discussion work.
In addition to the flipped pedagogical approach, a number of traditional presentation style lectures were provided to the students following their engagement with digital content and participation in an active learning session to reinforce content delivery. All materials were made available through the modules' Moodle page. The next section discusses the result of this research.
3.7 Limitations of the study
The study followed the running of the Computer Architecture module over a period 4 years and consequently was relying on students entering into the module to participate in the study. Both the female group and the mature student group (a mature student is categorized by an individual that is over 23 years old when starting a course a higher education college) were affected by the number of students participating in each year of the study in their corresponding groups. This makes it difficult to generalize the findings of the study to a larger population for both these groups. In addition, as the TE measures all learning outcomes at the end of the semester, it was not possible to decouple each question on the paper and map it against the flipped content for each iteration of the examination.
4 RESULTS
This study evaluates the effect on student performance in both CA and TE instruments as a module migrated towards a flipped learning delivery. The results include an analysis of 4 years of data recorded across three assessment components: CA1, CA2, and TE. Additionally, the results for the overall module grades, a weighted combination of the assessment components, are presented. The results are also presented from three perspectives: all participants in the study, within gender subgroups, and within mature and nonmature student subgroups. The Jonckheere–Terpstra test, testing for ordered alternatives or evidence of increased student performance across years, due to increased exposure to a flipped learning delivery, is also presented. The analysis includes the baseline year 2010 when the traditional direct instructional teaching method was used and also excluding the baseline year, focusing on the years where a portion of the module was delivered using a flipped learning delivery method.
The study considered a total of 185 participants (n = 185), of which 22 took assessments in 2010, 49 in 2011, 50 in 2012, and 64 in 2013. Only participants who submitted CAs or completed a TE were included in the analysis for each of the three assessment components. The analysis of assessment type CA1 consisted of 166 participants from the total sample size, with 19 missing assessment pieces. The analysis of assessment type CA2 consisted of 147 participants from the total sample size. The TE included 169 participants. The following subsections present the findings from the study when considering all participants and then for each of the subcategories (all years, baseline year removed, gender, and age).
4.1 The effects observed from all participants
In Figure 2, Panels a through d are shown scatterplots for each of the three assessment types, spanning four academic years for the delivery of the Computer Architecture module. The horizontal axis, in each case, depicts academic Year 1 indicating the results distribution for the 2010 academic year; Year 2 indicating results for the 2011 academic year; Year 3 indicating results for the 2012 academic year; and Year 4 indicating the results for the 2013 academic year. In all figures, the vertical axis indicates a student's grade achieved in the particular assessment scored out of a maximum of 100 points. Additionally, a regression line of best fit for the data was added to each panel as a visual aid to the identification of any latent trends.
Panels a and b suggest the presence of positive linear trends across years in both CA1 and CA2 results with Panels c and d depicting negative linear trends for the TE results, albeit with shallow slopes in all cases. These descriptive results suggest CA scores increase as the module migrated towards flipped learning, and there appears to be a negative trend identified for the TE. The TE has the strongest weighting within the module, contributing to 50% of the overall module grade, which is why we also observe the negative trend impact on the overall module. This is interesting as it suggests that learners gain higher marks in micro assessments throughout the semester, but the apparent increase in their knowledge is not visible in the TE examination.
A more detailed examination of each of the three assessment categories completed under increased flipped learning delivery, as represented across years, shows a drop in median grades from the year 2010 to 2011 with statistically significant declines as seen in Table 1.
| Assessment type | Median grade 2010 | Median grade 2011 | Mann–Whitney U | Z | Asymp. sig (two‐tailed) |
|---|---|---|---|---|---|
| CA1 | 65.00 | 46.50 | 227.00 | −3.322 | 0.001 |
| CA2 | 62.00 | 18.00 | 70.50 | −4.921 | 0.000 |
| TE | 56.50 | 26.00 | 143.50 | −4.634 | 0.000 |
Median grades monotonically increased from 2011 through to 2013 except for TE results. The Computer Architecture module is a Year 1 second‐semester module.
Table 2 shows the results for the Jonckheere–Terpstra test for ordered alternatives and shows statistically significant trends for higher median CA1 scores: TJT = 6,071.5, z = 3.097, p = .002, τb = .183; and CA2 scores: TJT = 4,605.5, z = 2.395, p = .017 with higher medians associated with higher levels of flipped learning pedagogic instruction (“from none in 2010,” “20% in 2011,” “50% in 2012,” and “100% in 2013,” as described in Section 3). A significant negative trend was observed for the TE assessment: TJT = 4,399.5, z = −2.218, p = .027. In addition, the magnitude of the statistical difference observed for the dependent variables: CA1, CA2, and TE were analysed through the application of the Kendall Tau test statistic with the respective effect magnitudes listed in Table 2. For CA1, CA2, and TE, our results show no practically meaningful effects with each respective tau statistic considered to be significant with p values less than .05.
| Period | Assessment type | N | Observed J‐T statistic | Mean J‐T statistic | Std. deviation of J‐T statistic | Std. J‐T statistic | Asymp. sig. (two‐tailed) | Kendal tau‐b |
|---|---|---|---|---|---|---|---|---|
| 2010–2013 | CA1 | 166 | 6,071.5 | 5,008.5 | 343.242 | 3.097 | 0.002 | 0.183 |
| CA2 | 147 | 4,605.5 | 3,920.5 | 286.012 | 2.395 | 0.017 | 0.150 | |
| TE | 169 | 4,399.5 | 5,181.0 | 352.298 | −2.218 | 0.027 | −0.130 | |
| OM | 167 | 5,082.5 | 5,008.5 | 345.248 | 0.214 | 0.830 | 0.013 | |
| 2011–2013 | CA1 | 144 | 4,614.0 | 3,424.5 | 271.562 | 4.380 | 0.000 | 0.285 |
| CA2 | 128 | 4,051.0 | 2,704.5 | 227.631 | 5.915 | 0.000 | 0.409 | |
| TE | 147 | 3,777.0 | 3,564.0 | 279.860 | 0.761 | 0.447 | 0.049 | |
| OM | 149 | 4,724.5 | 3,667.5 | 285.776 | 3.699 | 0.000 | 0.237 |
4.2 Removing the baseline year
The 2010 student cohort was excluded to investigate the effects of increasing the amount of flipped learning without comparison with the traditional lecturing approach. The Jonckheere–Terpstra test for ordered alternatives showed statistically significant linear trends associated with a practical effect size for the assessment CA2: TJT = 4,051, z = 5.915, p < .001, τb = .409 across the study. In addition, statistically significant linear trends and small practical effects were observed for both CA1: TJT = 4,614, z = 4.380, p < .001, τb = .285; and overall module: TJT = 4724, z = 3.699, p < .001, τb = .237. The TE exhibited no statistically significant linear trend TJT = 3777, z = 0.761, p = .447, τb = .049. All results are displayed in Table 3.
| Period | Gender | Assessment type | N | Observed J‐T statistic | Mean J‐T statistic | Std. deviation of J‐T statistic | Std. J‐T statistic | Asymp. sig. (two‐tailed) | Kendal tau‐b |
|---|---|---|---|---|---|---|---|---|---|
| 2010–2013 | Males | CA1 | 147 | 4,682.5 | 3,934 | 286.362 | 2.614 | 0.009 | 0.164 |
| CA2 | 130 | 3,471.5 | 3,080.5 | 238.363 | 1.640 | 0.101 | 0.110 | ||
| TE | 150 | 3,419.0 | 4,089.5 | 294.963 | −2.273 | 0.023 | −0.142 | ||
| OM | 149 | 3,940.5 | 4,000.0 | 291.426 | −0.204 | 0.838 | −0.013 | ||
| Females | CA1 | 19 | 85.5 | 61.5 | 13.293 | 1.805 | 0.071 | 0.334 | |
| CA2 | 17 | 69.5 | 46.0.0 | 11.015 | 2.133 | 0.033 | 0.423 | ||
| TE | 19 | 52.0 | 61.5 | 13.293 | −0.715 | 0.475 | −0.132 | ||
| OM | 18 | 67.0 | 53.5 | 12.155 | 1.111 | 0.267 | 0.212 | ||
| 2011–2013 | Males | CA1 | 128 | 3,580.5 | 2,718.0 | 228.085 | 3.781 | 0.000 | 0.261 |
| CA2 | 113 | 3,085.0 | 2,120.0 | 189.346 | 5.096 | 0.000 | 0.376 | ||
| TE | 131 | 2,971.5 | 2,845.0 | 236.007 | 0.536 | 0.592 | 0.037 | ||
| OM | 133 | 3,691.0 | 2,936.0 | 241.545 | 3.126 | 0.002 | 0.212 | ||
| Females | CA1 | 16 | 58.5 | 37.5 | 9.854 | 2.131 | 0.033 | 0.445 | |
| CA2 | 15 | 55.0 | 31.0 | 8.748 | 2.744 | 0.006 | 0.598 | ||
| TE | 16 | 40.5 | 37.5 | 9.846 | 0.305 | 0.761 | 0.064 | ||
| OM | 16 | 57.0 | 37.5 | 9.854 | 1.979 | 0.048 | 0.413 |
4.3 Gender‐based effects
Of the 185 participants considered in the study (n = 185), 166 were male, and 19 were female. Of the males, 147 took assessment CA1, 130 took CA2, and 150 took TE. Of the female participants, 19 took assessment CA1, 17 took CA2, and 19 took assessment TE. Across years, there were 19 male and 3 female participants in 2010, 45 males and 3 females in 2011, 47 males and 3 females in 2012, and 55 males and 9 females in 2013.
Excluding the 2010 student cohort and considering only the years 2011 through to 2013 where a proportion of flipped learning instruction was included within teaching and learning practice, the Jonckheere–Terpstra test for ordered alternatives for the female subgroup showed a statistically significant linear trend associated with a significant moderate effect on the assessment CA2: TJT = 55, z = 2.744, p = .006, τb = .598. Significant trends and small practical effects were observed for females on CA1: TJT = 58.8, z = 2.131, p = .033, τb = .445; and for males on CA2: TJT‐CA2 = 3,085, z = 5.096, p < .001, τb = .376. Statistically significant small practical effects were observed for males on CA1: TJT = 3,581, z = 3.781, p < .001, τb = .261. All results are presented in Table 3.
4.4 Age‐based effects
Of the 185 participants considered in the study (n = 185), 145 were nonmature students and 40 were mature students. Of the nonmature student participants, 130 took assessment CA1, 115 took CA2, and 134 took TE; of the mature student participants, 36 took assessment CA1, 32 took CA2, and 35 took TE. Across years, there were 15 nonmature and 7 mature participants in 2010, 41 nonmature and 8 mature participants in 2011, 39 nonmature and 11 in 2012, and 50 nonmature and 14 mature participants in 2013.
Table 4 shows the results of the Jonckheere–Terpstra analysis used to determine whether or not statistically significant linear trends existed for nonmature students or mature students in each of the three assessment categories (excluding the 2010 results). These results showed a statistically significant linear trend associated with a moderate effect size for the assessment CA2: TJT = 176.5, z = 3.185, p = .001, τb = .505. Statistically significant small practical effects were observed for the mature student subgroup and were associated with statistically significant linear trends for the two assessments CA1: TJT = 204.5, z = 2.818, p = .005, τb = .422; TE: TJT = 174.5, z = 2.071, p = .038, τb = .316. Small practical effects were associated with significant linear trends for the nonmature student group for both assessment CA1: TJT = 2,869.5, z = 3.517, p < .001, τb = .257 and CA2: TJT = 2,529.5, z = 4.991, p < .001, τb = .288.
| Period | Mature status | Assessment type | N | Observed J‐T statistic | Mean J‐T statistic | Std. deviation of J‐T statistic | Std. J‐T statistic | Asymp. sig. (two‐tailed) | Kendal tau‐b |
|---|---|---|---|---|---|---|---|---|---|
| 2010–2013 | Nonmature | CA1 | 130 | 3,683.5 | 3,049.5 | 237.667 | 2.668 | 0.008 | 0.179 |
| CA2 | 115 | 2,785.5 | 2,383.0 | 197.793 | 2.035 | 0.042 | 0.145 | ||
| TE | 134 | 2,719.5 | 3,230.5 | 248.455 | −2.057 | 0.040 | −0.136 | ||
| OM | 133 | 3,175.5 | 3,158.5 | 245.251 | 0.069 | 0.945 | 0.005 | ||
| Mature | CA1 | 36 | 295.5 | 236.5 | 35.125 | 1.680 | 0.093 | 0.217 | |
| CA2 | 32 | 225.5 | 187.5 | 29.518 | 1.287 | 0.198 | 0.177 | ||
| TE | 35 | 203.0 | 224.0 | 33.708 | −0.623 | 0.533 | −0.082 | ||
| OM | 34 | 230.0 | 207.5 | 32.095 | 0.701 | 0.483 | 0.094 | ||
| 2011–2013 | Nonmature | CA1 | 115 | 2,869.5 | 2,187.0 | 194.077 | 3.517 | 0.000 | 0.257 |
| CA2 | 102 | 2,529.5 | 1,720.0 | 162.200 | 4.991 | 0.000 | 0.388 | ||
| TE | 119 | 2,364.5 | 2,338.0 | 204.107 | 0.130 | 0.897 | 0.009 | ||
| OM | 120 | 2,949.0 | 2,378.5 | 206.740 | 2.760 | 0.006 | 0.197 | ||
| Mature | CA1 | 29 | 204.5 | 135.0 | 24.664 | 2.818 | 0.005 | 0.422 | |
| CA2 | 26 | 176.5 | 109.5 | 21.037 | 3.185 | 0.001 | 0.505 | ||
| TE | 28 | 174.5 | 126.0 | 23.420 | 2.071 | 0.038 | 0.316 | ||
| OM | 29 | 216.5 | 135.0 | 24.648 | 3.307 | 0.001 | 0.497 |
5 DISCUSSION
The aim of this study was to investigate the effects of increasing the number of custom, context‐sensitive digital resources and migrating the learning ecosystem towards a flipped learning environment for a first year Computer Architecture module. Students were provided with a flexible learning environment, nondidactic active learning, and context‐aware videos together with a professional educator in line with the recommendations of the FLN. The research was conducted over a period of 4 years at a higher education College in Ireland where the impact of these additional resources on student performance in both CA and TE was investigated. The study focused on all participants in the Computer Architecture module.
When considering all participants, this study supports the quantitative literature showing the positive effect on student performance in the instance of CA (CA1 and CA2) grades. However, negative trends were found to be present in the TE which challenges the broader literature and shines a light on the research which has produced mixed or no evidence of effect (Bishop & Verleger, 2013; Strayer, 2007). The TE has the strongest weighting within the module contributing to 50% of the overall module grade which is why we also observe the negative trend impact on the overall module.
Our study has shown a significant negative change in all categories of assessment results from the year 2010 to 2011. In particular, we observed a significant negative change in median grades from 2010 (the baseline year) to 2011. The 2010 baseline year in this study followed a direct instruction pedagogical approach, whereas the 2011 academic year was a departure in the form of inclusion of flipped learning instructional content. Speculating on the fundamental cause of this decrease, we would direct attention to the work of Hattie (2009). His meta‐analysis identified that direct instruction teaching strategies were associated with large effect sizes with regard to student performance. Flipped learning instruction being a departure from a direct instruction pedagogic approach has failed to exceed or even match the effectiveness of the direct instructional approach in this case.
In 2011, the module introduced the first integration of flipped learning into the course; and, although the literature proposes that flipped learning supports active learning, the transition away from the accustomed and predominant didactic approach followed in other modules raised many challenges for learners. Initially, the introduction of flipped learning delivery had negative impacts on performance as seen in Table 1. This supports existing research where students experience difficulty in simultaneously creating new knowledge while also learning to navigate technology tools or platforms. Despite the recognized ease of integrating flipped learning into a module (Frand, 2000; Gannod et al., 2008; Jonas‐Dwyer & Pospisil, 2004), the lecturer noted that the learners' engagement in the module and with the delivery method increased throughout the semester as learners became accustomed to the delivery.
Notwithstanding the baseline year, the transition to flipped learning appeared to be less disruptive, as more instructors used technology in their classrooms. Therefore, learners entering the Computer Architecture module had already been exposed and were perhaps more accustomed to this pedagogic approach prior to entering Computer Architecture.
This may have contributed to better results as evidenced by the monotonical increase in median grades after 2011. Therefore, reinforcing the need for a gradual incorporation of ICT (Saeed et al., 2015).
However, this increase was present in the CAs and not in the terminal exam (see Table 3). The reasons why learners gained higher marks in micro assessments throughout the semester without the apparent increase in their knowledge being visible in the TE are likely to be varied. It is recognized that flipped learning enhances the potential for active learning, which in turn has been recognized to contribute to many perceived learning activities, such as, higher order thinking, problem solving, and critical analysis (Wenderoth et al., 2007; Bonwell & Eison, 1991; Bransford et al., 1999; McKeachiel, 1987), and increased learner collaboration as identified by Strayer (2012). However, from an interpretivist perspective, the lecturer observed that students became reliant on the expectation of additional digital resources, and the knowledge construction during class time dropped during the delivery of the module in 2011. This is counterintuitive as student's domain competence and active engagement grew during the semester; the ability to contextualize in‐class experience against the knowledge gained in the module should have converted into performance outcomes across all assessments. Shifting learning from didactic instruction to digital resources may have resulted in the consumption of information but not necessarily the cognitive construction (Philips, 2000) necessary to perform in a TE. The literature suggests “taking notes from lectures makes huge demands on the limited resources of the central executive and storage components or working memory” (Anderson & Armbruster, 1986, p. 221) by creating skills of “encoding, articulation and rehearsal” (Bligh, 2000, p. 121). In a flipped environment, note‐taking practices are less time constrained and transparent to the instructor and therefore might provide some insight into the TE performance. The proliferation of information and notes, together with the noise of technology, might be inhibiting learners' ability to actively construct or generate their knowledge. However, further research is warranted to investigate if this is a contributory factor.
The results also suggest differences within gender categories. Excluding the baseline year, the general results suggest a greater overall acceptance and benefit from flipped learning pedagogic inclusion for females in comparison with their male counterparts. In addition, as the module migrated towards a flipped learning delivery model, particular increases were observed among mature students. Results show that learning gain in the CA components transferred into a positive small practical effect on the TE for mature students. It was noted by the lecturer that the mature students engaged in more note taken during class time when compared with the nonmature students as discussed. This may be the key factor differentiating between the effects observed on the TE.
6 CONCLUSION
This action research study provides evidence that integrating flipped learning into a technical module at an early undergraduate stage has a significant positive impact on the CAs throughout the module and a significant negative impact on the TEs. It should be noted that there are many varying implementations of flipped learning, and this study cannot address the full spectrum of these approaches but only consider the effects of the approaches used within the study, described within the Section 3. When including the baseline year, the results show that migrating the Computer Architecture module towards a flipped learning approach has not resulted in a practical effect on the learning experience across the 4 years. For instructors, the investments into digital pedagogy, which requires enormous time resources in additional to technical upskilling, might not produce a commensurate output in gained student performance. These findings were in line with other studies that observed no effects following the migration towards a flipped learning approach (Bossaer et al., 2016; Nishigawa et al., 2016).
The paper also presented deeper analyses of the results by removing the baseline year and comparing only modules that contained elements of flipped learning. Interpretivist observations contend peer‐to‐peer engagement increased throughout the delivery of each module and further as the module migrated towards the flipped learning approach. This engagement between the students had a statistically significant positive effect with a medium effect observed for both males and females and a strong effect on mature students. It would be expected that as students become more use to participating in flipped learning approaches, the effects on CA1 would also become stronger. Preliminary evidence suggests that inclusion of flipped learning at an early stage in the learning ecosystem is not necessarily suitable for Computer Architecture, as learners are working toward understanding and building domain competence. Equally, the flipped learning approach may be better tested at a later stage in the academic path, when learners have built‐up sufficient domain competence to enable them to participate in active learning sessions, particularly in technical modules.
Finally, this study posits the question; does providing students with multiple digital resources impact the students' note taking during class time? This is important considering the statistically significant negative impact on the TE. This research contends that changing the vessel of didactic rigidity (Koehler & Mishra, 2009) from the lecturer to a technology platform may merely mean a transfer of agents rather than an evolution of pedagogy. Thus, this research proposes the consideration of removing technology from the educational ecosystem and delivering a Chalk and Talk approach could potentially provide a better approach for early undergraduate students, and flipped learning could be integrated later into the educational path. Equally, research into “note taking” in the digital environment may also be a contributing factor and warrants further investigation. Ultimately, further investigation into digital pedagogy practices that lever technology with learners' (Bates, 2010; Danciu & Grosseck, 2011) performance outcomes is required particularly in relation to the TE to determine if the effects on the TE decreases as a module migrates towards a flipped learning approach during a technical module delivered at a later stage in the educational path.
