Study on the use of a combination of IPython Notebook and an industry‐standard package in educating a CFD course

It is common that industry‐standard packages are used in teaching professional engineering courses in final‐year undergraduate and postgraduate levels. To improve the competency of students in using such professional packages, it is important that students develop a good understanding of theoretical/fundamental concepts used on the packages. However, it is always a challenge to teach theoretical/fundamental concepts in the computational‐related courses. The teaching of such subjects can be improved by the use of advanced open‐source web applications. The present research proposes an approach based upon the combination of Jupyter Notebook and an industry‐standard package to teach an applied, computationally related course. We investigate the use of backward design and a novel tool called IPython (Jupyter) Notebook to redesign a postgraduate Computational Fluid Dynamics (CFD) course. IPython Notebook is used to design a series of integrated lecture slides and tutorial tasks, and also one of the assignments for the blended‐learning‐based, semester‐run, CFD course. The tool allows the implementation of backward curriculum design and a learn‐by‐doing approach in redesigning the course. The materials produced were used on the first part of the course which contributed 40% towards the course's final mark and delivered the fundamental concepts of CFD over the first half of the semester. The remaining 60% of the mark was based on a final project from the materials taught on using an industry‐standard CFD package in solving complex CFD problems during the second half of the semester. It was shown that the Ipython environment is a very useful tool which provides learning‐by‐doing practices allowing students to have a coherent integrated lecture, tutorial, and assignment material in a highly interactive way. It improved (a) students' engagement in teaching complex theoretical concepts, (b) students satisfaction of the course and (c) students performance in working with the industry‐standard package over the second half of the semester.

The education of engineers is generally targeted to (a) prepare students for research, (b) prepare graduates for employment in the engineering industry, (c) prepare engineering/science/numerically literate citizens for society and (d) provide an intellectually stimulating education [1]. It is widely acknowledged that academic performance in Science, Technology, Engineering, Mathematics (STEM) subjects plays a pivotal role in sustaining economic growth and advancing technological innovations. However, a diminishing interest of young people to pursue STEM subjects increasingly becomes a serious concern for many nations such that the issue is sometimes called the STEM crisis or national crisis [17], for example, Australia [6], Canada [25], India [13], Ireland [11], Japan [23], UK [7,26] and the US [18]. In the case of the UK, it is expected to become a more challenging issue after Brexit [5]. It seems a pedagogical reform may be needed to improve motivation, participation, retention and graduation of students in STEM subjects. Among the important elements to achieve this are leveraging active inclusive learning environments and innovative teaching pedagogy. The learning environment can be improved by introducing new technologies in blended formats into the teaching that combine classroom, online and community-based learning opportunities.
Backward Design is an approach that proposes an innovative way of designing curriculum, pedagogical and education research [15]. On the traditional approach, the pedagogy deals with the coverage and activity-oriented design. In contrast, the backward design identifies the goal of teaching the course first, and then appropriate curriculum, assessment, and finally learning activities for study are designed. In summary, the approach involves the following three stages: (a) identify desired results, (b) determine acceptable evidence of successful learning and (c) plan and design learning experience and activities.
IPython (Jupyter) Notebook is a technology which allows an interactive web environment, in which one can create and share documents that contain live code, rich text, mathematics, plots and rich media (see Figure 1a). Another interesting feature of the platform is that it is open-source and the code can be run in multiple programming languages (e.g., Python, Julia, MATLAB, etc). Hence, the users would not necessarily need to learn a new programming language, rather they can run the codes with an available kernel of their choice. The use of the platform has increased significantly in the last few years in comparison with other popular computing packages (e.g., Matlab and R; see Figure 1b).
Given the variety of toolsets which can be used within the platform, it provides an interactive and exploratory environment for teaching, tutorials, as well as assignments of computing-related teaching. Moreover, computer codes can be embedded in books; hence, a reader can interactively change the content of the code [9]. The following provides a brief summary of some of the examples in which IPython Notebook has been used for teaching.
Chemical and geoscience engineering [24] used IPython Notebook in teaching an electronic structure to undergraduate physical chemistry and materials science courses. An integrated environment was developed for the theory and programming code that converts real space vectors to reciprocal space vectors. The authors proposed IPython Notebook as an effective tool particularly for teaching the conversion of real space vectors to reciprocal space vectors which is necessary for more elaborate electronic structure calculations. Moreover, a systematic pedagogical framework was proposed for teaching this tool to students in physical chemistry and materials science courses. Another very successful experience of using IPython Notebook in teaching is given by Jacobs et al. [14] to teach computer programming to geoscientists.
Using anonymized student marks and feedback data over the academic years between 2010 and 2014, the authors concluded that the tool is a very useful environment to improve teaching programming. Computational biology [19] developed a flipped-classroom approach for an undergraduate course on computational biology, by making the use of IPython Notebook. A new classroom was also designed for the course to improve the engagement of the students in the class. Economics, business and finance [22] used the IPython Notebook environment to develop some interactive tools for teaching economic models and cash management courses. The authors studied an illustrative example of the application of the approach on a cash management course. They discussed relevant questions raised by the students in the classroom. The authors claimed that the tool could improve the teaching of the subject (particularly those with quantitative models) through promoting a learn-by-doing approach. Moreover, it could connect theory to practice by enabling students to implement their own decision support tools. Data literacy. IPython Notebook was used by Wermelinger [27] for teaching data literacy at the Urban Data School. The author indicated that "It is a no-brainer, when you think about it, even for primary, because you have your code, inputs and outputs all in one window, and can step through code execution instead of (or as well as) using more conventional tools like Trinket, Thonny or IDLE". Physics & Astronomy, Mathematics. A very useful series of IPython quantum mechanics lectures are provided by Dawes [8] by making the use of QuTiP (Quantum Toolbox in Python) package. Bowler [3] used IPython Notebook to teach physics to Physics & Astronomy students at University College London. A number of useful IPython Notebooks were produced for the third year quantum mechanics course. The Notebook was found to be an interactive tool. Ramz [21] developed a platform to improve teaching statistics through IPython. As per the feedback which the authors got from the learners, the content of the ICT course was sufficient, short and accurate.
Computational Fluid Dynamics (CFD) is one of the pioneering IPython Notebook teaching materials for STEM courses and was delivered by Barba and Forsyth [1]. A step-by-step approach towards computational solution of Navier-Stokes equations, as one of the most important and complex equations in fluid dynamics, was provided using 12 Jupyter Notebooks. As per the feedback from quite a few academics around the world who have used the tool, the work was not only very successful in teaching this particular subject but has inspired for many other Jupyter course developments, including the present project.

| Present work
MSc and final-year MEng students in Maritime and Mechanical Engineering at LJMU took a course called Advanced CFD. For the academic year 2018-2019, a total of 61 students were enrolled in the class. The course aims to (a) explore the underlying theory of CFD industry-standard codes; (b) investigate performance and reliability of CFD codes in engineering applications and (c) enable students to have a critical approach towards the appraisal of CFD predictions. The present research aims at introducing a novel approach for the pedagogical design of the CFD course. A framework called Backward Design is used and new state-of-the-art Jupyter Notebook technology is embedded into the course.
F I G U R E 1 (a) IPython (Jupyter) Notebook; (b) monthly active users (MAU) of IPython Notebook and other two popular computing tools [10] 2 | METHOD This section describes the structure of the CFD course and also the approach used to redesign the course. Figure 2 shows the intended learning outcomes (ILOs) of the course. Although the first two ILOs are focused on understanding the fundamental concepts about CFD, the last two ILOs deal with teaching and enhancement of students ability to carry out an appropriate CFD simulation using an industry-standard CFD package.

| Course's old design
The old design refers to the academic year 2017-2018, with a total number of 53 students registered in the CFD course. The teaching and learning materials included a series of PowerPoint lecture slides and tutorial sheets (see Figure 3a).
The theoretical aspects of CFD are covered in the lectures, with students having the opportunity to practice CFD simulation of flow via a range of tutorials and assignments: • First-half of the course: 4-week-tutorial sessions to do tutorials on CFD simulations using an industrystandard software, and 1 week to practice several approaches of CFD simulations of steady advectiondiffusion problems, using an Excel spreadsheet. An invigilated CANVAS virtual learning environment (VLE) test (AS1; weighted 40% towards the course's final mark) was taken in Week 6, concerning CFD simulations of a simple model using the CFD industrystandard software. • Second-half of the course: 3 weeks of tutorial sessions to do further tutorials on CFD simulations using the industry-standard software. A project concerning CFD simulation of a complex flow was then introduced to the students and they worked on the project over the last 3 weeks of the course. For the second assignment (AS2; weighted 60% towards the course's final mark), students submitted a scientific report on the CFD project at the end of the semester.

| Course's new design
Course feedback from both the students and staff on the old design indicated that (a) students were struggling (and less interested) to understand the theoretical aspects of CFD; (b) there was no appropriate assessment for such aspects of the course (the first two ILO's in Figure 2). This was a problem because the understanding of the theoretical aspects is imperative to help students to achieve ILOs. Inspired by the improvements of the pedagogical practice obtained from the literature discussed in Section 1, we redesigned the course for the subsequent academic year, 2018-2019, in which a total number of 61 students were registered in the course. The major changes were made to the first part of the course, in which complex CFD theories were taught (i.e., firsthalf of the course in Figure 3b). Accordingly, we (a) used a framework of Backward Design and (b) embedded some Jupyter Notebooks, to integrate lecture slides, tutorials and assignments in an interactive environment.
Feedback discussed in the class with students, examination results and comments from other CFD instructors whom the authors were in contact with has been used to "qualitatively" evaluate the performance of the implemented approach. This section discusses details of the ways in which Backward Design is applied to the course, and also the structure of the Jupyter notebooks and how they are delivered. Three Backward Design principles, for curriculum design, are followed in redesigning the course: Step 1: Identify desired results. This concerns what the lecturer expects students to achieve when they have completed the course. Hence, this step may characterise the ILOs of the course. We found the ILO's associated with the course's old design as a series of very appropriate results could be expected from the students who take this course. Hence, no change was made to the ILOs ( Figure 2).
Step 2: Determine acceptable evidence of successful learning. This determines the method used to assess what students have learnt from the course and whether they have accomplished the learning outcomes. Hence, this step may characterise the type of coursework, assessment and exam. The first assignment on the course's old design consisted of the use of the CFD industrystandard software to simulate a simple model, without in-depth knowledge of the theory underpinning commercial CFD codes. Therefore, it was unable to fully assess students particularly for the first two ILO's. The assignment was replaced with the new first assignment which was an open-book, invigilated, CANVAS VLE test. Five multi-choice questions (selected response) together with three open-ended questions (constructedresponse) were designed on CANVAS. The questions required students to run Jupyter scripts to generate results, plot relationships in IPython or Excel spreadsheets (the files can also be submitted) and comment on the findings.
As feedback, the Jupyter Notebook scripts were returned to students, with marks and feedback, and answers were discussed in class to enhance learning.
Step 3: Plan and design the learning experience and activities. This characterises the activities that students need to do during the course to prepare for the course assessments. This step characterises tutorials, homeworks and any other lecture material that will be used over the lecture and tutorial sessions. We designed a series of Jupyter Notebooks each of which consists of an introduction to the theoretical concepts, programming codes, and tasks. Figure 4 shows three snippets for different parts of one of the IPython Notebooks, which concerns the finite volume method for steady advectiondiffusion problems with zero source term (Figure 4a). Note that the snippets demonstrate just a section of each part of the notebook. Below are some details about the structure of the notebooks: • Introduction to the theoretical concepts. The notebook starts with some descriptions as an introduction to the theoretical aspects; Markdown cells are used for this part (Figure 4a). This component of the document is generally delivered during a 1-hr lecture session in each week. • Programming code. The programming part (Figure 4b) is embedded using code cells and contains the mathematical theories which are discussed in the first part. It is mainly delivered during 2-hr tutorial sessions during the week. Considering the fact that some students started the course with limited (or even no) experience of programming, the code is presented in a step-by-step manner. Moreover, some comments are added next to each line, to further describe the line. For instance, for the notebook shown in Figure  An introduction session was delivered to students, as a crash course on basics and how to run the Jupyter Notebook, during the first tutorial session of the course ( Figure 5). Despite being short, the introduction was prepared on a step-by-step basis, and this helped students (in particular those who had limited knowledge of programming skills) to get into the topic. It is believed that it helped to prepare and expose students to the techniques and tools built into the course. To get students fully prepared for running CFD codes in Jupyter, they were also asked to practice the Introduction notebook on their self-study time over the coming week, before the Week 2 session.

| Weeks 2-6
We went through one Jupyter Notebook each week. The theoretical concepts (which were implemented as the first part of the Jupyter Notebook) were discussed in a 1-hr morning lecture session which was held in a lecture theatre. It was then followed by 2-hr, afternoon, practical session during which students were asked to follow the tasks outlined in the notebook.

| Week 7
A 3-hr session was devoted to assessing the students on the first part of the course using an open-book invigilated VLE test, details of which are described earlier in Section 3.

| FEEDBACK ON THE STUDENT EXPERIENCE
An anonymous survey was used to collect students' views and feedback to the implemented approach, though it may not give an accurate measure of actual learning. The survey contented six questions and was designed using Vevox. It was released 2 weeks from the end of the course and kept open for 1 week. Table 1 shows the questions and also the options provided for each question.

| 957
The total number of students in the class was 61 from which 30 participated in the survey. The results of the first five questions of the survey are shown in Figure 6. Dominant participation of MEng students (83%) compared with that of the MSc (17%; Figure 6a) is consistent with the fact that MSc students were 25% of the total number of students on the course. A good student's satisfaction is demonstrated (see • "The use of IPython notebook was an excellent way to learn the fundamental concepts of CFD and was a very useful and enjoyable aspect of the module". • "Overall enjoyed the module, the lectures were informative and engaging and the tutorial helped increase the level of understanding". • "I think this is one of the best analysis modules I have ever done! All members of staff who work on the module are so helpful and nice!" • "One of the best-taught modules of the year, clear instructions and fed well into the in-class assessment".

| DISCUSSIONS AND CONCLUSIONS
We have presented a novel framework for teaching an advanced CFD course. A successful model of curriculum planning in education, called Backward Design, has been used in redesigning the CFD course. The novel framework proposes to split the teaching weeks into two: (a) the first half of the semester consists of teaching primarily some important fundamental theories used in the industry-standard CFD packages. This includes doing tutorials and lectures, and eventually taking an openbook, invigilated assignment, using Jupyter Notebook; (b) over the second half of the semester, teaching and tutorial sessions devote to teaching/practising various aspects of CFD simulations using industry-standard packages. Finally, students are required to study a CFD problem using the package and submit a report on the project as their second assignment. IPython (Jupyter) Notebook was used to design assessment and teaching materials for steps two and three of the Backward Design model, respectively. There is an increasing growth of embedding Jupyter Notebook in various Engineering and Science subjects. The consensus conclusion by all the studies reviewed herein is that Jupyter Notebook is an incredible tool to provide blended learning and embed learning-by-doing into lectures, tutorials, as well as open-book, open-web, computer-based assignments. Moreover, its flexible programming section allows the learners to examine and debug their programming codes and make comments for the instructor on the notebook. In addition, the approach can help to promote active learning which involves students in doing things and thinking about the things they are doing [2]. Finally, Jupyter Notebook is starting to be considered as a great tool for use by publishers which gives the readers an opportunity to interactively change the content of the programming codes [9]. Specific conclusions which can be made from implementing the approach to CFD module at LJMU are as follows: • Assignments. An open-book invigilated VLE test was designed to assess students for the first two ILO's of the course. Compared with the paper-based, the computer-based assignment provides faster, more accurate marking, and reduced test administration times, and make possible relatively low-cost scaling to large numbers of test-takers. Computer-based assignments could also be designed to meet the needs of special populations, including people with physical disabilities and people from-diverse cultural or linguistic backgrounds [16]. Due to the nature of the open-book, open-web assignment, students skills on higher levels of Bloom's taxonomy can be evaluated [12]. The assignment constructed for the CFD module was designed to comply with the requirement of assessment practice, namely validity, reliability, rigour, probity and fairness [20]. Fundamental theories used in the industry-standard package theory assignment were made available, using the same IPython scripts which students had worked with for a few weeks. Hence, on the exam, students knew the code environment and could focus on analysis and interpretation of the results which were asked (Validity). All questions were either multiple choice or a specific, detailed pre-prepared model answers were used. This made the marking more consistent (Reliability). The questions were encrypted with a password, but the password was announced to students at the beginning of the exam. Also, before the start of the exam, lecturers made sure that IT technicians had preloaded and opened Ipython on all machines and they were all ready to use. We also provided a printed version of the questions and asked students to write their answers on the paper as well F I G U R E 6 Results of the survey as entering them into VLE, to avoid possible interruption of the exam just in case CANVAS/internet connection went down (Rigour, Probity and fairness). • Lectures/tutorials. IPython Notebook is a useful tool to design learn-by-doing teaching materials. It combines computations, writing and mathematics in a highly interactive environment. The Notebooks are designed in a manner that the taught materials can be integrated into the second part of the module on which the use of a commercial package in the simulation of more complex problems was taught. This helped us to teach some of the challenging theoretical/fundamental concepts that appeared when using some of the parameters in the industry-standard package, by providing a blended, learning-by-doing learning environment for the students: (i) The concept/parameter was referred to a particular Jupyter notebook in which the embedded programming code contains the particular parameter, for a simple problem, (ii) The particular notebook was projected on the screen and explanations were given while the lecturer running the notebook live. (iii) In the computer-lab tutorial session, students were asked to run the annotated Jupyter Notebook themselves, and practice the effect of changing the parameter being investigated. Despite problemsolving with no guidance, it is believed that the worked examples, established within the IPython Notebook for the complex CFD concepts, provide learning improvements [4]. The improved mark for AS2 in the academic year 2018-2019 may suggest that when the students were running the industry-standard CFD package (in the second half of the course) to simulate reallife complex fluid dynamics problems, they had a better understanding (compared with the student cohort took the course in the previous year on the old design) of the parameters required to set-up a simulation in the package's solvers. The students' F I G U R E 6 (Continued) SEDDIGHI ET AL.
survey and also comments from other instructors suggest that the implemented approach enhanced engagement of students in particular for teaching the fundamental theory of CFD. Moreover, compared to the old framework taught before, the class average mark on the second assignment, AS2, increased from 62% to 66%. Given the fact that the new design did not change much on the second part of the course, this can be an indication that student's understanding obtained from the first part on the theoretical concepts and CFD analysis may have helped to improve their performance for the second part as well. Also, it was possible to include more interests and aspirations of all of the students in the cohort, in designing the lecture, tutorial, and assignment. In addition, one of the most important benefits of using the approach was that the Jupyter notebooks made it much easier (and sensible to students) to explain a concept used in the industrystandard CFD codes by referring to some simple programming parts in the notebooks.
Finally, the tool is open-source, needs zeroinstall hosted service, and is compatible with various platforms.
• Contribution of the present study to the literature. The major contribution into the available literature from the present research is to propose an approach based upon the combination of a Jupyter Notebook and an industrystandard package in teaching an applied, computationally related course. Applied computationally related courses in engineering subjects are usually taught using either "programming-based" (e.g., using IPython Notebook, Matlab, Excel) or "commercial package-based" approaches. The former approach mainly focuses on teaching theoretical/fundamental aspects of the modelling but usually limited to considering simplified problems only. The latter approach, on the contrary, can consider the modelling of more complex problems but is generally lacking in the teaching of theoretical/fundamental aspects underpinning the packages. We have shown here that IPython Notebook is a great tool that can be used in combination of commercial packages in the teaching of computationally related courses. The results are presented for the implementation of the blended approach to a thermofluids course. However, we believe the proposed blended teaching method can be applied to many other Engineering subjects (e.g., Civil Engineering, Chemical Engineering) in which numerical courses are taught. • Suggestions for future work. As discussed in Section 2, evaluation of the effectiveness of the implemented approach was based on feedback obtained from students and other instructors and examination results. More quantitative data may have to be analysed to judge better the effectiveness of the proposed framework. Moreover, for the use of the developed assignment, it was a bit difficult to assess the engineering process necessary to reach the correct answer. So, perhaps some new-type questions should be designed to improve the assessment for that aspect.

AUTHOR BIOGRAPHIES
Mehdi Seddighi is a Senior Lecturer in Fluid Mechanics at Liverpool John Moores University. He is a MSc in Aerospace Engineering (Aerodynamics) and holds a PhD degree in Fluid Mechanics from University of Aberdeen. His research interests include unsteady turbulent flow, control of turbulent flow, hydrodynamics and marine renewable energy, using numerical and experimental techniques.
The numerical approach includes, a high-fidelity inhouse DNS (direct numerical simulation) and LES (large eddy simulation) code, CHAPSim, which is developed initially by him and then further developed by other researchers under his supervision over the past decade. Mehdi is a Fellow of the Higher Education Academy and is interested developing novel techniques for teaching Thermofluids subject and Computational Fluid Dynamics courses at undergraduate and postgraduate levels.
David Allanson has held an academic post at LJMU since January 1998. Prior to that he was a member of the Advanced Manufacturing Technology research group and completed his PhD in 1995. During his time in the laboratory he was involved in the development of intelligent and adaptive control systems and the development of supporting process models. Since joining the academic staff, David has delivered modules over a range of engineering topics including Thermodynamics, Fluid Dynamics, Computational Fluid Dynamics (CFD) at undergraduate and postgraduate levels. He is currently interested exploring different techniques for the teaching of engineering CFD and encouraging the development of a robust approach to engineering analysis. David also spent some time working for National Nuclear Corporation (NNC) and worked in the commissioning division at Heysham 2 AGR where he worked on the testing of reactor instrumentation and safety circuits. David is currently Programme Leader for the Marine and Offshore Engineering MSc programme and assist with the management and running of the Mechanical and Marine Engineering undergraduate programmes.
Glynn Rothwell received his BSc degree in mechanical engineering from the University of Manchester in 1983. He subsequently spent some time in industry working on a variety of nuclear engineering projects. In 1991, Glynn was appointed as a senior lecturer in mechanical engineering at Liverpool John Moores University. He received his PhD from in 2003 for a thesis entitled "Fracture Toughness Determination Using Constraint Enhanced Sub-Sized Specimens". His current position is Subject Head for Mechanical Engineering. Glynn is a Chartered Engineer and a Fellow of the Institution of Mechanical Engineers.