Using technology to develop teachers as designers of TEL: Evaluating the learning designer

This paper reports on an iterative design-based research project to develop an online design tool (the Learning Designer) to support “teachers as designers.” The aim is to evaluate the potential of the tool to develop and support a knowledge-building teaching professional community. The Learning Designer was embedded and evaluated through international online “design challenge” events, and a series of MOOCs, providing both quantitative and qualitative data. Findings indicate that the Learning Designer enables an online community of teachers from across Our education systems could be understood as massive uncoordinated experiments where, every day, every teacher has the opportunity to test and discover new techniques, and learn from their students what works, and what does not. If this were to happen, it would result in a transformation of into a design science “to keep improving its practice, in a principled way, building on the work of others” (Laurillard, 2012). This paper reports on a Design-Based Research project to develop an online tool (The Learning Designer) to support the process of sharing “instructional products” by developing the teacher-as-designer, in this case by scaffolding teachers’ development and representation of learning designs that can be shared online and adapted by other teachers. Our aim is to harness the inquiry that goes on every day in educational and channel it that of is a very good tool for all the teachers to use and share their work with their colleagues, as well as getting feedback to help us—as teachers—improve or modify our plans.


Introduction
Our education systems could be understood as massive uncoordinated experiments where, every day, every teacher has the opportunity to test and discover new techniques, and learn from their students what works, and what does not. If this were to happen, it would result in a transformation of teaching into a design science "to keep improving its practice, in a principled way, building on the work of others" (Laurillard, 2012). This paper reports on a Design-Based Research project to develop an online tool (The Learning Designer) to support the process of sharing "instructional products" by developing the teacher-as-designer, in this case by scaffolding teachers' development and representation of learning designs that can be shared online and adapted by other teachers. Our aim is to harness the inquiry that goes on every day in educational institutions and channel it into shared knowledge that could benefit the community of teachers.

Teachers' professional development: the potential of technology Teacher collaborative inquiry
Teacher professional development has increasingly been seen through a socio-constructivist lens that shifts the focus from what can be done for teachers to what teachers can do for themselves (for example, Makri, Papanikolaou, Tsakiri, & Karkanis, 2013;Najafi & Clarke, 2008;Schnellert, Butler, & Higginson, 2008;Van Driel & Berry, 2012). This research shows "teachers' learning is facilitated and sustained in communities that are collaborative endeavours, acknowledge and value participants with different levels of expertise, and focus on practice-related issues" (Najafi & Clarke, 2008, p. 244). To go beyond local endeavours to the whole community, Hiebert and Morris propose an approach to improving teachers' methods of teaching by creating and sharing artefacts or instructional products (Hiebert & Morris, 2012;Morris & Hiebert, 2011). Such an approach enables understanding to be accumulated over time. Teachers and researchers create materials, try them out, and "feed the information from the trials back into the products" (Hiebert & Morris, 2012, p. 94). This iterative process of collaborative sharing of scholarship-informed, practitioner knowledge could be the key to reconfiguring teaching as a design science and engaging every teacher in teacher-led inquiry (Laurillard, 2012). This would help teachers

Practitioner Notes
What is already known about this topic • Technological support for learning design could help teachers collaborate online.
What this paper adds • The Learning Designer is a tool being used by thousands of teachers every month. • Teachers' evaluations of the tool indicate a high level of acceptance by teachers around the world and across all sectors. • Teachers suggested improvement in collaborative authoring of learning designs, which is now included in planned development of the tool.
Implications for practice and/or policy • The Learning Designer tool is free to use for all teachers and is available at learningdesigner.org • The tool provides constructionist support for learning design using the theory-based Conversational Framework. • Designs can be shared with other teachers and adapted for their own use.

1046
British Journal of Educational Technology Vol 49 No 6 2018 build their knowledge of teaching by encouraging them to engage in "the production and continual improvement of ideas of value to the community" (Hong, Scardamalia, Messina, & Teo, 2009, p. 374).
Collaborative sharing in teacher networks or communities has long been recognised as important for teacher professional development along with tools to support this: "especially technological ones" (Avalos, 2011, p. 11). However, Buckley (2012) observed that knowledge sharing is a complex process and issues of trust, a lack of incentives, or shared sense of identity, and the absence of a "culture of learning in an organisation," have an impact on the capacity of educators to achieve this effectively (p. 333). Buckley (2012) called for the transformation of educational institutions into "learning organisation[s]" (Argyris, 1994) to facilitate the sharing of practice, especially tacit knowledge "that cannot be replicated and is part of the intellectual capital of the institution" (p. 336). Our research aims at developing a tool to support such sharing.
Tools to support learning design Technology could provide "new ways to share great teaching ideas" in the form of learning designs (Dalziel, 2016, p. xi). Learning design is an approach to planning teaching and learning activities that makes visible teachers' "intuitive processes" (Ghislandi & Raffaghelli, 2015, p. 281).
A learning design is a representation of what happens in a teaching and learning session to help learners achieve specified learning outcomes. It is often structured as a sequence of learning activities that can be shared with others. For online and blended learning, this approach is able to show, not only what is happening when the teacher is with the learners, but also what learners should be doing when the teacher is absent and they are being supported by technology. This helps teachers design in TEL in a more learner-centred way (Dobozy, 2013). Agostinho (2006, p. 2-3) envisaged the representation of learning design in the form of a notation of some kind that could "provide academics with a scaffold to help them design high quality learning environments without investment of excessive amounts of time." The development of computational tools to support the representation of learning designs and to allow others to adapt the designs, has become an important part of learning design research, such as LAMS, DialogPlus, OULDI (Dalziel et al., 2016) but these tools have not yet become part of most teachers' everyday practice (Ghislandi & Raffaghelli, 2015). Our research set out to see if we could develop a tool to make this happen.
A tool to support a learning design community The aim of the Learning Designer project was to implement the well-established and widely referenced Conversational Framework (Laurillard, 2002) as a design tool for teachers, i.e., to embed the theory that had informed its development into something more helpful for practising teachers than a book. The project set out, therefore, to test the idea of building a digital tool to support (i) good pedagogy design, and (ii) the sharing of effective design practice. We wanted to build a pedagogical knowledge-building community by creating a constructionist learning environment for teacher development . Then we could test the extent to which this "open teaching" system enables teachers not simply to use interventions designed by others, but to become innovators in their own right (Laurillard, 2008). As a "teacher-designer" (Goodyear, 2004, p. 341), they would be working in a similar way to the knowledge-building process that operates in the academic research community: improving practice through sharing knowledge products, building on the work of others in the community, developing and testing their own innovative ideas, and then sharing the results with the community (Ferrell & Kelly, 2006;Laurillard & Masterman, 2009). Being online, the community would have the capacity to reach teachers in far greater numbers than could be recruited face to face, providing greater support for new users and helping the community to extend ever wider.

Methodology: Design Based Research
The project used "design-based research" (DBR) as its methodological framework. DBR combines a theory-driven approach with empirical evaluation to construct innovative learning environments that are indicated in theory to be potentially effective, but that remain only marginally understood and practiced. This creates "useable knowledge" as well as advancing theoretical understanding of learning (The Design-Based Collective, 2003, p. 5). The project methodology reflected the six characteristics of the DBR approach (Anderson & Shattuck, 2012) as shown in Table 1.
The Learning Designer project has evolved a set of design principles that meet the needs of teachers coping with heavy workloads and continual demands for innovation, ie, they "reflect the conditions in which they operate" (Anderson & Shattuck, 2012, p. 17).
The cycles of design based research (Plomp, 2007) proposed three main phases for DBR, which were instantiated in our project in the following way (and represented in Figure 1): 1. Preliminary research (including development of theoretical framework) 2. Prototyping (microcycles of empirical testing, evaluation, adaptation and redesign) 3. Evaluation (that may result in recommendations for further refinement) The preliminary research and early prototyping phase The preliminary research stage investigated teachers' conceptions of learning design in higher education to establish a set of requirements for a tool that would support progressive and collaborative innovation by teachers (Laurillard, 2013). The project drew from the related field of Being situated in a real educational context Our users were teachers in schools, colleges and universities; their designs are exported to documents or to a VLE for students to work through Design and testing of a significant intervention Our intervention is the learning design tool, enabling teachers to act as design professionals, itself subject to testing and redesign in the light of user feedback Using mixed methods Our methods were quantitative: analytics of the tool, surveys; and qualitative: open-ended survey questions interviews, blogs, and forum posts Involving multiple iterations Iterative design phases of the project used successive trials to test the conditions where learning designs and pedagogical insights could be shared as a process of teaching as a design science in building community knowledge Partnership between researchers and practitioners We engaged researcher-practitioner collaboration throughout to focus on the co-construction of pedagogic knowledge, and community requirements for the tool Evolution of design principles The design principles reflected in the Learning Designer itself evolved in response to user feedback British Journal of Educational Technology Vol 49 No 6 2018 knowledge engineering to develop a community vocabulary of learning design (see Charlton & Magoulas, 2011). Three main user requirements for learning design support were elicited, and these were addressed in developing a prototype, the Learning Design Support Environment (LDSE).
Iterative evaluation of the LDSE demonstrated strong user support for the way the user requirements were being met, and led to the significant change to an online version .

The microcycles of advanced prototyping
The research prototype was successful as a design tool, but too complex to run online. Building on the recommendations from the teaching community a follow-up project developed the online version, the Learning Designer, which is free and open to use for everyone at https://learningdesigner.org. This more accessible tool addressed the main user requirements in the following ways: Building on the work of others by linking design to educational research findings  The tool guides design decisions by inviting teachers to specify which of six learning types their learning activities would elicit. The six learning types are: learning through Acquisition (ie, to read/watch/listen), Collaboration, Discussion, Investigation, Practice, and Production, from the theory-based Conversational Framework, where each type of learning activity is a cycle between learner and teacher, or learner and peers, at the concept and/or practice level (Laurillard, 2012).
The tool provides feedback on their design as a pie-chart showing the relative proportions of these activities, prompting reflection and re-design of their pedagogy.
Teachers can build on each other's ideas by browsing the collection of designs submitted to the Learning Designer website. All elements representing the design-topic, level, outcomes, activities, group size, duration, resources, and distribution of activities-can be edited by other users as their own version, with versions tracked by the software. The criterion for alignment between outcomes, activities and assessment is tested through peer review, as we discuss below.
The Learning Designer went through further microcycles of user testing workshops and refinement to produce the current implementation (see Figure 2) for the large-scale evaluation reported here.

Evaluation phase
In order to scale up the usage of the tool we ran an "International Learning Design Challenge" (ILDC), in the form of an open online five-day course, using the Blackboard CourseSites platform. The course asked the participants to post their personal design challenges, create and publicly share a learning design, conduct a peer review of another shared design and reflect on their own design in the light of peer reviews. We also embedded the use of the tool as an optional activity in a Coursera-based MOOC on "ICT in Primary Education" (ICTPEd).
These online courses provided large-scale data for testing both the value of the tool for teachers' everyday use, and the viability of a long-term learning design community to build pedagogic knowledge. The impact of the Learning Designer tool was evaluated in two ways: via quantitative data from surveys and platform analytics, and qualitative data from participants' responses in surveys, interviews, forum posts, and blogs. Surveys were anonymous and conducted before the beginning and at the end of the courses and no grading or certification was associated with either course. However, there were slight differences in the survey instruments and type of data collected due to characteristics of the different platforms and course designs. Descriptive statistics were used to analyse the quantitative data, while (anonymised) qualitative data in blogs, interviews, forum posts, and open responses to the survey were analysed using thematic analysis (Braun & Clarke, 2006). The results are reported in the next section.

Results and Analysis
Over 300 participants from 29 countries registered for the ILDC. 133 completed the pre-course survey, and 55 the post-course survey. There were 28 participants who created 84 blog posts about the tool, which provided further qualitative data.
Of the 1162 participants who visited the ICTPEd course in the week featuring the tool, there were 508 first time visitors to view the Learning Designer, of whom 36% returned at least once. The optional activity was the focus of 166 of 177 post-course survey responses, and there were 68 open comments posted in the related forum, providing further qualitative data for analysis.
Inductive thematic analysis of the qualitative data (the open-ended ILDC survey questions, ILDC blogs and ICTPEd discussion forum comments) was conducted to identify patterns and yielded three broad themes: (i) the perceived value of the Learning Designer tool for design support, particularly in expressing and reflecting on teachers' pedagogic knowledge; (ii) evidence of community building around the tool; (iii) participants' evaluation of the process of peer review. The three themes facilitate an evaluation of the extent to which the tool supports the building of teacher community knowledge. They also inform the analysis of the quantitative data (the post ILDC survey data and the post ICTPEd survey data). The findings of both data sets are discussed below.
The value of the Learning Designer tool for design support Analysis of the quantitative survey data from both ILDC and ICTPEd events indicated that a very high proportion of both sets of participants responded favourably to the tool. A total of 95% of ILDC survey respondents indicated (highly or fairly applicable) that they could see benefits of using the tool for planning, with 78% intending to use the tool (Figure 3). The statement rated as "highly applicable" by the largest proportion of respondents was the value of sharing designs, indicating a strong desire to be part of a professional collaborative community.
The post-course survey for the ICTPEd MOOC asked participants to evaluate the general usefulness of the Learning Designer tool on a 1 to 5 scale of "poor" to "excellent". Figure 4 shows that 90% were positive, and 72% rated it very good or excellent.
Participants in both the ILDC and ICTPEd made a range of comments on the overall value of the Learning Designer tool, including its helpfulness, ease of use, and flexibility, describing their positive engagement with the tool with terms such as "amazing," "enjoyable," "interesting," "fascinating," "excellent," for example. Of the 68 comments on the tool posted in the ICTPEd forum only one was negative-raising the issue of teachers' limited time for learning design.
The ILDC blogs identified features of the Learning Designer that participants found most useful, for example, the pie chart "to have a quick overview" (B2, 5 & 22), the export to Word function since it provided "a very readable MS Word document" (B5), and the "very clear pedagogical framework" (B17), which encouraged "a structured approach to the preparation of lessons/modules" (B5).
ILDC and ICTPEd participants both emphasised the value of the tool in supporting their pedagogy during the design process, and for reflecting on the pedagogy of existing learning designs or lesson plans: A very intuitive tool that helped me a great deal to understand the pedagogy of how learning activities are structured. (FP 19) It is a very useful tool as it allows you to organize and structure your lessons as you can adapt them to your students' needs. It also enables teachers to understand how to structure the activities. (FP 53) ILDC bloggers commented that it helped them gain a perspective "of the whole learning path" (B3), and gave the lesson "a face, a visualisation", promoting a "balance between practice and other lesson activities" (B4). Participants found value in the way the tool encouraged attention to the timing of Teaching and Learning Activities (TLAs): "clearly some sort of indication is good for the students AND the designer-if only to get a sense of whether you have provided enough material" (B19).
In addition, the tool was valued for its capacity to provide new insights into teaching, to enable critical reflection on elements of existing learning designs and to foster better pedagogy: The tool allowed [us] to re-think about the structure of activities already undertaken. (ILDC survey 31) It also helped me to reflect about the planned teaching and learning activities for me to determine whether or not my design is supporting the type of learning experience I have in mind. (FP 16) This is a very good tool for all the teachers to use and share their work with their colleagues, as well as getting feedback to help us-as teachers-improve or modify our plans. (FP 43) The fact that the tool was perceived by the participants to have value in supporting teachers to express and reflect on their pedagogy is an important first step towards building a teacher community around the tool, which the next section will explore.

Evidence of community knowledge building
We evaluated the viability of building a community of teacher designers, using both quantitative and qualitative data. The Learning Designer website tracked use of the tool. These data indicate that the tool is sufficiently robust to support a large-scale community, and that there is indeed evidence of a community that is growing around its use.
The data show that use of the tool is increasing, as demonstrated by the following impact measures recorded over its first year of operation: • There have been 460 000 unique visitors to the Learning Designer site since 2014, and 172 000 return visitors • In 2017, 37% of visits were returners, and 15% of returning visitors come back 10 or more times; • Page views have built up from 41 000 in 2014 to 150 000 in 2017; • On average there are 12 100 unique visits per month; • Teachers have been engaged in co-developing a library of hundreds of learning designs including >590 public designs, showing that teachers are willing to share their learning designs with their peers.
In addition, the tool has proved to be robust enough to cope with >4000 simultaneous page views and >700 simultaneous visitors. The usage data demonstrates that a large-scale online community of teacher designers willing to develop and exchange structured learning designs is viable. There were 48 reviews of learning designs completed in ILDC, showing that peer review is feasible for practising teachers. With further development and promotion, it should be feasible to expand to a large international community, across all sectors, in the longer term.
The qualitative data sets also indicated a widespread enthusiasm among ILDC and ICTPEd participants for taking part in such a community of teacher designers. Bloggers during the ILDC posted that they would like more opportunity to collaborate and communicate with others during such events, appreciating the opportunity such exchanges provided: to look into the work of others and to get the views of others on our work. Thanks! (B5) As a new designer … I will certainly print and save many of these reviews to use as a resource. (B10) Some bloggers used the ILDC to communicate with other participants independently, collaborating on Google Docs, holding their own Google Hangouts, and offering email contact. The capacity of the Learning Designer to share designs was frequently considered an important feature to sustain the community: I like that is so easy to see each other's designs, that there are already some great examples and that is so easy to share it. (B6) I can see using this Learning Designer consistently. It is a great tool for teachers to share. I plan to use it this summer and share this site with colleagues. (FP 47) For me it's another new way of sharing ideas with other colleagues how to use ICT in the primary classroom. (FP 61) The value placed on undertaking and receiving peer reviews from other teacher-designers provides a further indication of collaborative knowledge building and will be presented next. The process of peer review Figure 5 shows that ILDC participants rated the process of providing a peer review as highly as creating their own. The discrepancy between doing and receiving a review is mirrored in survey data from the ICTPEd course (Laurillard, 2014), recorded as 85% and 78%, respectively. It is encouraging to see the high rating for the peer review activity itself, as this is the hard work that is the essence of genuine knowledge co-construction.
Participants who did receive a review indicated why they valued this kind of collaborative experience highly: Many bloggers offered critical reflections on the process of peer review and were able to articulate more fully the value of the activity for their personal development as teachers. Participants reflected that it was a "challenging task," but appreciated the criteria that were provided for undertaking the review process. Several reflected on how they had learnt from the criteria about what constituted a good learning design: So in reviewing a design I would be looking for how the student is explicitly supported through teacher engagement in teaching and learning activities (TLA) and in assessment, how students are guided and encouraged to think critically about what they are learning and how they are learning it, and how students are encouraged and supported to engage with each other. (B24) …really like doing review with predefined categories/issues I should take into consideration, because when doing a review I memorize those rules just by the way and I work with them more deeply. (B6) I like also the process of giving feedback … I've learnt a lot! (ILDC survey 16) This indicates that the process of putting the review criteria into practice was considered an important learning experience itself.
Participants took the review process very seriously, as shown by concerns that they were not sufficiently experienced to critique another's work. In order to conduct a review, participants had to "switch my perspective from teacher to student" (B13) to evaluate the experience of the design. The process was considered valuable, since it encouraged them to think about teaching in a different way, because they identified both with the prospective learners as well as tutors in different contexts and disciplines: Some reviewers used both the blog and other forms of communication to maintain a dialogue with the designer whom they had reviewed. Within these exchanges, the reviewers' attention to the details of the design were greatly appreciated, and the designer's positive responses to their suggestions affirmed their professional credibility and built up their self-esteem: I received an email from the designer who had taken on board my suggestions and made some changes to the original submission. I felt I had done a good job and this was a worthwhile aspect of this challenge.
Thus the peer review cycle is completed, and we can see how the teaching community could indeed be a scholarly professional community able to build "technological pedagogic content knowledge" (Bower, Highfield, Furney, & Mowbray, 2013).
The advanced prototype has little functionality to support peer review within the tool itself. For the ILDC we asked participants simply to add notes to a design in the Learning Designer itself, and in the ICTPEd, the peer review functionality was provided by the MOOC platform. Participants commented that they would like to communicate more seamlessly within the Learning Designer itself, and to be able to collaborate with others on a design in the tool. The research indicates therefore that peer review has great potential but the tool itself is not currently able to support it sufficiently.

Discussion
The evidence demonstrates that a constructionist online learning design tool can support a pedagogical knowledge-building community among practising teachers. The volume of users, the increasing numbers of learning designs and the positive evaluations of the tool indicate that this teacher-designer community is beginning to form at scale. The interactions between the participants on the two online courses we examined can be considered evidence of knowledge co-construction. Teachers used their participation in the online courses to reconsider their approach, incorporating their peers' expert insights into their learning design. This scholarly driven iterative pedagogical re-design is key to transforming teaching with technology (Kirkwood & Price, 2013), giving teachers the means to act as design scientists (Laurillard, 2012). (2011) proposed a "crowd-sourcing" approach to jointly constructed knowledge products, emulating the modes of professions such as health care, where nurses and doctors undertake "repeated small tests of small changes-required to refine and adapt a treatment to work effectively in multiple settings" (p. 6). For these professions, three significant features make this possible: shared problems across the system, small tests of small changes, and multiple sources of innovation. Teachers could do the same, but in education they would be refining and adapting learning designs to work in multiple settings. Our project succeeded in building a digital tool to support such a process, and demonstrated its feasibility and acceptability to hundreds of teachers, in schools, colleges and universities, across the world.

Morris and Hiebert
However, the tool is not complete. Our road map for the next phase of development will therefore respond to user comments and help to develop features that could more effectively support a crowd sourcing approach to jointly constructed knowledge projects. The following features are user-generated priorities for development: • capacity to support peer review • capacity to collaborate on developing designs • integration with Virtual Learning Environments.
This approach promises an online, international system that draws on the insights and experiments of teachers testing innovations in multiple local contexts-thereby ensuring their generalised effectiveness. The tool alone is not enough; it must be part of a larger system, but as Ardichvili (2008) observed, the technology can exert influence on the character of the community formed around it. To grow further, the Learning Designer community needs to engage actively in design science by conducting multiple small tests of teaching ideas, using the tool to British Journal of Educational Technology Vol 49 No 6 2018 represent those ideas as learning designs, and drawing on the community of peers to review the designs. The next step is for teachers to evaluate the implementation of their own and each other's designs and embed the results in the learning design in the tool itself. This would fulfil the vision of teaching as a design science: a community of teacher-researchers collaborating to review and test designs, share findings and build upon their collective knowledge of teaching.

Conclusions
This project set out to design an online tool that could support the co-construction of community knowledge about learning design. The Conversational Framework (Laurillard, 2012) was embedded into the Learning Designer tool to scaffold teachers' pedagogy by inviting them to represent their teaching ideas in six learning types, and providing feedback on the resulting balance of their design. The tool also supported the sharing and peer review of learning designs when embedded in an organised course, and thus the creation of an online teacher-designer community. Teacher evaluations indicated that they were prepared to use the tool and valued the way it helped them reflect on their pedagogy.
We are on the way to creating a tool to reconceptualise teaching as a design science. In parallel with developing the functionality of the tool, we have continued to embed it in MOOCs for teacher professional development, as a way of contributing to the truly massive challenges of global education. In this enterprise, we have been joined by other teacher educators who have embedded the tool in their own courses (including MOOCs). Courses such as these provide access to an international community regularly using the Learning Designer to plan and share ideas for teaching. An important next step is to investigate the impact of this process on the learners.