Online learning applied to a course on rational therapeutics: an international comparison between final year students of two medical schools



Dr Robert Likic, MD, PhD, Unit of Clinical Pharmacology, Department of Internal Medicine, University Hospital Rebro, University of Zagreb School of Medicine, Kispaticeva 12, 10000 Zagreb, Croatia.

Tel.: 385 (0) 1238 8288

Fax: 385 (0) 1242 1875

Mobile: 385 (0)9 8183 6916




Poor prescribing is probably the most common cause of preventable medication errors and many of these events involve junior doctors. In 2009, an electronic problem-based therapeutics course developed at the University of Michigan Medical School (UMMS) was translated and adapted for use at the University of Zagreb Medical School (UZMS).


After students from both schools took the course in 2010, we compared their responses with an online questionnaire addressing the course quality and its effectiveness.


There were no statistically significant differences in the overall average grades awarded for the course (UZMS 4.11 ± 0.86 vs. UMMS 3.96 ± 0.93; 95% CI mean difference (MD) – 0.36, 0.07; P = 0.175) with both student groups expressing high satisfaction rates with its quality, accessibility and overall design. UZMS students reported spending less time working through the course than their American colleagues (2.14 ± 1.01 vs. 2.89 ± 1.02 on a five point Likert scale; 95% CI MD 0.51, 0.99; P < 0.05). Furthermore, Croatian students indicated greater difficulty with course materials (3.54 ± 0.59 vs. 3.25 ± 0.59; 95% CI MD – 0.42, – 0.15; P < 0,05) and weekly multiple choice questions (3.83 ± 0.62 vs. 3.4 ± 0.61; 95% CI MD – 0.58, – 0.29; P < 0,05) compared with the UMMS students.


It is possible to adapt and translate successfully whole online teaching resources and implement them internationally in different countries and health care systems, achieving similar, high student satisfaction rates while decreasing administrative and cost burdens. Web based learning may have great potential to offer a cost effective and safe environment in which prescribing skills can be improved.

What is Already Known about this Subject

  • Web based learning is unrestricted by the available teaching space, staff, number of students or printed resources.
  • After the development costs, it can be very cost effective to deploy and maintain in the long run.
  • The aim of this study was to assess the feasibility and effects of adapting and international use of an electronic learning resource on rational prescribing.

What This Study Adds

  • Shared, therapeutics related, web based teaching resources may provide a cost effective means and a safe environment where students' therapeutics skills can be improved upon.
  • Shared online learning resources can be attractive for institutions with constrained budgets, low numbers of skilled faculty members and available teaching space.
  • Academic partnerships in shared or joint development of online teaching resources in prescribing may become more common in the near future.


Poor prescribing is probably the most common cause of preventable medication errors in hospitals. Many of these events involve junior doctors who have just graduated. Prescribing is a complex and challenging task that requires diagnostic skills, knowledge of medicines, communication skills, understanding of the principles of clinical pharmacology, consideration of risk and uncertainty and, ideally, experience [1]. In a recent study that evaluated preparedness for practice of new doctors graduating from three British medical schools, the authors revealed that more than 80% of the new graduates from two schools failed a written prescribing assessment [2]. Similar concerns are present elsewhere throughout Europe and around the world [3-5].

In 2007, the University of Michigan Medical School (UMMS) developed an electronic learning, problem-based course in ‘Advanced Medical Therapeutics’ (AMT). The course was designed around 90 clinical cases that centred on therapeutics problems. The cases were divided into specialty modules and distributed to final year students via a locally developed learning management system (LMS) called Ctools. In a collaboration between the UMMS course developers and a faculty member from UZMS (RL), this resource was translated into the Croatian language and adapted for use at the University of Zagreb Medical School (UZMS). This academic collaboration between the two institutions occurred under the Open Michigan Initiative, a long term project in which the UMMS plans to release all of its electronic learning resources as open resources using Creative Commons licensing.

After both student groups took the course in 2010, we assessed the students' perceptions of course quality and effectiveness using an online questionnaire and we assessed student performance in the course by comparison of their formative assessment results. Our intent was to assess the feasibility of adapting and deploying such a web-based learning resource globally.


The web-based course on prescribing

AMT was designed in hypertext markup language (HTML) to be a multidisciplinary, compulsory, fully online, problem-based, interactive electronic resource, comprising 90 clinical cases arranged in 20 modules (clinical pharmacology, cardiology, gastroenterology, pulmology, transfusion medicine, ear, nose and throat, obstetrics and gynaecology, geriatrics, infectious diseases, psychiatry, rheumatology, neurology, diabetes, electrolyte disorders, prescription writing, drug advertising, analgesia). Groups of modules were distributed to students on a weekly basis over a period of 1 month via the UMMS LMS, Ctools (Table 1). All clinical cases included: patient's history, physical examination and laboratory workup details, as well as diagnostic imaging (X-ray, CT, MR, pathohistology studies) where appropriate. Each case included student decision points focusing on a particular therapeutics problem presented as multiple choice questions (MCQ) with immediate feedback. Most decisions also included a clinician's video commentary in FLASH format, in which content experts explained the underlying concept and typically emphasized the therapeutics issues students were meant to master. The course made extensive use of hypertext, so that every MCQ answer linked to a separate web page with feedback regarding that option as well as links to selected, relevant learning materials (Powerpoint lectures, guidelines, external websites, articles) provided to students for further reading and use. In all, the course comprised 300 distinctive FLASH commentaries totaling around 6 h of video and around 1000 web pages (clinical cases, MCQ answer feedback, further resources). A demo version of the course is available for online access at:

Table 1. Layout of the electronic course over a 4 week period
WeekModuleCasesWeekly quiz questions
Week 1Drug developmentArticle2
Ear, nose, throat32
Infectious diseases 163
Week 2Geriatrics22
Infectious diseases 274
Clinical pharmacology31
Prescription writingArticle0
Transfusion medicine72
Week 3Diabetes52
Drug advertisingArticle1
Week 4Cardiology45
Electrolytes disorders71
Obstetrics and gynaecology93

In order to pass the course successfully, students from both medical schools were required to study through the weekly, online course materials, to pass the online quizzes that focused on the therapeutics problems covered during the respective week, and finally to submit to the course coordinators through Ctools an essay on a freely chosen therapeutics issue of their own interest.

Formative assessment

In order to ensure students went through the online clinical cases, a formative assessment consisting on average of five online quizzes per week was included in the course. Questionmark's Perception platform was used for distribution of quizzes among students and assessment of their scores. In all, there were 22 weekly quizzes which were distributed on a weekly basis in a timed manner (20 min were allowed per quiz question). Students took them in an open book format and could access them within the week they were offered whenever it was convenient to them. Students' Perception login details were kept distinctively different from their Ctools user details. Hence every student received a separate username and password for access to the online quiz system. A designated team of administrators and instructors dealt with students' access and quiz problems on an individual basis.

Course quality survey

At the end of the course at both institutions, we administered an online survey (see Appendix) with five-point Likert scale questions on course quality using, a commercial online survey service. The questionnaire has been in use for several years at the UMMS for the purposes of evaluating web-based teaching formats. Response rates of the UZMS and UMMS students were 86% and 66% respectively.


Final year students of medicine at both institutions took this compulsory course 1 month before graduating, 232 at the UZMS in 2010 and 161 at the UMMS. Since this was the first time UZMS students experienced a completely web-based format of teaching and learning assessment, 60 min tutorials were held for groups of up to 20 students during the first 2 days of the course. The purpose of the tutorial was to demonstrate logging into Ctools, browsing the AMT materials, and accessing and completing the Perception quizzes.


Descriptive statistics and independent-sample t-tests were used to compare the feedback obtained from the two student groups.

Course translation and adoption for use at the UZMS

The course was translated into the Croatian language (RL), while the two other UZMS faculty members (LB and IF) proofread the course materials and compared them with the original, corrected any mistakes that they encountered and introduced additional amendments where needed. HTML pages were translated with Adobe Dreamweaver CS4, clinicians' video commentaries were first transcribed and subsequently dubbed in Croatian with either AVS video editor or Audacity software, a new audio track was then re-encoded with the video file through AVS video editor. Weekly online quizzes were also first translated into Croatian in MS Office Word and then reloaded into Questionmark Perception by use of the Perception software's authoring tools.

Ethics Committee approval was deemed unnecessary since similar research was regularly being undertaken at both medical schools, no personal information was collected, students responded on a voluntary basis and their responses were kept on a secure server.

The cost estimate of the project was 10 000 US$.


Course grades

There was no statistically significant difference in the average final course grades given to Michigan and Zagreb students (UZMS 4.11 ± 0.86 vs. UMMS 3.96 ± 0.93; 95% CI MD –0.36, 0.07; P = 0.175). Student cohorts at UZMS and UMMS both evaluated the online AMT course favourably and results on many items were similar for the two institutions (Table 2).

Table 2. Comparison of USA and Croatian student evaluations of M4 advanced medical therapeutics course, 2009–10Thumbnail image of

Satisfaction with teaching methods

Statistically indistinguishable mean responses greater than 4 on a five point scale indicated that both student groups felt they understood the expectations of the course directors. Similar outcomes for other survey items indicated that learning outcomes were judged to be clear and the learning technology in this computer based course worked well. The students also provided favourable evaluation of the ease of navigation and ease of access to the online resources. There was also no difference among students with regards to the perceived quality of the overall design of the course. The three key components (online cases, quizzes and student individual projects) were thought to fit together well to ensure learning.

Time spent inside the course

An aspect of this teaching approach that was highly appreciated by the students of both groups was the flexibility it allowed concerning their curricular and extracurricular commitments. Students had unrestricted distant online access, thus permitting better planning and use of students' time. Another highly appreciated feature was the abundance of time to solve typically difficult quiz problems (20 min per item) in an otherwise open book environment (Question 13, Table 2), which fits well with the instructional intent of encouraging the students to think on a complex level about the topics covered in the course.

Differences between the student groups

UZMS students reported spending less time each week working on the course in comparison with their American colleagues (2.14 ± 1,01 vs. 2.89 ± 1,02 on a five point scale where 1 = 0 to 4 h and 5 = 20+ h; 95% CI MD 0.51, 0.99; P < 0.05). Additionally, Croatian students found more difficulty with the course materials (3.54 ± 0.59 vs. 3.25 ± 0.59; 95% CI MD – 0.42, – 0.15; P < 0.05) and also the weekly multiple choice questions (3.83 ± 0.62 vs. 3.4 ± 0.61; 95% CI MD – 0.58, – 0.29; P < 0.05) when compared with the UMMS students. They also felt more challenged by the course contents than the American students (3.47 ± 0.93 vs. 4.14 ± 0.75; 95% CI MD 0.48, 0.87; P < 0.05).

Opinions on the formative assessments

Finally, the two groups differed on the issue of weekly online quizzes.UMMS students were less convinced that the quizzes were based on material delivered by the course (3.29 ± 0.98 vs. 3.55 ± 0.77; 95% CI MD – 0.48, – 0.05; P < 0.05) than the UZMS students. However, UMMS students appreciated the immediate written feedback on their submitted quiz answers more than their Croatian colleagues (4.68 ± 0.51 vs. 4.17 ± 0.85; 95% CI MD 0.35, 0.66; P < 0.05).


During the past 10 years, the University of Zagreb Medical School has worked actively on transforming and adapting its curriculum to address better the expectations that Croatian society has been imposing on young physicians [6]. Doctors' competency with regards to clinical skills and therapeutic prescribing has been identified as priority. In that context, three 1 month long problem-based courses (PBL), one of which focused on rational prescribing of medicines, were introduced into the curriculum of the final year of medical school [7]. Research done later into the effectiveness of the PBL teaching approach demonstrated high student and teacher satisfaction with this educational method. However when students' knowledge was compared with that of their peers who underwent the same course, albeit without PBL elements, no significant difference in knowledge gain was demonstrated [8]. Considering the requirement to prepare teaching staff to function as tutors with small student groups, the need for administrative support and the additional teaching space and materials, PBL was not deemed cost effective, at least for teaching rational therapeutic prescribing during the final year. A potential solution to this problem presented itself through sharing of an online course on advanced medical therapeutics that was developed at the UMMS. Consequently, our aim was to increase our students' capacity for safe and rational prescribing while simultaneously permitting them greater autonomy and increased flexibility in time utilization.

Even though UZMS students took part in a completely web based course for the first time during their medical education, with the same stakes as their American colleagues, feedback obtained through the online questionnaire and email correspondence with the course instructors, demonstrated how quickly and easily they adapted and embraced this teaching approach. Since this was the last course they took before graduating, its flexibility as well as the possibility for remote access, was widely praised. We believe the comparatively similar grades awarded to the course by both students groups could be attributed to the high enthusiasm for this new teaching approach exhibited by the UZMS group from the onset, even though they eventually found the course materials and formative assessments more challenging. It is worth noting though that the UZMS students reported spending significantly less time weekly working on the course materials in comparison with their American colleagues. Since UMMS students have to pay tuition fees for their medical education, this could perhaps explain the difference in invested study times. Although every effort was made to translate into Croatian the clinical cases (90), specialists' video commentaries of the clinical problems (300) as well as formative assessment quizzes (24), supporting materials such as presentations in Powerpoint, review or supporting articles and practice guidelines were kept in the English language. While the great majority of UZMS students understand and use the English language, this factor may have contributed to their perception of difficulty of the course materials as well.

This study also has several weaknesses. Firstly, part of the course that was considered most challenging by students were the weekly online quizzes, even though these were offered in an open book format, on remote PCs, in an unsupervised fashion. Students at both centres were bound by their honour codes in order to prevent copying of quiz answers from their colleagues. However, since we did not control for possible cheating during the weekly quizzes, comparison of these results between the student groups was not performed. Secondly, although we could have used login details data that were tracked/logged by the Ctools LMS to exactly compare the time duration students of both medical schools spent inside the course or accessing certain course materials, this was unfortunately not possible, since at the time this study was done, the version of Ctools (unlike some other LMSs) that was used was missing a module/plugin that would permit the calculation of the duration of time spent inside the platform on accessing certain items. Hence we compared students' self reported time spent inside the LMS obtained via the online questionnaire. Finally, the different electronic course evaluation questionnaire response rates are consequence of practice at both medical schools where students routinely file in teaching evaluation questionnaires on a voluntary basis. Despite potential bias, we believe that a response rate of more than 65% at both institutions still allows for a meaningful comparison.

From the instructors' perspective, web-based learning enables teaching of students at different locations, such as various practice sites in the same city, different cities and even, such as in our case, different countries [9-12]. The resulting experience for the learners is that they are able to participate in exactly the same instructional activities. This feature becomes very important when there is varying expertise or lack of a skilled teaching faculty [13]. Furthermore, electronic learning is based on the premise of individualized adult learning, allowing students to solve clinical problems independently and to move at their own pace, just as they will eventually do during their future clinical work. It is therefore reasonable to suggest that this approach may prepare students better to become more independent and confident in their therapeutics prescribing skills [14]. Additionally, once the electronic resource is constructed, it can be shared among collaborating institutions, thus avoiding redundancy and development costs. The number of participants is then unrestricted by the availability of dedicated space, supporting materials or teaching staff, but may be constrained by server capacity, bandwidth and students' ownership or access to personal computers [15, 16]. If instead of developing effective online tutorials or virtual patients, instructors choose to exploit only the electronic resource distribution by publishing their texts, existing books or syllabus material online as a web-based course, they will miss the benefits of the interactive teaching design altogether [9, 17-19].

In 2007 a poll of US academic leaders stated that students generally appear to be at least as satisfied with their online classes as they are with traditional ones [20]. Our experience confirms this and demonstrates that it is possible to adapt and translate successfully an entire web-based teaching resource. With some effort, such resources can be implemented internationally and to different health care systems, achieving similarly high student satisfaction rates while decreasing administrative and cost burdens. E-learning is likely to transform into ‘U-learning’, as ubiquitous portable and fixed Web access devices appear and become common place over the next few years [21]. Initiatives like the ‘Open Michigan’ with which the UMMS strives to make their electronic learning resources available internationally under a Creative Commons licence will become even more important for the improvement of medical teaching throughout the world [22].

In conclusion, for many prescribers of medicines, mistakes proved to be their best teachers. Web-based, interactive learning may have great potential to offer a cost effective and safe environment in which prescribing skills can be sharpened without the painful lesson of a treatment mistake followed by unwanted consequences for the patient.

Competing Interests

We declare no competing interests.

Course contributing faculty Peter Hagen, MD, John Nicklas, MD, Ragavendra Baliga, MD (OSU), Alan Weder, MD, Robert Lash, MD, Ram Menon, MD, Julie Surhigh, MD, Joseph Messana, MD, Timothy Nostrant, MD, Karen E. Hall, MD, Ph.D, Tami Remington, MD, Carol Chenoweth, MD, M.S., James Riddell IV, MD, Cary Engleberg, MD, Steven Leber, MD, Margaret Gnegy, PhD, Mark Chames, MD, Susan Urba, MD, Wendel Weber, PhD, Jimmy Rae, PhD, David Stutz, MD, Michael Jibson, MD, Thomas Schwenk, MD, Margaret Gnegy, PhD, Cyril Grum, MD, Laura Cooling, MD, M.S. and Scott Gitlin, MD

Robert Likic received a Josip Matovinovic fellowship in 2009 which allowed him to spend 6 months in two visits working as a visiting scholar at the University of Michigan Medical School.



M4 Medical Therapeutics Elective Instructions: Please provide your assessments below by clicking on the appropriate circle. Please complete all questions in this section of the evaluation. You must press the ‘Submit Evaluation’ button at the end of this block of questions in order for your responses to be saved.

In COMMENTS boxes, you may provide up to 32 lines of typed text.

  1. Approximately how many hours per week did you spend working on this course?

    ○ 0–4

    ○ 5–9

    ○ 10–14

    ○ 15–19

    ○ 20+

  2. Course learning outcomes were clear.

    ○ Strongly Disagree ○ Disagree ○ Neutral ○ Agree ○ Strongly Agree

  3. I understood what was expected of me in the course.

    ○ Strongly Disagree ○ Disagree ○ Neutral ○ Agree ○ Strongly Agree

  4. The technology in this computer-based course worked well.

    ○ Strongly Disagree ○ Disagree ○ Neutral ○ Agree ○ Strongly Agree

  5. All of the on-line course material was easy to access and navigate.

    ○ Strongly Disagree ○ Disagree ○ Neutral ○ Agree ○ Strongly Agree

  6. The course content challenged me at an appropriate level.

    ○ Strongly Disagree ○ Disagree ○ Neutral ○ Agree ○ Strongly Agree

  7. The three major components of the course (weekly online topics and resources, 3 symposia, project) fit together well to assure learning.

    ○ Strongly Disagree ○ Disagree ○ Neutral ○ Agree ○ Strongly Agree

  8. If you answered Strongly Disagree or Disagree above, please explain:
  9. The course met my own learning expectations.

    ○ Strongly Disagree ○ Disagree ○ Neutral ○ Agree ○ Strongly Agree

  10. Course material was …

    ○ Too Easy ○ Somewhat Easy ○ Just Right ○ Somewhat Difficult ○ Too Difficult

  11. Weekly multiple choice questions were …

    ○ Too Easy ○ Somewhat Easy ○ Just Right ○ Somewhat Difficult ○ Too Difficult

  12. Quizzes were based on material delivered by this course.

    ○ Strongly Disagree ○ Disagree ○ Neutral ○ Agree ○ Strongly Agree

  13. It was useful to have immediate feedback on the quizzes.

    ○ Strongly Disagree ○ Disagree ○ Neutral ○ Agree ○ Strongly Agree

  14. I had sufficient time to finish the quizzes.

    ○ Strongly Disagree ○ Disagree ○ Neutral ○ Agree ○ Strongly Agree

  15. As part of the weekly quizzes, describing my thought processes was an intellectually challenging exercise. ○ Strongly Disagree ○ Disagree ○ Neutral ○ Agree ○ Strongly Agree
  16. Please provide any additional comments about the quizzes.
  17. Expectations for the final project were …

    ○ Too Easy ○ Somewhat Easy ○ Just Right ○ Somewhat Difficult ○ Too Difficult

  18. Overall, this course challenged me to think at a complex level about the topics covered.

    ○ Strongly Disagree ○ Disagree ○ Neutral ○ Agree ○ Strongly Agree

  19. Overall, the quality of this course was … ○ Poor ○ Fair ○ Good ○ Very Good ○ Excellent
  20. The flexibility in the course provided by computer-based distance learning was of great assistance in scheduling my resident interviews.

    ○ Strongly Disagree ○ Disagree ○ Neutral ○ Agree ○ Strongly Agree

  21. What worked well in this course?
  22. What did not work well in this course?
  23. How can this course be improved?