Emergency medical residents' use of Google® for answering clinical questions in the emergency room
Emergency Department (ED) residents perceive Internet resources as reliable and regularly search the Internet to answer clinical questions in the ED. To determine the validity of the Internet and the Google® Search Engine, as a source for accurate information to answer clinical questions in ED, a single blinded prospective study was conducted. The search strategies used by Emergency Medical (EM) Residents and Internet resources most frequently visited to find answers to clinical questions were reviewed.
Residents completed a PreTest consisting of questions that might routinely occur in ED, and were instructed to answer questions without outside resources, and if they were confident the information was suitable for patient care. Questions that EMR were unsure of, or had answered incorrectly on the PreTest, were used to create a Google®Test. On the pretest 32% of the questions that residents answered were correct, while 28% were incorrect. Residents were unsure of 40% of the questions that they responded to. On the Google®Test 59% of residents' responses were correct, 33% were incorrect and residents were unsure of 8%. Percent of unsure answers dropped significantly when residents searched Google®.
Results indicated EM Residents are naive searchers who use single word or short phrase searches and derive query terms directly from the questions; advanced search features of Google® were not used; and websites frequently visited were designed for laypersons, sites for publishers, medical journals, and Google®Scholar and Google®Books. Results also indicated that while EM Residents often retrieved inaccurate information using Google® they believed the information was reliable enough to use in patient care.
Findings have major implications for medical informatics, medical practice, and patient safety. Providing Internet access in ED should be carefully reviewed for reliability as a tool for teaching and clinical decision-making. Also, education of EM Residents should include training on Internet and database searching.
The advent of widespread computing in clinical practice and medical education settings has had a significant impact on how physicians and other healthcare professionals use health information systems to find answers to patient-related questions. While information science research has included research into physician's information behaviors (for example Elayyan, 1988; Gruppen, 1990; Verhoeven, Boerman, and Meyboom-DeJong, 1995; Schwartz, 1995; Mullaly-Quijan, Ward, and Woefle, 1994; Forsythe, Bucahnan, Bankowitz, 1992), much of this research was conducted prior to the Internet and development of Internet search engines (ISEs). Research has focused on the use of specific online databases (MEDLINE) to find information. For example, Haynes, et. al. (1990) conducted extensive assessments of physicians' ability to search the medical literature using Medline. This landmark work found that 47% of the search results had a direct impact on a clinical decision made by a physician. In a related study McKibbon, et.al. (1995) found that experienced physician searchers produced more precise search results than did novice searchers, but the differences were not generally statistically significant. A study by Ely, et. al. (2007) demonstrated that participants were able to answer only 40% of the questions correctly using software that was commercially available at that time. Participants encountered a variety of problems, such as obtaining incomplete information, difficulty using the software, asking the questions in an appropriate manner, and inadequate training. Other research has focused on the types of questions posed during clinical teaching and practice by physicians and medical students (Osheroff, Forsythe, Buchanan, and Bankowitz, 1991; Gorman, Helfand, 1995; and Wildemuth, DeBliek, Friedman, and Miya, 1994). Still other related research to this study includes the work by Bird & Heekin (1994) and Lindberg, Siegel, Rapp, Wallington (1993), who each examined sources used by physicians to answer questions in clinical practice.
As of 2010 Google® is the most widely used Internet search engine (ISE) and is one of the most commonly used sources of clinical information by emergency medicine physicians (Graber, Randles, and Ely, 2008). Google® indexes web pages based on relevance and popularity, yet Google's® search results ranking are not a reflection of the accuracy of the information retrieved. To date there has been little published research on the accuracy of using Google® to search for answers to medical questions. Tang and Ng (2006) studied the use of Google® as a “diagnostic aid”. The authors searched Google® for terms that were selected from published case records, which they designated “diagnostic cases”, and from the first three to five pages of results returned by Google® the authors selected the “three most prominent diagnoses that seemed to fit the symptoms and signs”. If one of the three diagnoses was correct, they determined that the Google® search had provided the correct diagnosis. Tang and Ng concluded that it is often useful to “Google for a diagnosis”, though they acknowledged that there were many limitations to their findings.
In their clinical practice and teaching the two attending physician authors (RK and RM) observed that many of their searches, as well as those of their colleagues and residents, focused on finding information for use in patient care, e.g., drug dosage, drugs of choice for specific indications, characteristics of diagnostic tests, frequency of certain findings in a specific disease, and acceptable treatment alternatives, among others. This differs considerably from the searches performed by Tang and Ng who searched Google to arrive at a diagnosis.
The research team consisted of two Board Certified Emergency Medicine (EM) attending physicians (authors RK and RM), an EM resident (author SH), an information scientist (author JA) and a health sciences librarian (author DS). The two attending physicians developed a 71 question test. The questions were clinically oriented, and referenced answers were prepared using current “gold standard” textbooks using in medical training. Thirty-three EM residents volunteered as subjects. The residents were initially given a questionnaire regarding their personal use of computers and search engines, as well as their level of confidence in finding relevant medical information on the Internet, their confidence in searching the Internet to assist in clinical decision-making, and their confidence in the reliability of the medical information they found on the Internet. The residents were also asked to assess their skills in searching bibliographic databases, and related medical information resources, and how frequently they searched online tools for answers to clinical questions, among other items. The subjects were given the 71 question test and were instructed not to use any reference materials to answer the questions. This was termed the PreTest.
The EM Residents were instructed to answer the questions only if they were confident enough of their answers that they would use the information in a patient care situation. Questions marked unsure or answered incorrectly were used to create a second individualized test for each subject.
On the second test, termed the Google® test, subjects were permitted to use a computer with Google® as the only search engine. Transaction logs of each resident's search activity were recorded using the USAProxy tracking program. The results of the Google® test were scored with percentages of correct, incorrect and unsure answers. The answers were then compared with the resident's level of training, that is, postgraduate year, individual computer experience, and the self-assessed confidence ranking that each resident had previously ascribed to his or her self. After each resident completed the Google®test, the librarian and the information scientist on the team (DS and JA) cleaned the Google® transaction logs and eliminated irrelevant information. The next step in the process, which is currently underway, was to analyze the searches conducted by each EM Residents, and to develop a profile of how each resident searched Google® to answer the questions. The preliminary results and analysis of our findings are reported here.
Thirty-three EM residents completed both tests. The mean score on the PreTest was 32% correct, 28% incorrect and 40% unsure. On the Google®test, the mean score for correct answers was 59% (95% CI 56–62). Thirty-three percent of the answers were incorrect and 8% were unsure. Of the background variables examined, only age, confidence in using a computer and confidence in finding medical information on the Internet were correlated with higher percentages of correct answers on the Google®test.
After cleaning the transaction logs, DS and JA began developing a search profile for each resident. Preliminary results provided the following important findings with regard to the search strategies employed by EM Residents, the choice of websites visited from the selections offered by Google, and the critical thinking processes used by residents to answer the questions.
Resident's search strategies were simple and basic. The residents restated the question as a search string and none of the residents analyzed thus far, used Google's® advanced search capability. Most residents searched only the first or second page of Google® results. Unfortunately the residents were both naive and uninformed searchers and included in their search strings stopwords such as “an”, “as” and “from”.
Web sites visited
Residents visited a wide range of websites including sites containing professionally-oriented materials and sites designed to appeal to the layperson seeking health information online. Among the most frequently visited professionally-oriented websites were Medscape, eMedicine, WebMD, and Accessmedicine. The researchers noted that this outcome was potentially the result of Google's® page ranking infrastructure. Many .com publisher's websites also were visited, such as Elsevier's Science Direct, Springer, Cambridge University Press, and the National Academy Press, to cite a few examples. Residents repeatedly visited the Science Direct website, but did not learn that the site required a paid subscription in order to view fulltext articles. Since there is no evidence that they visited the University Library's website, they remained unaware that the Library had a subscription to Science Direct.
The most frequently visited individual journal websites were New England Journal of Medicine, JAMA, the American College of Surgeons, and the Journal of Trauma. Among the .org websites visited were theheart.org, labtestonline.org, and Bandolier and Pier (evidence-based medicine) websites. Many .gov websites also were visited with the most prominent being the Food and Drug Administration, Centers for Disease Control and Prevention, NASA and the National Library of Medicine's PubMed Central site, presumably seeking a full text article to use in answering a question. The residents also selected with some regularity Google®Scholar and Google®Books. A selection of .edu sites also were visited, such as, poems.msu.edu/InfoMastery, cla.edu/training/suppanswers.pdf, and Hawaii.edu/medicine/pediatrics.
The residents also spent a significant amount of time visiting sites normally geared to the information needs of laypersons including Wikipedia, Google® Maps and Images as well as wrongdiagnosis.com, healthboards.com, druglibrary.com and MedlinePlus. Among the unusual sites visited by residents were geocities.com, lectlaw.com, healthcommunities.com, hypothermia.org, socnetcentral.com, druglibrary.com, thescoop.org, medal.org, gepsoft.com, publicsafety.com, burstnet.com, livingdonorsonline.org, healthboards.com, medical.justanswer.com, healthline.com, and medicalcriteria.com. Occasionally residents also browsed unrelated websites such as, drspock.com, lacellnot.webcristiano.org, parkhurstexchange.com, ptone.com., digg.com., adsrm.looksmart.com., and rcm.amazon.com.
We are currently correlating the number of .com, .org, .edu and .gov websites visited with number of correct, incorrect, and unsure answers in order to understand how the number of correct, incorrect and unsure questions answered were affected by the resident's choice of websites. It is possible that when a resident had a higher number of correct answers that s/he may have probed beyond the first page or two of sites identified by Google®. Some residents also selected more publishers' websites than others, and while these also are .com sites, the selection implies that the resident knew that an answer to a question might be found in a peer-reviewed journal article.
We expected residents to spend approximately five minutes per question. They were instructed to use the time to search Google® to find an answer that met their internal criteria for clinical certainty. The important implication is that residents were overconfident of the information obtained when searching the Internet. In the real world of the ED, residents would have been instructed to stop using Google® and seek an alternate source of information for answering the question. The assessment of confidence presumably stemmed from the perception that the Internet was a reliable source of medical information, and perhaps by their over-estimation of their own search and informatics skills (Scott, Schaad, Mandal, et. al., 2000; Shurtz, 2009). The questions created for the study were based on the kinds of information the two attending physician authors (RK and RM) look for daily in their clinical practice and which they have observed EM residents seeking in the ED. It is clear that our subjects often retrieved inaccurate information using Google® yet the residents believed the information was reliable enough to use in patient care. This may represent a previously unrecognized source of medical error and a threat to patient safety. As Ely, Osheroff, Maviglia, and Rosenbaum (2007) have pointed out, the degree of prior knowledge of a subject may influence search strategies and may also influence the searcher's ability to arrive at an accurate answer. Our research suggests that searchers who scored high on the PreTest also scored high on the Google® Test.
The preliminary findings of this study also imply that EM residents need more effective and comprehensive instruction in learning how to search, and the search training must include how to search the Internet for quality, reliable information as well as how to search bibliographic databases. The EM Residents who participated in this study had not previously received any search training, as indicated by their responses on the demographic questionnaire. However, other researchers (Schwartz & Schwartz, 1995; Allen, Kaufman, Barrett, et. al, 2000; and Ely, Osheroff, Maviglia, and Rosenbaum, 2007), have found that training improved search capabilities. In many respects the outcomes have both favorable and unfavorable consequences. Training and education generally results in improved performance, but these findings also indicate that efforts should also focus on improving the residents' knowledge base. The combined protocol could result in searches that produce a higher percentage of correct answers when using an ISE to answer a clinical question. The results also suggest that an experienced physician may be more capable of finding the correct answer to a question using an ISE because of his/her advanced knowledge-base and experience.
EM residents' ability to identify correct answers to clinical questions using Google® was poor. Of greater concern was that the number of unsure answers decreased while the number of incorrect answers increased when the Internet was used as the only source of information. This occurred despite instructions to only answer if they were clinically certain of the answer. Using Google® appeared to have given the residents a false sense of security. Innovations, such as Internet access in the ED, should be studied carefully before being accepted as a reliable tool for teaching clinical decision making. The study results also bring into question the accuracy of the search results themselves. Even casual Google® users, when searching in an area of personal expertise, will often find erroneous information.
The study also included two additional authors:
Ronald Moscati Erie County Medical Center, University at Buffalo 462 Grider Street 716–898–5925 firstname.lastname@example.org
Shravanti Halpern St. Elizabeth Medical Center 2209 Genesee Street, Utica, NY 13501–5999315–798–8111 email@example.com