No conflicts of interest to declare.
BRIEF REPORT: Development of a Prescription Medication Information Webliography for Consumers
Version of Record online: 25 SEP 2006
Journal of General Internal Medicine
Volume 21, Issue 12, pages 1313–1316, December 2006
How to Cite
Ko, Y., Brown, M., Frost, R. and Woosley, R. L. (2006), BRIEF REPORT: Development of a Prescription Medication Information Webliography for Consumers. Journal of General Internal Medicine, 21: 1313–1316. doi: 10.1111/j.1525-1497.2006.00620.x
An abstract of earlier related research has been presented at the American Pharmacists Association (APhA) Annual Meeting & Exposition, April 4, 2005, Orlando, FL, and was published in the Journal of the American Pharmacists Association 2005;45(2):269.
- Issue online: 25 SEP 2006
- Version of Record online: 25 SEP 2006
- Manuscript received September 1, 2005Initial editorial decision September 22, 2005Final acceptance August 15, 2006
- medication information;
- prescription drug information
BACKGROUND: Websites offering drug information vary in coverage and quality, and most health care consumers are poorly equipped to assess the quality of internet medication information.
OBJECTIVE: To establish a webliography of recommended prescription medication information websites for health care consumers and providers.
DESIGN AND METHODS: Drug information websites were systematically identified based on recommendations from health professionals and text-word searches of MEDLINE and Google. The resulting sample of websites was evaluated in a 2-step process. Candidate websites were first screened using inclusion/exclusion criteria representing minimum information requirements. Websites that passed the inclusion/exclusion criteria were then rated on 16 quality criteria using a 5-point scale by 3 trained judges. Website ratings were averaged, then multiplied by the corresponding importance weight of each criterion and summed to generate a total score. Websites with the highest total scores were included in the webliography.
RESULTS: Ten websites were selected for inclusion in the webliography. The 3 highest-scoring websites were Anthem Blue Cross and Blue Shield (http://home.anthemhealth.com/topic/drugcenter), U.S. National Library of Medicine (http://www.nlm.nih.gov/medlineplus/druginformation.html), and Healthvision (http://www.yourhealthinformation.com/library/healthguide/en-us/drugguide/default.htm).
CONCLUSION: Medication information websites vary widely in quality and content. The online webliography is a valuable and easily accessed tool that can be recommended by health care professionals to patients who request referral to reliable websites.
The internet has become an important source for consumers searching for health information.1 The convenience, privacy, and nearly instant access to innumerable health-related resources are transforming the delivery of health information.2 However, seeking health information on the internet poses risks as well as benefits. The proliferating amount of health information available online is overwhelming, and the variability in quality is problematic.1,3 One area of particular concern is online drug information. Approximately 26% of Americans have used the internet to seek prescription drug information.4 Incomplete, misleading, and inaccurate information on the Web may contribute to harmful outcomes, such as drug misuse and medication errors.5–8
Locating and evaluating medication-related websites is time consuming and requires a certain level of internet search proficiency, health literacy, and knowledge to distinguish good quality information from bad. Most consumers, lacking some portion of such knowledge, would benefit from professional assistance. Many sets of criteria, checklists, and rating instruments have been developed for evaluating internet health information quality,9–12 such as Health On the Net (HONcode) and Utilization Review Accreditation Commission (URAC) accreditation.13,14 While these approaches are useful to assess general health information quality, few tools have been developed to specifically evaluate the quality of online prescription medication information.
The overwhelming volume, technical complexity and uncertain reliability of drug information on the internet can be confusing to patients, and they often ask health professionals for guidance in selecting reliable websites or evaluating internet information.2 A webliography, or annotated list of websites relating to a given subject,2 that is current, developed systematically by professionals, and readily accessible, can help health professionals respond to patients' requests. We applied evaluation criteria developed by pharmacists and health communication specialists to produce an internet-accessible prescription medication information webliography for consumers.
Websites to be considered for the 2-step evaluation process were identified by a search of MEDLINE, an internet search, and recommendations by health professionals. A Google search was performed in April 2006 using the search terms “medicine information,”“medication information,”“drug information,”“pharmaceutical information,” and “health information” individually and in combination with “webography” and “webliography.” The same search terms, with the addition of the terms “internet” and “online,” were searched as text words in MEDLINE using the PubMed interface in May 2006. Websites mentioned in published articles obtained from MEDLINE were visited to ascertain whether they were still operating. Additional sites were recommended by a convenience sample of 6 health professionals knowledgeable about medication safety. All websites that provided prescription medication information in English were included. To avoid unbalanced information, pharmaceutical company-maintained websites were excluded. Because patients are targeted as end-users of the webliography, journal websites were excluded due to their professional-oriented content. In addition, to be selected for evaluation, sites were required to provide free access, require no user registration, and not be limited to drugs of one class or for a specific disease.
Identified websites were evaluated against 15 screening criteria (including 9 criteria used to select potential websites) and 16 quality criteria. All items in the 2 sets of criteria were chosen or derived from existing standards and guidelines obtained from 17 published or internet sources. A panel composed of 1 primary care physician, 3 pharmacists, and 2 health communication experts reviewed both sets of criteria. To pass the screening criteria and be selected for quality assessment, potential sites were required to provide information about drug identification information (including both generic and trade names), therapeutic indications, and possible undesirable effects. In addition, sites were required to have no endorsement of particular products, and clearly differentiate advertising from information content. Additional requirements included contact information, date of last update, and disclosure of authorship.
Websites that passed the screening were evaluated for quality using a method adapted from an established decision-making process.15 The 16 quality criteria were weighted according to relative importance as determined by the investigators (see Table 1). Weights were assigned on a scale of 1 (least important) to 3 (most important).
|Drug information quality|
|Assurance of information quality||Site has a written policy or procedure that states: Information provided has been subjected to review and the process/individuals involved have been described Claims relating to the benefit/performance of a specific treatment, commercial product, or service are justified based on appropriate and balanced evidence||3|
|Attribution||Citations for original sources of evidence are provided Links to these referenced citations are available Copyright statement is provided||1|
|Objectivity||Site contains information on other medications or nonmedication treatments for the same condition||2|
|Currency||Drug information is updated within last 6 mo||1|
|Feedback||A feedback mechanism is available for users to offer comments, corrections, and criticisms||1|
|HONcode||Subscribes to the HONcode principles||2|
|URAC accreditation||Displays URAC health website accreditation seal||1|
|Drug information content and design|
|Comprehensiveness||Includes: Instructions for proper use of the drug Therapeutic category Information about the condition that the medication is used to treat or a link to a website with that information Contraindications Drug-drug and drug-food interactions Storage instructions||3|
|Layout||Font size at least 12-point Adequate white space between paragraphs and between columns||2|
|Readability||Intended audience (i.e., consumer) is clearly defined Reading/comprehension level, graphics, audio, and other multimedia features are appropriate for the intended user||3|
|Languages||Entire webpage is available in both English and Spanish Drug information is available in both English and Spanish||1|
|User protection policies|
|Disclosure||Resources are free of advertising Source of funding or sponsorship is clearly stated||2|
|Disclaimer||A disclaimer describing limitations, purpose, scope, authority, and currency of the information is provided||2|
|Privacy protection||Site specifies whether and how users' personal information and anonymity are protected Offers users a choice to accept or decline collection and use of personal information provided by the user Specifies who has access to user information||2|
|Design and navigation||Organizational structure is easy to understand (i.e., clear headings and clear separation of different items) Use consistent organization across pages Links work and no dead ends A site map is provided User's location is apparent within a site (e.g., leaves a trail of breadcrumbs or provides consistent and well-placed menus and submenus) or when user leaves one site for another Browser “Back” buttons enable users to view previous pages||3|
|Accessibility||Web pages are accessible to people who are partially sighted or blind, cannot use a mouse or other pointing device, or have limited hardware or software or low bandwidth||2|
Two senior undergraduate communication majors were trained as screening evaluators. Screeners first assessed the websites independently; then reached agreement on inclusion/exclusion decisions through consensus. Websites meeting all screening criteria were assessed for quality by a separate, trained panel of 2 doctoral students (1 in pharmacy and 1 in communication) and 1 graduate student who majored in communication. Ratings for the quality criteria ranged from “1” (does not meet criterion at all) to “5” (fully meets criterion), except for 3 criteria that were dichotomous in nature (i.e., HONcode, URAC accreditation, and currency), which were assigned either 1 or 5 points. Interrater consistency was examined using the generalizability theory (GT) analysis, which allows us to separately estimate multiple sources of errors.16 Ratings were averaged, then multiplied by the corresponding weight of each criterion and summed to generate a total score for each website. Data were analyzed using SPSS 13.0 for Windows.17
The search procedures identified 45 potential websites for evaluation. Application of the screening criteria eliminated 15 sites, leaving 30 to be evaluated for quality. The most frequent reasons for failing to meet screening criteria were: lack of comprehensive information (6 sites); lack of differentiation between informational content and advertising (5 sites); lack of stated authorship (4 sites); lack of contact information (3 sites); and lack of update date (3 sites).
The GT analysis indicated that the variance due to the interaction between website and rater, and the interaction between website and evaluation criteria contributed to 4.5% and 73.6%, respectively, of the error variance. The result indicated good interrater consistency and substantial variability in website quality depending on evaluation criteria. The 10 websites with the highest total scores were selected for inclusion in the webliography. The differences in the features of these 10 websites are shown in Table 2.
|Type of Organization or Sponsor||Drug Interaction Checker||Drug Photo or Identifier||Webpage Available in Spanish||Drug Information Available in Spanish||Drug Information Readable at or Below 9th Grade Level||Includes OTC Medications||Includes Information on Herbs|
|Anthem Blue Cross and Blue Shield: Drug Information http://home.anthemhealth.com/topic/drugcenter||For profit||Yes||No||Yes||Yes||No||Yes||Yes|
|Cheshire Medical Center: Healthwise Knowledge Base http://www.cheshire-med.com/index.php?option=com_wrapper&Itemid=286||Nonprofit||No||No||No||No||Yes||Yes||Yes|
|DrugDigest: Drug Library http://www.drugdigest.org/DD/Home||For profit||Yes||Yes||No||No||No||Yes||Yes|
|Drugs.com http://www.drugs.com/||For profit||Yes||Yes||No||Yes||No||Yes||Yes|
|Healthvision: Drug Guide http://www.yourhealthinformation.com/library/healthguide/en-us/drugguide/default.htm||For profit||No||No||No||No||No||Yes||Yes|
|Kaiser Permanente: Drug Encyclopedia http://members.kaiserpermanente.org/kpweb/drugency/entrypage.do||Nonprofit||No||Yes||Yes||Yes||No||Yes||Yes|
|MedLibrary.org: Medication Information Library http://medlibrary.org/||For profit||No||No||No||No||No||No||No|
|PersonalMD: Drug Database http://www.personalmd.com/drugdatabase.shtml||For profit||Yes||No||No||No||No||Yes||Yes|
|U.S. National Library of Medicine: Drug Information http://www.nlm.nih.gov/medlineplus/druginformation.html||Government||No||No||Yes||Yes||No||Yes||Yes|
|WebMDHealth: Drugs A-Z http://www.webmd.com/medical_information/drug_and_herb/||For profit||No||Yes||No||No||No||Yes||Yes|
In this study, we identified a set of high quality medication information websites using a 2-step evaluation procedure. The results indicate that about one-third of the websites we evaluated did not include features deemed necessary for a high quality consumer-oriented website. We further confirmed that websites offering medication information vary greatly in contents and quality.
Similar findings were reported in a previous study conducted by Doupi and van der Lei18 assessing 14 websites which offered prescription medication information to the public. These websites showed wide variation in coverage of topics relevant to prescription medications and none met all the quality criteria (i.e., authority, attribution, disclosure, currency).18 Although the authors raised issues and concerns about the quality of internet drug information, they did not provide a list of recommended websites.
To date, health-related webliographies have focused mainly on general or disease-specific health information. Most webliographies targeting drug information either lack specified selection criteria or are not kept updated. A recent study by Coleman19 in the United Kingdom is one of the few that provided a list of consumer-oriented quality drug information websites. The current report updates Coleman's results using a similar 2-step selection process with extended criteria. Our screening criteria resulted in a more selective candidate group than did Coleman's 3 exclusion criteria. In contrast to Coleman's study, we identified 15 more candidate websites and selected 7 more finalists. Among Coleman's 3 finalists, only MEDLINE plus was selected in our webliography. Coleman19 created an information leaflet listing the selected websites, frequently cited assessment criteria, and internet search tips. However, the leaflet is not available to the general public. The present webliography is accessible for internet users worldwide on the “For Consumers” page of the Arizona Center for Education and Research on Therapeutics (AzCERT) website, http://www.azcert.org. Printable versions of the webliography with 10 recommended websites are available in 2 downloadable (PDF) formats: a tri-fold brochure and a patient information sheet. These materials can be used by health care providers as quick references for patients.
Most patients lack adequate time, knowledge, and motivation to evaluate the quality of internet health information themselves. Professionally compiled webliographies can attenuate the need for patient education on internet literacy, save patients' time and effort, and reduce their uncertainty in searching for quality health information. The 2-step evaluation process employed in this study can serve as a model for developing other webliographies targeting specific topics, diseases, or therapeutic categories, using evaluation criteria based on relevance and appropriateness.
This study has several limitations. Because online information and websites change rapidly, the results of this assessment are valid only for the period of April to June 2006, when evaluations were conducted. The substantial differences in the results of a previous similar study conducted in 2004 confirm the changeability of internet-based information. Owing to resource constraints, the search strategies used may not have identified all existing websites for quality assessment. Lastly, as these websites were selected based on their total rating scores, they may not score high on all features characteristic of a consumer-oriented quality website. Not surprisingly, the result of GT analysis implies that application of a different set of quality criteria might result in different ratings. Individuals seeking websites containing a specific feature may need to investigate further.
We thank Drs. Lane Johnson, Amy Grizzle, Jacob Abarca, and Edward Armstrong for their thoughtful comments on the assessment criteria. We appreciate Drs. Dan C. Malone and Jacob Abarca's comments on an early draft of this manuscript. We are grateful for assistance from the medical librarians at the Arizona Health Sciences Library and for website evaluation provided by Cynthia A. Condit.
Financial Disclosure: None.
Grant Support: Agency for Healthcare Research and Quality multi-year grant (HS10385) to the Arizona Center for Education and Research on Therapeutics.
- 3The mysterious maze of the World Wide Web: how can we guide consumers to high-quality health information on the Internet?. In: MureroM, RiceRE, eds. The Internet and Health Care: Theory, Research, and Practice. Mahwah, NJ: Lawrence Erlbaum Associates. 2006:195–212.
- 4Many Americans seek drug information—but not drugs—online. Drug Benefit Trends. 2004;16:536–7.
- 13URAC's Health Web Site Accreditation Program. http://www.urac.org/. Accessed June 5, 2006.
- 14HON Code of Conduct (HONcode) for medical and health Web sites. http://www.hon.ch/HONcode/. Accessed June 5, 2006.
- 15The New Rational Manager. Princeton, NJ: Princeton Research Press; 1981.,
- 16Generalizability Theory: A Primer. Newbury Park, NJ: Sage Publications Inc.; 1991.,
- 17SPSS. SPSS Base 13.0 for Windows User's Guide. Upper Saddle River, NJ: Prentice Hall; 2001.
- 19Producing an information leaflet to help patients access high quality drug information on the internet: a local study. Health Info Libr J. 2003;20:160–171.