SEARCH

SEARCH BY CITATION

Keywords:

  • internet;
  • webliography;
  • medication information;
  • prescription drug information

Abstract

  1. Top of page
  2. Abstract
  3. METHODS
  4. RESULTS
  5. DISCUSSION
  6. Acknowledgments
  7. REFERENCES

BACKGROUND: Websites offering drug information vary in coverage and quality, and most health care consumers are poorly equipped to assess the quality of internet medication information.

OBJECTIVE: To establish a webliography of recommended prescription medication information websites for health care consumers and providers.

DESIGN AND METHODS: Drug information websites were systematically identified based on recommendations from health professionals and text-word searches of MEDLINE and Google. The resulting sample of websites was evaluated in a 2-step process. Candidate websites were first screened using inclusion/exclusion criteria representing minimum information requirements. Websites that passed the inclusion/exclusion criteria were then rated on 16 quality criteria using a 5-point scale by 3 trained judges. Website ratings were averaged, then multiplied by the corresponding importance weight of each criterion and summed to generate a total score. Websites with the highest total scores were included in the webliography.

RESULTS: Ten websites were selected for inclusion in the webliography. The 3 highest-scoring websites were Anthem Blue Cross and Blue Shield (http://home.anthemhealth.com/topic/drugcenter), U.S. National Library of Medicine (http://www.nlm.nih.gov/medlineplus/druginformation.html), and Healthvision (http://www.yourhealthinformation.com/library/healthguide/en-us/drugguide/default.htm).

CONCLUSION: Medication information websites vary widely in quality and content. The online webliography is a valuable and easily accessed tool that can be recommended by health care professionals to patients who request referral to reliable websites.

The internet has become an important source for consumers searching for health information.1 The convenience, privacy, and nearly instant access to innumerable health-related resources are transforming the delivery of health information.2 However, seeking health information on the internet poses risks as well as benefits. The proliferating amount of health information available online is overwhelming, and the variability in quality is problematic.1,3 One area of particular concern is online drug information. Approximately 26% of Americans have used the internet to seek prescription drug information.4 Incomplete, misleading, and inaccurate information on the Web may contribute to harmful outcomes, such as drug misuse and medication errors.5–8

Locating and evaluating medication-related websites is time consuming and requires a certain level of internet search proficiency, health literacy, and knowledge to distinguish good quality information from bad. Most consumers, lacking some portion of such knowledge, would benefit from professional assistance. Many sets of criteria, checklists, and rating instruments have been developed for evaluating internet health information quality,9–12 such as Health On the Net (HONcode) and Utilization Review Accreditation Commission (URAC) accreditation.13,14 While these approaches are useful to assess general health information quality, few tools have been developed to specifically evaluate the quality of online prescription medication information.

The overwhelming volume, technical complexity and uncertain reliability of drug information on the internet can be confusing to patients, and they often ask health professionals for guidance in selecting reliable websites or evaluating internet information.2 A webliography, or annotated list of websites relating to a given subject,2 that is current, developed systematically by professionals, and readily accessible, can help health professionals respond to patients' requests. We applied evaluation criteria developed by pharmacists and health communication specialists to produce an internet-accessible prescription medication information webliography for consumers.

METHODS

  1. Top of page
  2. Abstract
  3. METHODS
  4. RESULTS
  5. DISCUSSION
  6. Acknowledgments
  7. REFERENCES

Site Selection

Websites to be considered for the 2-step evaluation process were identified by a search of MEDLINE, an internet search, and recommendations by health professionals. A Google search was performed in April 2006 using the search terms “medicine information,”“medication information,”“drug information,”“pharmaceutical information,” and “health information” individually and in combination with “webography” and “webliography.” The same search terms, with the addition of the terms “internet” and “online,” were searched as text words in MEDLINE using the PubMed interface in May 2006. Websites mentioned in published articles obtained from MEDLINE were visited to ascertain whether they were still operating. Additional sites were recommended by a convenience sample of 6 health professionals knowledgeable about medication safety. All websites that provided prescription medication information in English were included. To avoid unbalanced information, pharmaceutical company-maintained websites were excluded. Because patients are targeted as end-users of the webliography, journal websites were excluded due to their professional-oriented content. In addition, to be selected for evaluation, sites were required to provide free access, require no user registration, and not be limited to drugs of one class or for a specific disease.

Evaluation Criteria

Identified websites were evaluated against 15 screening criteria (including 9 criteria used to select potential websites) and 16 quality criteria. All items in the 2 sets of criteria were chosen or derived from existing standards and guidelines obtained from 17 published or internet sources. A panel composed of 1 primary care physician, 3 pharmacists, and 2 health communication experts reviewed both sets of criteria. To pass the screening criteria and be selected for quality assessment, potential sites were required to provide information about drug identification information (including both generic and trade names), therapeutic indications, and possible undesirable effects. In addition, sites were required to have no endorsement of particular products, and clearly differentiate advertising from information content. Additional requirements included contact information, date of last update, and disclosure of authorship.

Websites that passed the screening were evaluated for quality using a method adapted from an established decision-making process.15 The 16 quality criteria were weighted according to relative importance as determined by the investigators (see Table 1). Weights were assigned on a scale of 1 (least important) to 3 (most important).

Table 1. Quality Criteria
CriterionDescriptionAssigned Weight
  1. URAC, Utilization Review Accreditation Commission; HON, Health On the Net.

Drug information quality
 Assurance of information qualitySite has a written policy or procedure that states:  Information provided has been subjected to review and the process/individuals involved have been described  Claims relating to the benefit/performance of a specific treatment, commercial product, or service are justified based on appropriate and balanced evidence3
 AttributionCitations for original sources of evidence are provided Links to these referenced citations are available Copyright statement is provided1
 ObjectivitySite contains information on other medications or nonmedication treatments for the same condition2
 CurrencyDrug information is updated within last 6 mo1
 FeedbackA feedback mechanism is available for users to offer comments, corrections, and criticisms1
 HONcodeSubscribes to the HONcode principles2
 URAC accreditationDisplays URAC health website accreditation seal1
Drug information content and design
 ComprehensivenessIncludes:  Instructions for proper use of the drug  Therapeutic category  Information about the condition that the medication is used to treat or a link to a website with that information  Contraindications Drug-drug and drug-food interactions  Storage instructions3
 LayoutFont size at least 12-point Adequate white space between paragraphs and between columns2
 ReadabilityIntended audience (i.e., consumer) is clearly defined Reading/comprehension level, graphics, audio, and other multimedia features are appropriate for the intended user3
 LanguagesEntire webpage is available in both English and Spanish Drug information is available in both English and Spanish1
User protection policies
 DisclosureResources are free of advertising Source of funding or sponsorship is clearly stated2
 DisclaimerA disclaimer describing limitations, purpose, scope, authority, and currency of the information is provided2
 Privacy protectionSite specifies whether and how users' personal information and anonymity are protected Offers users a choice to accept or decline collection and use of personal information provided by the user Specifies who has access to user information2
Website design
 Design and navigationOrganizational structure is easy to understand (i.e., clear headings and clear separation of different items) Use consistent organization across pages Links work and no dead ends A site map is provided User's location is apparent within a site (e.g., leaves a trail of breadcrumbs or provides consistent and well-placed menus and submenus) or when user leaves one site for another Browser “Back” buttons enable users to view previous pages3
 AccessibilityWeb pages are accessible to people who are partially sighted or blind, cannot use a mouse or other pointing device, or have limited hardware or software or low bandwidth2

Evaluation Procedure

Two senior undergraduate communication majors were trained as screening evaluators. Screeners first assessed the websites independently; then reached agreement on inclusion/exclusion decisions through consensus. Websites meeting all screening criteria were assessed for quality by a separate, trained panel of 2 doctoral students (1 in pharmacy and 1 in communication) and 1 graduate student who majored in communication. Ratings for the quality criteria ranged from “1” (does not meet criterion at all) to “5” (fully meets criterion), except for 3 criteria that were dichotomous in nature (i.e., HONcode, URAC accreditation, and currency), which were assigned either 1 or 5 points. Interrater consistency was examined using the generalizability theory (GT) analysis, which allows us to separately estimate multiple sources of errors.16 Ratings were averaged, then multiplied by the corresponding weight of each criterion and summed to generate a total score for each website. Data were analyzed using SPSS 13.0 for Windows.17

RESULTS

  1. Top of page
  2. Abstract
  3. METHODS
  4. RESULTS
  5. DISCUSSION
  6. Acknowledgments
  7. REFERENCES

The search procedures identified 45 potential websites for evaluation. Application of the screening criteria eliminated 15 sites, leaving 30 to be evaluated for quality. The most frequent reasons for failing to meet screening criteria were: lack of comprehensive information (6 sites); lack of differentiation between informational content and advertising (5 sites); lack of stated authorship (4 sites); lack of contact information (3 sites); and lack of update date (3 sites).

The GT analysis indicated that the variance due to the interaction between website and rater, and the interaction between website and evaluation criteria contributed to 4.5% and 73.6%, respectively, of the error variance. The result indicated good interrater consistency and substantial variability in website quality depending on evaluation criteria. The 10 websites with the highest total scores were selected for inclusion in the webliography. The differences in the features of these 10 websites are shown in Table 2.

Table 2. Features of Medication Information Websites Scoring Highest in Quality*†
 Type of Organization or SponsorDrug Interaction CheckerDrug Photo or IdentifierWebpage Available in SpanishDrug Information Available in SpanishDrug Information Readable at or Below 9th Grade LevelIncludes OTC MedicationsIncludes Information on Herbs
  • *

    Websites were accessed for evaluation during April to June 2006. For updated information, please visit http://www.azcert.org.

  • Websites are listed in alphabetical order. A complete list of all websites evaluated is available from azcerted@email.arizona.edu.

Anthem Blue Cross and Blue Shield: Drug Information http://home.anthemhealth.com/topic/drugcenterFor profitYesNoYesYesNoYesYes
Cheshire Medical Center: Healthwise Knowledge Base http://www.cheshire-med.com/index.php?option=com_wrapper&Itemid=286NonprofitNoNoNoNoYesYesYes
DrugDigest: Drug Library http://www.drugdigest.org/DD/HomeFor profitYesYesNoNoNoYesYes
Drugs.com http://www.drugs.com/For profitYesYesNoYesNoYesYes
Healthvision: Drug Guide http://www.yourhealthinformation.com/library/healthguide/en-us/drugguide/default.htmFor profitNoNoNoNoNoYesYes
Kaiser Permanente: Drug Encyclopedia http://members.kaiserpermanente.org/kpweb/drugency/entrypage.doNonprofitNoYesYesYesNoYesYes
MedLibrary.org: Medication Information Library http://medlibrary.org/For profitNoNoNoNoNoNoNo
PersonalMD: Drug Database http://www.personalmd.com/drugdatabase.shtmlFor profitYesNoNoNoNoYesYes
U.S. National Library of Medicine: Drug Information http://www.nlm.nih.gov/medlineplus/druginformation.htmlGovernmentNoNoYesYesNoYesYes
WebMDHealth: Drugs A-Z http://www.webmd.com/medical_information/drug_and_herb/For profitNoYesNoNoNoYesYes

DISCUSSION

  1. Top of page
  2. Abstract
  3. METHODS
  4. RESULTS
  5. DISCUSSION
  6. Acknowledgments
  7. REFERENCES

In this study, we identified a set of high quality medication information websites using a 2-step evaluation procedure. The results indicate that about one-third of the websites we evaluated did not include features deemed necessary for a high quality consumer-oriented website. We further confirmed that websites offering medication information vary greatly in contents and quality.

Similar findings were reported in a previous study conducted by Doupi and van der Lei18 assessing 14 websites which offered prescription medication information to the public. These websites showed wide variation in coverage of topics relevant to prescription medications and none met all the quality criteria (i.e., authority, attribution, disclosure, currency).18 Although the authors raised issues and concerns about the quality of internet drug information, they did not provide a list of recommended websites.

To date, health-related webliographies have focused mainly on general or disease-specific health information. Most webliographies targeting drug information either lack specified selection criteria or are not kept updated. A recent study by Coleman19 in the United Kingdom is one of the few that provided a list of consumer-oriented quality drug information websites. The current report updates Coleman's results using a similar 2-step selection process with extended criteria. Our screening criteria resulted in a more selective candidate group than did Coleman's 3 exclusion criteria. In contrast to Coleman's study, we identified 15 more candidate websites and selected 7 more finalists. Among Coleman's 3 finalists, only MEDLINE plus was selected in our webliography. Coleman19 created an information leaflet listing the selected websites, frequently cited assessment criteria, and internet search tips. However, the leaflet is not available to the general public. The present webliography is accessible for internet users worldwide on the “For Consumers” page of the Arizona Center for Education and Research on Therapeutics (AzCERT) website, http://www.azcert.org. Printable versions of the webliography with 10 recommended websites are available in 2 downloadable (PDF) formats: a tri-fold brochure and a patient information sheet. These materials can be used by health care providers as quick references for patients.

Most patients lack adequate time, knowledge, and motivation to evaluate the quality of internet health information themselves. Professionally compiled webliographies can attenuate the need for patient education on internet literacy, save patients' time and effort, and reduce their uncertainty in searching for quality health information. The 2-step evaluation process employed in this study can serve as a model for developing other webliographies targeting specific topics, diseases, or therapeutic categories, using evaluation criteria based on relevance and appropriateness.

This study has several limitations. Because online information and websites change rapidly, the results of this assessment are valid only for the period of April to June 2006, when evaluations were conducted. The substantial differences in the results of a previous similar study conducted in 2004 confirm the changeability of internet-based information. Owing to resource constraints, the search strategies used may not have identified all existing websites for quality assessment. Lastly, as these websites were selected based on their total rating scores, they may not score high on all features characteristic of a consumer-oriented quality website. Not surprisingly, the result of GT analysis implies that application of a different set of quality criteria might result in different ratings. Individuals seeking websites containing a specific feature may need to investigate further.

Acknowledgments

  1. Top of page
  2. Abstract
  3. METHODS
  4. RESULTS
  5. DISCUSSION
  6. Acknowledgments
  7. REFERENCES

We thank Drs. Lane Johnson, Amy Grizzle, Jacob Abarca, and Edward Armstrong for their thoughtful comments on the assessment criteria. We appreciate Drs. Dan C. Malone and Jacob Abarca's comments on an early draft of this manuscript. We are grateful for assistance from the medical librarians at the Arizona Health Sciences Library and for website evaluation provided by Cynthia A. Condit.

Financial Disclosure: None.

Grant Support: Agency for Healthcare Research and Quality multi-year grant (HS10385) to the Arizona Center for Education and Research on Therapeutics.

REFERENCES

  1. Top of page
  2. Abstract
  3. METHODS
  4. RESULTS
  5. DISCUSSION
  6. Acknowledgments
  7. REFERENCES
  • 1
    Eysenbach G, Powell J, Kuss O, Sa ER. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA. 2002;287:2691700.
  • 2
    Ryan M, Smith KM. Webographies: a patient's guide to health information on the Internet. Am J Health Syst Pharm. 2003;60:8767.
  • 3
    Seidman JJ. The mysterious maze of the World Wide Web: how can we guide consumers to high-quality health information on the Internet?. In: MureroM, RiceRE, eds. The Internet and Health Care: Theory, Research, and Practice. Mahwah, NJ: Lawrence Erlbaum Associates. 2006:195212.
  • 4
    Many Americans seek drug information—but not drugs—online. Drug Benefit Trends. 2004;16:5367.
  • 5
    Eysenbach G, Kohler C. Does the internet harm health? Database of adverse events related to the internet has been set up. BMJ. 2002;324:239.
  • 6
    Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the quality of medical information on the internet: caveant lector et viewor—let the reader and viewer beware. JAMA. 1997;277:12445.
  • 7
    Wax PM. Just a click away: recreational drug web sites on the internet. Pediatrics. 2002;109:e96.
  • 8
    Crocco AG, Villasis-Keever M, Jadad AR. Analysis of cases of harm associated with use of health information on the internet. JAMA. 2002;287:286971.
  • 9
    Gagliardi A, Jadad AR. Examination of instruments used to rate quality of health information on the internet: chronicle of a voyage with an unclear destination. BMJ. 2002;324:56973.
  • 10
    Jadad AR, Gagliardi A. Rating health information on the internet: navigating to knowledge or to Babel? JAMA. 1998;279:6114.
  • 11
    Kim P, Eng TR, Deering MJ, Maxfield A. Published criteria for evaluating health related web sites: review. BMJ. 1999;318:6479.
  • 12
    Winker MA, Flanagin A, Chi-Lum B, et al. Guidelines for medical and health information sites on the internet: principles governing AMA web sites. JAMA. 2000;283:16006.
  • 13
    URAC's Health Web Site Accreditation Program. http://www.urac.org/. Accessed June 5, 2006.
  • 14
    HON Code of Conduct (HONcode) for medical and health Web sites. http://www.hon.ch/HONcode/. Accessed June 5, 2006.
  • 15
    Kepner CH, Tregoe BB. The New Rational Manager. Princeton, NJ: Princeton Research Press; 1981.
  • 16
    Shavelson RJ, Webb NM. Generalizability Theory: A Primer. Newbury Park, NJ: Sage Publications Inc.; 1991.
  • 17
    SPSS. SPSS Base 13.0 for Windows User's Guide. Upper Saddle River, NJ: Prentice Hall; 2001.
  • 18
    Doupi P, Van Der Lei J. Rx medication information for the public and the WWW: quality issues. Med Inform Internet Med. 1999;24:1719.
  • 19
    Coleman B. Producing an information leaflet to help patients access high quality drug information on the internet: a local study. Health Info Libr J. 2003;20:160171.