Commonly cited website quality criteria are not effective at identifying inaccurate online information about breast cancer

Abstract

BACKGROUND

Consumers increasingly consult the Internet for breast cancer information. Concerned about accuracy, multiple organizations developed quality criteria for online content. However, the effectiveness of these tools is unknown. The authors determined whether existing quality criteria can identify inaccurate breast cancer information online.

METHODS

The authors identified 343 unique webpages by using 15 breast cancer-related queries on 5 popular web search-engines. Each page was assessed for 15 quality criteria and 3 website characteristics, link type (sponsored or not), search engine used to find the page, and domain extension. Two clinician-reviewers independently assessed accuracy and topics covered. The authors then determined whether quality criteria, website characteristics, and topics were associated with the presence of inaccurate statements.

RESULTS

The authors found 41 inaccurate statements on 18 webpages (5.2%). No quality criteria or website characteristic, singly or in combination, reliably identified inaccurate information. The total number of quality criteria met by a website accounted for a small fraction of the variability in the presence of inaccuracies (point biserial r = −0.128; df = 341; P = .018; r² = 0.016). However, webpages containing information on complementary and alternative medicine (CAM) were significantly more likely to contain inaccuracies compared with pages without CAM information (odds ratio [OR], 15.6; P < .001).

CONCLUSIONS

Most breast cancer information that consumers are likely to encounter online is accurate. However, commonly cited quality criteria do not identify inaccurate information. Webpages that contain information about CAM are relatively likely to contain inaccurate statements. Consumers searching for health information online should still consult a clinician before taking action. Cancer 2008. © 2008 American Cancer Society.

More than 44% of women recently diagnosed with breast cancer use the Internet to find health information.1 The Pew Internet and American Life Project estimate that every day, more patients seek health information online than visit a physician.2 The recent National Cancer Institute-sponsored Health Information National Trends Survey3 found that healthcare providers remain a trusted source of health information; however, many respondents obtained health information online before going to their physician, and few patients were obtaining information from specific websites recommended by their physicians.4, 5 Therefore, consumers who search for health information online usually do so without professional guidance.

Many consumers are satisfied with the information they find online and make treatment choices on the basis of this information.6, 7 In some cases, patients do not discuss their use of online complementary and alternative medicine (CAM) treatments with their clinicians.8

Multiple studies9–11 have evaluated websites that display health information, specifically those relating to cancer. In fact, online information has been studied so frequently that this field has been given a name, “information epidemiology” or “infodemiology”.12 These studies found a high prevalence of false or misleading information online, leading clinicians, researchers, and healthcare consumers to become concerned about the accuracy of online health information.6, 10, 13, 14 Multiple organizations have developed quality-rating instruments to help healthcare consumers evaluate websites that provide health information; hundreds of such instruments now exist. However, it is not known whether these tools can identify inaccurate online health information.11, 15–17 The purpose of this study was to determine whether existing quality assessment tools can identify false or inaccurate breast cancer information that is online.

MATERIALS AND METHODS

Website Selection

Most consumers find online information by using general-purpose search engines rather than medical sites or portals, and most do not go beyond the first page of search results.18 Therefore, we used 5 popular search engines—Google, Yahoo Directory, AltaVista, Overture, and AllTheWeb—to identify webpages that consumers are likely to encounter. We used Yahoo Directory because at the time that the study was performed, the nondirectory web search on Yahoo's website was actually being carried out by Google. For each search engine, we performed 15 searches by using the most frequently encountered topics from the first 322 breast cancer-related entries in the NetWellness database of user questions19 (Table 1). All searches were performed from June 1 to July 30, 2004. We recorded all URLs found on the first search results page, including sponsored (advertisements) and unsponsored results (n = 1585 [Fig. 1]). After we eliminated duplicate uniform resource locators (URLs), 870 remained.

Figure 1. Depicted is the experimental design of the study.

Table 1. Search Terms Used to Select Websites

Search term
Breast cancer heredity
Breast cancer prognosis
Breast cancer radiation
Breast cancer (antiperspirant OR antiperspirant)
Breast biopsy
Breast cancer
Breast cancer diet
Breast cancer prevention
Breast cancer risk
Breast cysts
Breast discharge
Breast lump
Mammogram
Mastectomy
Tamoxifen

Relevance was operationally defined as the presence of breast cancer content on the webpage under evaluation. To ensure reliable assessment of relevance, the URLs were reviewed for relevance by 2 independent reviewers. If the content displayed by the retrieved URL page was not relevant, links on the first page of the retrieved page were reviewed, and the first relevant link was included in the study. Reviewers were instructed to follow links sequentially, starting from the top of the page, left to right. The reviewers agreed that 344 URLs were relevant. However, each reviewer identified an additional 25 URLs that were unique (eg, followed different links from the original page). To avoid bias, we included 35 of these 50 URLs in the analysis, excluding 15 duplicates (final n = 344 + 25 + 25 − 15 = 379).

The pages were downloaded to a local computer to create a static data set for analysis. Because online content changes continuously, the information stored in search-engine databases is often not an exact copy of the information in real time. Thus, some websites and webpages were unavailable (down) or required redirection and could not be downloaded. Webpages were downloaded for 2 review efforts; the original URL was downloaded for technical quality assessment, and the original URL plus webpages up to 2 links away were downloaded for assessment of quality and accuracy. Downloading the original URL generated 4 errors, leaving 375 pages. Downloading the webpage plus webpages up to 2 links away from the original webpage generated 30 errors, leaving 349 webpages. Six of these webpages were eliminated; 4 duplicates, 1 survey, and 1 blank. A total of 343 webpages were evaluated for both technical quality criteria and accuracy (Fig. 1). These webpages were also evaluated for website characteristics shown in Table 2, Column 1.

Table 2. Characteristics of Breast Cancer Websites

Characteristic	No. of websites	No. (%)
		Accurate websites	Inaccurate websites
* There were 9 sponsored results and 3 “additional sponsored” results. Additional sponsored results, which appeared separately from the main sponsored results in the search engine output, were analyzed as a separate group and contained no inaccuracies; sponsored and additional sponsored groups were combined into a single group to simplify presentation.
Link type
Sponsored*	12	11 (92)	1 (8)
Nonsponsored	331	314 (95)	17 (5)
Search engine
AllTheWeb	76	68 (89)	8 (11)
AltaVista	105	100 (95)	5 (5)
Google	63	63 (100)	0
Overture	44	42 (95)	2 (5)
Yahoo Directory	55	52 (95)	3 (5)
Domain extension
.com	193	180 (93)	13 (7)
.org	66	65 (98)	1 (2)
.gov	36	36 (100)	0
.edu	14	12 (86)	2 (14)
Other	34	32 (94)	2 (6)

Technical Quality Evaluation

A dedicated reviewer (M.W.) assessed each webpage for 15 technical quality criteria that were found to have acceptable interobserver reliability in a previous study. In that study, we measured the degree to which 2 raters agreed on the assessment of 22 popularly cited quality criteria in a sample of 21 CAM websites. We found poor interobserver reliability for several criteria. Therefore, we repeated the evaluation on an additional 21 websites by using a specific operational definition for each quality criterion. Interobserver reliability was considered acceptable if κ > 0.6 (Table 3, Column 1) when we used our specific operational definitions.15 Technical quality criteria are not necessarily independent (eg, if there is no disclosure of authorship, one cannot assess the authors' credentials).

Table 3. Compliance of Webpages With Quality Criteria

Quality criterion		No. of websites	No. (%)
			Accurate websites	Inaccurate websites
S indicates evaluated at website level; P, evaluated at webpage level; Other, No or Not Applicable (eg, when no author was identified, then author credentials were identified as not applicable).
Display of authorship (P)	Yes	114	109 (96)	5 (4)
	No	229	216 (94)	13 (6)
Authors' credentials identified (P)	Yes	68	67 (99)	1 (1)
	Other	275	258 (94)	17 (6)
Physicians' credentials identified (P)	Yes	17	17 (100)	0
	Other	326	308 (94)	18 (6)
Author affiliation disclosed (P)	Yes	62	61 (98)	1 (2)
	Other	281	264 (94)	17 (6)
Sources clear (P)	Yes	315	298 (95)	17 (5)
	No	28	27 (96)	1 (4)
General disclaimers provided (P)	Yes	340	322 (95)	18 (5)
	No	3	3 (100)	0
References provided (P)	Yes	75	71 (95)	4 (5)
	No	268	254 (95)	14 (5)
Disclosure of ownership (S)	Yes	338	320 (95)	18 (5)
	No	5	5 (100)	0
Date created identified (P)	Yes	94	92 (98)	2 (2)
	No	249	233 (94)	16 (6)
Date of update identified (P)	Yes	126	121 (96)	5 (4)
	No	217	204 (94)	13 (6)
Date of creation or last update identified (P)	Yes	191	185 (97)	6 (3)
	No	152	140 (92)	12 (8)
Editorial review process identified (S)	Yes	120	119 (99)	1 (1)
	No	223	206 (92)	17 (8)
Internal search engine present (S)	Yes	234	228 (97)	6 (3)
	No	109	97 (89)	12 (11)
Feedback mechanism provided (S)	Yes	325	308 (95)	17 (5)
	No	18	17 (94)	1 (6)
Copyright notice (S)	Yes	261	248 (95)	13 (5)
	No	82	77 (94)	5 (6)

Accuracy and Content Topic Evaluation

Two medically trained reviewers (S.S. and D.S.) independently reviewed the accuracy of each webpage after first familiarizing themselves with the National Comprehensive Cancer Network breast cancer guidelines and other applicable guidelines.20–23 Statements were considered inaccurate only when both reviewers identified them as inaccurate. Information that did not fall within the scope of the applicable guidelines was only considered inaccurate if it contradicted the reviewers' clinical knowledge. The same 2 reviewers also evaluated each webpage to identify which of 14 breast cancer topics were covered. A topic was considered to be present on a webpage when both reviewers independently identified the topic on that page. Of 343 webpages, 337 covered at least 1 of the 14 breast cancer topics.

Statistical Analyses

Accuracy was scored both as number of inaccuracies displayed on the webpage and dichotomously as whether or not the webpage contained at least 1 inaccurate statement. We could not apply parametric tests of hypotheses addressing significance of multiple correlations of website attributes with number of inaccuracies due to violations of significance tests' homoscedascity and normality assumptions.

Consequently, we performed binary logistic regression analyses and provide omnibus significance tests (α = .05) of associations between dichotomous accuracy and each of 3 families of attributes (3 website characteristics, 15 quality criteria, 14 topics). In addition, we performed individual significance tests of unique associations (controlling for other attributes within the family) with each family's constituent site attributes. If a logistic regression analysis did not converge to a solution, we removed constituent attributes whose iterative parameter estimates did not converge from the omnibus analysis, but we addressed these by using Fisher exact tests of associations between dichotomous accuracy and each of the 3 family's constituent attributes. For the 14 topics, odds ratios were used to assess the effect of topic presence upon dichotomous accuracy. Because we tested multiple hypotheses, we used the multistage Bonferroni procedure.24 The association between the number of quality criteria met by each webpage and dichotomous accuracy was measured and tested for statistical significance by point biserial correlation. Given 2-tailed significance tests, the sample size of 343 provided approximately 0.80 statistical power for detecting as significant effect sizes of approximately r = φ = 0.17 (α = .05) or r = φ = 0.19 (α = .01) for associations between 2 dichotomous variables, or 2 × 2 contingency tables.25 All statistical analyses were performed by SPSS software, versions 13–15 (SPSS, Chicago, Ill), running on Windows XP.

RESULTS

Website Characteristics

Link type

As shown in Table 2, most retrieved URLs (331 of 343 [97%]) were not sponsored, meaning that the search engines identified them using a search algorithm rather than advertisements.

Domain extension

The most common domain extension was .com (193 [56%]); this domain suggests that the site is maintained by a commercial organization rather than a noncommercial (.org; 66 [19%]), federal government (.gov; 36 [10%]), or educational (.edu; 14 [4%]) organization. The “other” category (34 [10%]) included websites that were maintained by state governments and organizations outside of the United States. We note that domain extensions do not always accurately represent the organization responsible for a website. For example, some state governments use the .gov extension.

Quality of Websites

As shown in Table 3, we found wide variation in websites' compliance with the 15 quality criteria. For example, 99% of webpages displayed general disclaimers such as “Not a substitute for professional care;” however, only 18% disclosed authors' affiliations, and only 5% displayed physicians' credentials.

Prevalence of Inaccuracies

We found 41 false or misleading statements on 18 (5.2%) of the 343 websites evaluated. In determining accuracy, agreement between reviewers was excellent; only 2 statements on 3 webpages were identified as inaccurate by 1 but not both reviewers; these were excluded from the list of inaccurate sites.

Website Characteristics and Accuracy

We determined that website characteristics (link type, search engine used, and domain extension) were not significantly associated with the existence of false or misleading statements. The logistic regression analysis converged after “Google” and “.gov” were deleted from the equation, resulting in a nonsignificant omnibus likelihood ratio X² = 8.88; df = 7; P = .20, and nonsignificant unique associations for constituent attributes. Fisher exact tests for associations in all 21 2 × 2 contingency tables of pairs of categories of the 2 link types, 5 search engines (including Google), and 5 domain extensions (including .gov) versus dichotomous accuracy using the multistage Bonferroni procedure (α = .05/21 = .0024) revealed no significant effects (Table 2).

Quality and Accuracy

Multiple quality criteria have been proposed to identify high-quality websites, which are presumably better sources of information. We found little evidence of association between technical quality criteria and accuracy. The logistic regression analysis only converged after 3 of the 15 quality criteria (“Physicians' credentials identified,”, “General disclaimers provided,”, and “Disclosure of ownership”) were deleted, resulting in nonsignificant omnibus likelihood ratio X² = 20.48; df = 12; P = .059, and nonsignificant unique associations for constituent quality criteria (including “Internal search engine present”).

Fisher exact tests addressed associations between each of the 15 quality criteria and dichotomous accuracy. With the multistage Bonferroni procedure (α = .05/15 = .0033), these tests revealed only “Internal search engine present” to be significantly associated with dichotomous accuracy (P = .0027; φ = 0.176; r² = 0.03; OR = 4.70; 95% CI, [1.72, 12.89]). The total number of quality criteria met by a webpage accounted for only a small fraction of the variability in dichotomous accuracy (point biserial r = −0.128; df = 341; P = .018; r² = 0.016) (See also Table 3). In summary, no technical quality criteria, tested alone or as sets of quality criteria, were consistent and significant predictors of accuracy.

Content Topic and Accuracy

“Complementary and alternative medicine” (CAM) was significantly associated with the presence of at least 1 inaccuracy (dichotomous accuracy) (Wald = 17.24; P < .001) (Table 4). The logistic regression analysis converged after “Chemoprevention” and “Prophylactic mastectomy” were deleted from the equation, resulting in a significant omnibus likelihood ratio (X² = 26.67; df = 12; P = .01, Nagelkerke R² = 0.25). Fisher exact tests for associations of content topic with dichotomous accuracy using the multistage Bonferroni procedure (α = .05/14 = .0036) for all 14 topics also revealed CAM to be the only significant predictor of inaccuracy (P < .001; φ = 0.302; R² = 0.09; OR = 15.60; 95% CI, 4.50, 54.15). We found that 5 of the 15 (33%) websites that displayed information pertaining to CAM contained inaccurate statements. The odds of inaccuracy for webpages having CAM among their topics were approximately 15.6 times those of sites not having the complementary medicine topic.

Table 4. Topic Versus Inaccuracy Odds Ratios

Topic	Sites with topic present n = 337	Inaccuracy odds ratio* n = 15†
* Ratio of odds of inaccuracy for sites with topic to odds of inaccuracy for sites without topic. † Three pages with inaccuracies did not contain at least 1 of the 15 content topics. ‡ Significant independent predictor of inaccuracy. § No sites identified by both reviewers as having chemoprevention and prophylactic mastectomy as a topic and errors.
Complementary medicine	15	15.6‡
Psychology	8	3.21
Reconstruction	24	2.1
Risk factors	123	2.1
Diet or nutrition	33	1.44
Diagnosis	86	1.1
Screening	112	1
Radiation treatment	47	0.95
General prevention	52	0.84
Surgical treatment	60	0.7
General information	190	0.5
Systemic therapy, including chemotherapy and hormonal treatment	86	0.44
Chemoprevention	38	0§
Prophylactic mastectomy	13	0§

DISCUSSION

We found that commonly cited website quality criteria that are recommended to healthcare consumers were not effective at screening out inaccurate breast cancer information. However, only 5.2% of webpages identified in our study contained inaccurate statements. Furthermore, the content topic was associated with inaccuracy. Specifically, webpages that contained information about CAM were 15.6 times more likely to contain inaccurate statements than pages not containing information about CAM.

The strengths of this study include our use of 2 independent reviewers to determine the relevance, topic, and accuracy of online content. Although technical quality criteria were evaluated by a single reviewer, only previously validated criteria that could be reliably assessed were included. Furthermore, our study was designed to obtain information that is likely to be encountered by consumers who are searching for breast cancer information online. To this end, we used popular search engines to search for topics that are of interest to consumers and included only those webpages listed on the first page of our search results.

In our previous study of breast cancer websites,11 we found that sites that complied with 3 or more Journal of the American Medical Association (JAMA) benchmarks26 were less likely to contain inaccuracies (10% vs 2%; P = .047). A possible reason for the difference in our current findings is the more precise operational definitions used. For example, the JAMA disclosure benchmark is described as follows:

… website ‘ownership’ should be prominently and fully disclosed, as should any sponsorship, advertising, underwriting, commercial funding arrangements or support, or potential conflicts of interest. This includes arrangements in which links to other sites are posted as a result of financial considerations. Similar standards should hold in discussion forums.

In our current study, disclosure of advertising was not considered because it could not be reliably assessed, even with an explicit operational definition.15

There are several important limitations of our study. First, online information changes rapidly, and specific webpages are likely to have changed by the time our review is published. Thus, we do not emphasize specific errors or websites. However, general trends, such as the relation between technical quality criteria and accuracy, should be more enduring. Because we reviewed only breast cancer information, our findings may not be generalizable to information about other malignancies. However, in our previous study of CAM websites that provided information about 3 popular herbal remedies (ginseng, ginkgo biloba, and St. John's wort), we also found that technical quality criteria were not effective at identifying potentially harmful CAM information.17 Taken together, the results of these studies suggest that consumers may not be able to identify accurate websites with the assistance of a simple quality criteria checklist and that it may be difficult to guide consumers to more accurate websites through an automated screening process that relies only on technical quality criteria.

We found a relatively low prevalence of inaccurate statements (5.2%), which is in contrast to findings in some other studies. For example, Berland et al.9 found that 37% of breast cancer information that received more than minimal web-based coverage was not completely accurate. In that study, information about childhood asthma, depression, and obesity was even less reliable. Biermann et al.10 on the other hand, found that only 6% of nonpeer-reviewed sources about Ewing sarcoma were inaccurate. The differences between these studies may reflect differences in study design. Specifically, Berland et al.9 studied a combination of coverage and accuracy, whereas we and Biermann et al.10 focused on only accuracy. Furthermore, we selected webpages identified by using general-purpose search engines, webpages that are likely to be encountered by actual searchers, whereas Berland et al.9 identified studies manually. The low prevalence of inaccurate statements could have obscured a relation between quality and accuracy (ie, the study was effectively underpowered), but this is an important finding in and of itself.

Few consumers consider the quality of online health information as they conduct their web searches.11, 18 In light of our findings, perhaps this is not a bad thing. Instead of encouraging patients to use published quality standards, we should encourage them to maintain a healthy skepticism of online health information, especially of CAM issues; consider the reputation of the source of the information, just as they should offline; and consult an appropriate clinician before taking action.

Our finding that breast cancer information is generally accurate does not mean that consumers are not misled by online information. Information is scattered across webpages and websites. Users find it relatively easy to answer specific questions (eg, “What is the 5-year disease-free survival rate of patients with stage I breast cancer?”), but users find it relatively difficult to answer questions that require comprehensive knowledge (eg, “What are the indications and contraindications of trastuzumab?”).27 Therefore, even a dedicated searcher is likely to retrieve incomplete information on any particular topic. In other words, the Internet may tell the truth but not the whole truth.

A key unresolved issue is the effect of online information on health outcomes. In a 2002 systematic review of published cases of harm resulting from the application of online health information, investigators found only 3 articles describing cases of harm.28 However, the authors acknowledged the possibility (and perhaps the likelihood) that cases of online “information toxicity” may be underreported in the published medical literature.

In summary, we found that commonly cited technical quality criteria were not effective at screening out inaccurate breast cancer information online, although webpages containing information about CAM were more likely to contain inaccurate statements compared with pages without CAM information. We also found a low prevalence of inaccurate information about breast cancer. Therefore, consumers who consult the Internet for information about breast cancer are likely to find accurate answers to their questions. However, they should be encouraged to maintain a healthy level of skepticism of online health information, consider the reputation of the source, and consult an appropriate clinician before taking action.