Quality of Internet information referring to mental health and mental disorders in Japan

Authors


Kiyotaka Nemoto, MD, Division of Psychiatry, Tsukuba University Hospital, 2-1-1 Amakubo Tsukuba, Ibaraki 305-8576, Japan. Email: kiyotaka@nemotos.net

Abstract

Abstract  Although the Internet has been widely used in Japan, the quality of information on mental health-related issues has not been evaluated so far. The purpose of the present study was to investigate the current status of Japanese websites that deal with mental health, mental disorders, and their associated matters. Using a search engine, Google, 37 websites were identified that exclusively contain information on mental health or mental disorders. The characteristics of the sites were then examined, along with variety of mental disorders mentioned, and quality of contents for each of the 37 websites. More than half of the websites were set up, at least in part, for commercial purposes and only 27% of sites were owned by professionals. Mood disorder, panic disorder, and schizophrenia were the three most commonly referred disorders on websites. Aside from some exceptions, the quality of information was inadequate, especially that regarding treatment. Most of the websites on mental health and mental disorders examined in the present study have scope for improvement. The challenge is to establish a system to evaluate the sites and to motivate each webmaster to improve the sites.

INTRODUCTION

The Internet has become one of the most popular and frequently used sources of information. It is currently estimated that approximately 80 million Japanese or almost 62% of the Japanese population use the Internet.1 Although there has been an explosion in the amount of health information available on the Internet, a number of studies have raised concerns about Internet sites that provide health-care information too soon after it became available, that is, before it has been scrutinized by experts.2–6 Most of studies that evaluated the quality of health-care information on the web are limited to websites in Western countries, and in Japan there has been only one study as far as we know. Tatsumi et al. reported that both Internet users and physicians thought that 32% of health-related websites described in Japanese are unreliable and put an emphasis on establishing codes of ethical conduct governing Internet site content.7

Studies on psychiatry websites have focused on websites about depression. Lissman and Boehnlein conducted online searches using the phrase ‘depression and treatment’ on 10 major Internet search engines. They reported that both the quality of diagnostic symptoms or criteria and treatment were quite low.8 Griffiths and Christensen evaluated the quality of information on treatment of depression in 21 popular websites about depression. They reported that the general quality of information on the treatment of depression was poor, which is consistent with the conclusion of Lissman and Boehnlein, and that site ownership was a better indicator of the quality of health-related information on these websites than other criteria used for evaluation,suggesting the importance of site ownership.9 In another similar study of 15 Australian depression websites, Griffiths and Christensen found that DISCERN score, which is a measure of treatment information and scientific accountability was a good indicator for the quality of content.10 Murphy et al. evaluated the quality of information on eating disorders and they also concluded that the quality was low.11 However, there have been no comprehensive studies evaluating the quality of website information on mental health and other mental disorders. Mental health issues attract attention in Japan. Although searching the web with the key words ‘mental health’ results in more than 333 000 hits,12 no studies have evaluated the quality of those sites.

Therefore, we conducted a study to investigate the current status of Japanese websites. This investigation attempted to answer the following questions: (i) what mental disorders are covered by these sites; (ii) how well are the diagnostic criteria or symptoms described; (iii) how much information on treatment is presented; and (iv) who owns the sites.

METHODS

Website search

We conducted a prospective, observational literature review by performing keyword searches using the popular search engine, Google, in May 2005. To simulate an Internet search that was likely to be undertaken by clinicians or patients looking for information about mental disorders, we used the 13 keywords that covered most of the diagnostic categories of DSM-IV-TR13 as follows: developmental disorder, dementia, substance abuse, schizophrenia, mood disorder including the word ‘depression’, anxiety disorder, somatoform disorder, dissociative disorder, eating disorder, sleep disorder, personality disorder, and mental health.

The first 200 sites generated by Google were reviewed for each keyword, for a total of 2600 sites. We selected sites to evaluate in two steps. The first step was to choose sites that had at least one page (or subdirectory) of information related to the key words within the site, which resulted in selection of 286 sites. In the second step, we chose sites in which almost all pages contained information on various mental disorders. This step yielded 37 websites, which were to be assessed.

Site characteristics

To rate the characteristics of the sites, we used a method similar to that of Griffiths and Christensen.9 Briefly, the following items were assessed: ownership (organization or individual), type of ownership (commercial, professional, consumer, public, or medical), profit or non-profit, involvement of a drug company, professional editorial board, and involvement of health professionals. Then, we counted the variety of mental disorders referred in each site.

Content quality

Currently, codes of conduct or third-party certification dominate the philosophy of self-regulatory initiatives for presenting health information on the Internet.14 These philosophies require that safety information be provided to users. However, our interest in this study was to evaluate the quality of the information itself. To our knowledge, there is no tool-based instrument available to evaluate information on symptoms or diagnostic criteria. Therefore, we used a subjective five-point rating scale for information about symptoms or diagnostic criteria, of which 5 is the ‘best described’. As for information on treatment, the DISCERN instrument15 was used for evaluation. DISCERN is one of the tools used to judge the quality of written information about treatment choices.14 The first eight questions address the reliability of the publication and help users consider whether it can be trusted as a source of information about treatment choices such as ‘Is it clear when the information used or reported in the publication was produced?’. The next seven questions focus on specific details of the information about treatment choices such as ‘Does it describe the benefits of each treatment?’ or ‘Is it clear that there may be more than one possible treatment choice?’. A DISCERN score was calculated by summing the scores on all 15 items of the DISCERN questionnaire (except the last item, which rates overall quality).

After evaluating the quality of symptoms or diagnostic criteria and treatment, a global score was estimated on the basis of a subjective 10-point rating scale, of which 10 is the ‘best described’. To exclude rating biases, three psychiatrists (NK, TH, SN) independently rated each site. Because Kendall's coefficient of concordance16 for each score (symptoms or diagnostic criteria, DISCERN, and global) indicated that all of the scores made by the three judges were highly correlated with each other (0.88, 0.94, and 0.86, respectively, P < 0.0001), the scores of the three judges were averaged.

RESULTS

Table 1 shows the ownership characteristics of the sites. Of the 37 sites that contained information exclusively on mental health, only five sites had editorial boards. Most of the sites belonged to some sort of organizations, and only 8% to individuals. Approximately half of them had their sites for profit motives. As for type of ownership, most were professional, commercial, and clinic or hospital sites. There were only four consumer sites and three public health center sites. Seven of 37 sites were owned by drug companies. Medical professionals were involved in >70% of the sites. The mean score on symptoms or diagnostic criteria was 3.4 out of 4. Although diagnostic criteria from the DSM-IV were available at many sites, explanations of symptoms were not comprehensive and were presented in abbreviated column format to facilitate quick comprehension. The mean DISCERN score on the recommended treatment was 46.2 out of 75. Most of the sites put an emphasis on pharmacotherapy, and the description of other therapies was limited. Moreover, compared to the description of symptoms or diagnostic criteria, information on treatment was limited. Although the mean global score was 6.1 out of 10, the global score of each site ranged from 4.0 to 9.3, suggesting that there was a considerable variation among websites. Further evaluation of quality of sites according to their characteristics showed that editorial board, ownership, and drug company involvement tended to indicate good quality of sites, although this was not statistically significant.

Table 1.  Ownership characteristics of sites
 n%Symptoms or diagnostic criteria
Mean ± SD
DISCERN
Mean ± SD
Global
Mean ± SD
  1. To evaluate information on symptoms or diagnostic criteria, we used a subjective 5-point rating scale, of which 5 is the best described. DISCERN is an instrument for judging the quality of written consumer health information on treatment choices. A DISCERN score was calculated by summing the scores on all 15 items of the DISCERN questionnaire. Its range is 15–75. Global score was estimated on the basis of a subjective 10-point rating scale, of which 10 is the best described.

Editorial board
 Yes5144.0 ± 1.058.0 ± 9.17.3 ± 1.6
 No32863.3 ± 1.143.4 ± 16.95.9 ± 1.5
Ownership
 Organization34923.4 ± 1.145.2 ± 16.16.0 ± 1.4
 Individual383.7 ± 1.547.0 ± 28.17.1 ± 2.8
Profit or non-profit
 Profit20543.4 ± 1.143.9 ± 16.86.2 ± 1.8
 Non-profit17463.4 ± 1.147.1 ± 17.15.9 ± 1.2
Type of ownership
 Professional10274.1 ± 1.355.4 ± 18.47.1 ± 1.9
 Commercial10272.9 ± 0.939.6 ± 16.15.3 ± 1.2
 Clinics or hospitals10273.3 ± 1.041.9 ± 13.45.7 ± 1.2
 Consumer4113.5 ± 0.448.8 ± 14.96.3 ± 1.4
 Public health center383.3 ± 1.338.3 ± 20.26.0 ± 1.8
Drug company involved
 Yes7193.8 ± 1.152.1 ± 13.86.8 ± 1.7
 No30813.4 ± 1.143.8 ± 17.25.9 ± 1.5
Professionals involved
 Yes27733.4 ± 1.145.2 ± 16.56.2 ± 1.6
 No10273.6 ± 1.045.9 ± 18.55.8 ± 1.3
Total371003.4 ± 1.145.4 ± 16.86.1 ± 1.6
 Range  1.0–4.815.0–70.04.0–9.3

Table 2 summarizes the variety of the illnesses described at each site. The mean number of mental disorders described in one site was 4.8. Of the 37 sites, 78% mentioned mood disorder, followed by panic disorder and schizophrenia. Only a few sites dealt with attention-deficit–hyperactivity disorder, substance-related disorder, epilepsy, and developmental disorder, suggesting that information available on the Internet about these disorders was limited.

Table 2.  Coverage of mental disorders in 37 websites
Mental disordern%
Mood disorder2978
Panic disorder1951
Schizophrenia1746
Generalized anxiety disorder1335
Obsessive-compulsive disorder1232
Alcohol-related disorder1232
Sleep disorder1130
Dementia1130
Eating disorder1027
Dissociative disorder924
Personality disorder719
Post-traumatic stress disorder514
Attention-deficit–hyperactivity disorder38
Substance-related disorder25
Epilepsy25
Developmental disorder13

In Table 3 we categorized the sites into four groups according to quality. Even though the way of assessing the information quality of symptoms or diagnostic criteria and treatment is different, we found that information on treatment had scope for improvement.

Table 3.  Quality of information on each site
Symptoms or diagnostic criteriaDISCERNGlobal
Ratingn%Scoren%Ratingn%
  1. To evaluate information on symptoms or diagnostic criteria, we used a subjective 5-point rating scale, of which 5 is the best described. DISCERN is an instrument for judging the quality of written consumer health information on treatment choices. A DISCERN score was calculated by summing the scores on all 15 items of the DISCERN questionnaire. Its range is 15–75. Global score was estimated on the basis of a subjective 10-point rating scale, of which 10 is the best described.

4.1–5.0164360.1–75.09247.6–10.0924
3.1–4.0123245.1–60.012325.1–7.51130
2.1–3.082230.1–45.011302.6–5.01746
1.0–2.01315.0–30.05141.0–2.500

DISCUSSION

To our knowledge this is the first study to investigate the current status of Japanese websites on mental disorders. Also, this is the first comprehensive search of mental illness-related sites on the Web, using key words appearing in the subheadings of DSM-IV. Of the 37 sites that contained information exclusively on mental disorders, 27% were commercially owned and another 27% belonged to clinics or hospitals. One purpose of the sites held by clinics and hospitals was to advertise, so, taken together, >54% of those websites were set up, at least in part, for commercial purposes. In contrast, only 27% of sites were owned by professionals. Among these, only two were owned by academic institutions. Academic institutions have the potential to provide comprehensive information. Indeed, Griffiths et al. reported that among the four best websites about depression in Australia, two were university-based sites.10 The present results indicate that academic institutions in Japan should provide comprehensive information on mental disorders.

The most common mental disorders described on the Web were mood disorders, especially depression, followed by panic disorder and schizophrenia. Mood disorders have been given much attention because increasing numbers of suicides among middle-aged people is a major social problem in Japan. To our surprise, panic disorder was the second most mentioned disorder. Panic disorder has received much attention recently because of its association with the use of serotonin selective re-uptake inhibitors and this may be one of the reasons why many sites mention it. Information on other mental illnesses was limited to only a few sites, and consumer self-help groups owned most of those sites.

In general, the quality of content on the web is low, although there is a wide variation between sites. Recently, Christensen et al. concluded from their randomized controlled study of a community sample that information about depression and use of cognitive behavioral therapy (CBT) to treat depression were both effective in reducing symptoms of depression, and noted that depression literacy could be as effective as CBT in reducing symptoms.17 This indicates that information about the symptoms of mental disorder itself could reduce the symptoms, and suggests that providing information might help prevent suffering from mental disorders and reduce their severity. Although all 37 sites in the present study provided useful information on symptoms and diagnosis, most of the information was not comprehensive and a column format was used to convey information. It is true that a column format is a user-friendly way of conveying information, but this format may not provide the readers with a comprehensive view of the illness. Although all sites do not always have to be comprehensive, the existence of comprehensive sites might help readers have a better understanding.

High-quality information on mental disorders could reduce the stigma or prejudice attached to the illness. Tanaka et al. examined the effects that a lecture on mental health had on public attitudes towards mental illness. They reported that attitudes improved significantly after the lecture, and suggested the effectiveness of this type of educational program in reducing the stigma attached to mental disorders.18 The advantage of the Internet is that it can reach a mass audience, and therefore improving site content could help reduce the stigma of mental illness in the future.

In the present study, content on recommended treatment was less informative in all sites. Most sites mentioned merely pharmacotherapy. This may reflect the current status of psychiatric therapy in Japan; For example, the most widely used therapy for depression remains pharmacotherapy, although other useful therapies including CBT, bright light therapy, and modified electroconvulsion therapy are also available. Griffiths and Christensen evaluated the Australian depression websites using DISCERN score.10 Their results were similar to ours, suggesting that the quality of sites in Japan and Australia is similar.

Despite their generally low scores for content quality, many sites did contain important and potentially useful information. Presentation of psychiatric information on the web is still in its infancy and there is much room for improvement of content, which may result in the improvement of the mental health outcomes of those who visit the sites. The challenge is to establish a system to evaluate the sites and to motivate each webmaster to improve the sites.

Our study has some limitations. First, we used only one engine, Google. Google was used because it was the leading search engine. As of June 2006, Nielsen NetRatings reported that Google had 46.3% of the market.19 Also, the second most popular engine in Japan, Yahoo, also used Google as its engine for web-page retrieval when we did the survey. In addition, we searched for sites using 13 different keywords. The results overlapped in many fields and we assumed that by reviewing those 2600 web pages, the possibility of overlooking websites on mental health and mental disorders would be quite low. Second, we used subjective rating scales including DISCERN, although three of us rated each site independently to reduce the bias. Systems for rating website quality tend to be based on codes of ethics, and are difficult to use for making comparisons of the quality of websites. Better instruments are thus needed.

In conclusion, although some sites contain useful information, the overall quality of mental disorder-related sites in Japan is low. Because high-quality information on mental disorders has the potential to improve the mental health outcomes of those who visit the sites, the challenge is for academic institutes or other specialized institutes to improve comprehensive information.

ACKNOWLEDGMENT

This study was supported in part by The 21st Century COE Program.

Ancillary