Developing health website quality assessment guidelines for the voluntary sector: outcomes from the Judge Project
Sue Childs, Information Management Research Institute (IMRI), School of Informatics, Northumbria University, Newcastle upon Tyne NE1 8ST, UK. E-mail: firstname.lastname@example.org
Background: This paper describes the Judge project, the aim of which was to explore the need for health website quality assessment guidelines for the voluntary sector. Such guidelines would enable health consumers to assess the quality of health sites and assist support groups to produce their own good quality sites. The project was a partnership between the Information Management Research Institute, School of Informatics, Northumbria University and Contact a Family. It was supported by the Health Foundation.
Methods: The views of health consumers and support group members and workers were obtained by focus groups (35 attendees) and questionnaires (55 responses). They were asked questions about quality issues and concerns about Internet health information and any help they needed with judging the quality of health websites.
Results: The results supported the need for guidelines. The guidelines were written to reflect the requirements of health consumers and support groups articulated from the focus group and questionnaire data. They were then disseminated via a website.
Conclusions: There is a need to improve communication and information exchange between health consumers and professionals and official organizations; publicise support groups to patients and carers; set up local Webs of Trust, linking together voluntary and statutory organizations within a locality. Professionals and official organizations in the public sector should make information provision to patients and health consumers a priority. Information provision should be embedded in the clinical setting via guidelines, protocols and administrative structures. Professionals should be trained in the role of information and how to provide it in appropriate ways. Support groups should use the Judge guidelines to assist them in producing good quality websites. Support groups should promote the Judge guidelines to health consumers, to help them make their own quality judgement about health-information websites.
More and more people use the worldwide web as a source of information. In 2003–4, 12.1 million UK households had Internet access at home and 58% of adults had used the Internet.1 A 2003 US survey showed that 80% of adult Internet users have searched for health information.2 Many surveys have assessed the quality of health information on the Web,3,4 and found that most websites, even those produced by health professionals and organizations, contain incorrect information. Such surveys have raised concerns that health information on the Internet is inaccurate, misleading or even dangerous. Different types of quality assessment schemes exist: lists of principles and quality criteria;5−7 guidance on assessing the presence of these criteria;8,9 kitemarking schemes10−13; and technical mechanisms.14 These schemes have been reviewed.15−18 A fundamental debate exists between those who believe that kitemarking by trusted third parties is the way to tackle the problem,17 and those who believe this is ineffective because of the logistics of tackling such a huge number and range of sites.19,20 There have been studies assessing rating schemes21,22 which found that they lacked formal evaluation of interobserver reliability or construct validity and usually had a very short ‘shelf-life’. However there appears to be strong agreement on the individual criteria that can be used to evaluate sites.20,23−25 Such criteria can be categorised under authoritativeness (e.g. authorship, attribution), trustworthiness (e.g. disclosure), message characteristics (e.g. currency, accuracy), audience characteristics, and design features.
Voluntary organizations’ websites are important sources of health information for patients, their carers and health consumers.26,27 While they may contain medical information, they also include a rich source of information of varying kinds, e.g. factual information, often qualitative and descriptive; directory information on sources of help; contact information to put individuals in touch with people in the same circumstances; psychological and emotional support, including descriptions of individuals’ experiences; membership events and meetings; fundraising; challenges to the existing methods of treatment and care; and lobbying for improvements in services. Existing quality criteria do not cover all these information categories and focus on professional sites28; e.g. the requirement for information to be evidence-based is most applicable to disease management and treatment. The call for national kitemarking schemes causes concerns in the voluntary sector as their types of sites may be downgraded or sidelined. A more effective way of dealing with health site quality could be to help health consumers make their own judgements.18,20,23,27,29,30
The Judge project addressed two important issues facing health consumers and support groups. Firstly, empowering health consumers to judge the quality of Internet health information. Secondly, promoting the role of support groups by helping them produce their own good-quality websites. The Judge project was developed in partnership between the Information Management Research Institute, School of Informatics, Northumbria University, and ‘Contact a Family’, the national charity for families with disabled children. The project's aim was to explore the need for health website quality assessment guidelines for the voluntary sector. The project received support from the Health Foundation and ran from March 2002 to February 2003. The project Steering Group consisted of representatives from: the Alzheimer's Society; Centre for Health information Quality (CHIQ); National Electronic Library for Health; National Information Forum; OMNI: Organising Medical Networked Information; and Patient UK.
The research methodology was qualitative, comprising: a literature review of quality assessment of health information on the Internet, which informed the questions in the focus groups and questionnaires; obtaining health consumers’ views, by questionnaires and focus groups, on quality issues and concerns about Internet health information and any help they needed with judging the quality of health websites. The ‘health consumer’ subject group was composed of parents/carers of children with rare diseases and support group members/workers, chosen as their experiences meant they were active and in-depth users of health information. There was no clear separation between carers and support group members/workers. Carers often set up their own support group, or become members and often workers of existing support groups. Support group workers are often themselves parents of sick or disabled children and therefore have close, personal knowledge of the needs of health consumers. Therefore, the focus groups and questionnaires did not distinguish between carers or support group workers.
Four focus groups were held at Contact a Family branches across the UK, i.e. Birmingham, Edinburgh, London, Newcastle. A total of 35 people attended. The meetings were audiotaped, transcribed and the transcriptions analysed for themes. One thousand printed questionnaires were circulated as an insert in Contact a Family's newsletter, which is posted to health consumers, carers and support groups; 55 questionnaires were returned. The results were analysed to give counts, percentages and rankings. Because of the small numbers, these results are not statistically significant. The two methods provided for triangulation. Focus-group attendees were asked to complete a brief questionnaire to check that their characteristics were similar to those of the questionnaire respondents. There was good agreement between the two methods. No personal details of individual subjects were obtained and the results are anonymous.
Focus group results
The views of the focus group attendees are summarised below.
Views on judging the quality of websites
Judging the quality of information is very difficult and complex. This is not a new problem; use of traditional information sources also entails quality judgements. Judging the usefulness or quality of a website is very personal. Individual users are looking for information that suits them and their family circumstances. This might be different for other people with the same medical condition. A good-quality site may be in the eye of the beholder. The nature of medical information causes lay people problems, because of specialised terminology, newer findings replacing earlier views, disagreements between leading experts in the field, unorthodox views of some researchers, and different professional recommendations in different countries. Support groups validate the medical information they provide by using medical advisers. It is much harder for health consumers to validate medical information. Mechanism used by attendees include: noting areas of agreement between sites; checking how sites cover known facts; site recommendations; checking the information sources used by a site; contacting researchers directly; accessing support groups who have validated medical information; asking advice from GPs or consultants, although often this approach is not welcomed by the doctors concerned.
It is not just professional information that is important. Personal experiences are also vital. Such information must be clearly marked as ‘personal experiences’. However, some individual patients’ sites could contain incorrect information and the public may not be aware of the differences in the information quality of such sites compared with professional sites.
Views on the usefulness of standard quality criteria
Date was a very important criterion; websites should be regularly updated. The importance of trust in the reputation of the site depends on the individual user. Some attendees felt there were few organizations they would trust. The agenda of a site can affect the trust people place in it. Worries were expressed about the sponsorship or funding of sites biasing the information provided. Serious concerns were expressed about sites that sell drugs or treatments. Sites should provide unbiased, balanced information. They should give factual information and not be sensational. This balance includes providing both the professional and patient viewpoints. Information should be simply written, easy to understand, with no medical jargon. Sloppiness in the way information is written can be a negative quality signal. Attendees rarely talked about authors, although they did talk about the organizations producing the site. One page on the site where all the details about the organization, authorship, updating, etc. could be kept would make it far easier for the user to find such information.
There are many good non-British sites. However, there are a number of problems with the information they contain as different countries have different health systems and cultural practices, use different terminology and recommend different treatments. People can sometimes be unaware that they have accessed a non-British site, or are in e-mail correspondence with someone from another country. Providing information in different languages is important to reach people in the UK who do not speak English. However, this is a major difficulty because of the large number of languages spoken in the UK.
Logos are not commonly used. Some attendees expressed interest in kitemarking sites. Others were concerned about who would make these quality judgements, and whether this would mean that the type of information provided by support group sites would be downgraded. Website producers need to make sites accessible and navigable, so it is easy to find sites and to find information within sites. However, although good site design can be encouraging, it is not the final arbiter for acceptance of information. Interaction and feedback should be built into sites. This is a particular strength of support group and parent sites, which have the purpose of establishing communication and networking. Professional sites are far less likely to provide a feedback facility.
Links to other websites are useful sources of information, and a method of verification of a site's content. There are a number of problems for site producers in using links. The first level of links can be checked, but it is not practicable to check the sites that the first site links to, and so on. However, an organization could be legally challenged on the information in a site to which its site links There is also the time commitment in rechecking URLs for currency. Organizations need to take responsibility for the links they make, and ensure they are valuable and validated. Websites should include disclaimers.
Views on issues about Internet health information
The more interactive facilities of the Internet, e.g. e-mail lists and chat rooms, also need quality checking. Attendees felt there is a number of problems with such facilities, e.g. obsessives misusing the facility; new patients advocating treatments that experienced people know are ineffective; people criticising others; and people who pretend to have an illness or a sick child as they want empathy and support.
Attendees also discussed the process of searching the Web. Parents with a newly diagnosed child usually found information by trial and error, often over a long time period. Although this may seem inefficient, this process enabled them to gain their critical judgement skills. The process of looking for information is important, and part of a person's adjustment to their condition. There are different stages to a person's information gathering (just as there are different stages of an illness), i.e. looking for an explanation; looking for anything that will help; with experience, focusing down to the things that work for them. After a person has received a diagnosis, staged information is required. This enables the person to receive necessary information at the time they need it, without being overwhelmed or frightened. Support groups are a vital part of this staging process, providing discussion and emotional support.
Attendees expressed a number of concerns about health information generally. The main problem was the poor provision of information by professionals and official organizations, in health, education, social services, the local authority, etc. Local support groups would be a lifeline for people, providing both information and emotional support. However, professionals and official organizations often do not give patients/carers the contact details of support groups. Anger towards and suspicion of professionals is created in some people because of the bad experiences they have had trying to obtain information. This poor provision of information by professionals was caused by many reasons, e.g. poor communication skills; lack of time when information provision is a low priority; lack of knowledge of information; professional jealousy and secrecy; as a covert way of lowering demand on over stretched services; and refusal to accept the implications of patients directly accessing Internet health information. The result is that the ideal of an equal, two-way partnership between doctors and patients is often not realised in practice.
Attendees had concerns about dangerous or biased information on the Internet. Internet information has different characteristics e.g. quantity of information available; ease of obtaining information; ease with which anyone can publish information on the Web; lack of staging information; and lack of discussion to put information in context. Information does not always have a positive effect; it can frighten and disempower people and make them feel guilty, e.g. if a parent discovers that their child's disease has a genetic basis, therefore people need guidance in how to interpret and use Internet information.
Attendees felt there was a need for methods of providing Internet health information to disadvantaged groups, particularly ethnic groups.
Views on the help required to make decisions about the quality of websites and the health information they contain
Attendees felt that guidance is needed to assist health consumers in finding and using Internet health information. This guidance should cover searching skills, judging information quality, and managing information in partnership with health professionals. Guidance is also needed for site developers. Support groups have a very important role to play in the provision of information to health consumers. They provide emotional and personal support, as their workers and members have personal experience of the condition under question, and they can put patients in contact with each other. They provide information, often on rare conditions, which can be difficult to find elsewhere. They take on the task of sifting and staging information, making it easily understood and explaining the research literature. They can discuss information with enquirers and put it into context, advising on sources of information and what is good quality information. They act as a conduit between the patient and the professional and fight the health consumer's corner.
Attendees felt that professionals and official organizations should make information provision to patients and the public a priority. Such information would include leaflets, contact details of support groups and recommended websites. A number of mechanisms were suggested to achieve this, e.g. handing out information in the clinical setting; ‘clinical’ guidelines and protocols on information provision; checking that information had been provided as part of the administrative system; making information provision a legal requirement; and training professionals in the role of information and how to provide information in an appropriate way.
The need for kitemarking was raised by some attendees, as a method of dealing with sites containing dangerous information, e.g. the Charity Commission could take on the role of auditing charity websites. There is a need to set up sites that pull together local information on services and organizations, linked on to national sites. Health consumers need training in how to search the Web. Support groups need training in how to produce their own sites. A health specific search engine might improve people's chances of finding good quality sites.
Postal questionnaire results
The results of the postal questionnaire are summarised below (Tables 1–11). A copy of the questionnaire (containing the detailed results) can be seen in the Appendix (only available online).
Table 1. Sources used to find information on a health topic
|People at work||38|
Table 2. Reason for using the Internet for health
|Information on health topic||91|
|Details of health/social care service||33|
|Contact details of support group||33|
|Contact details of people in similar circumstances||31|
|Contacting health professional/service||20|
|Joining in e-mail discussion group||11|
|Using chat room|| 7|
Table 3. Location of Internet access
|Via family or friend||11|
|At support group|| 9|
|In public library|| 4|
|In school/college|| 2|
Table 4. Preference for organizational source of health information websites
Table 5. Importance of trust/reputation measures when choosing a health information website
|Health professional recommendation||27|
|Support group recommendation||24|
|Linked to from other sites|| 4|
Table 6. Importance of authorship and content style measures when choosing a health information website
|Gives all sides of topic||64|
|Written by health expert||49|
|Aimed at general public||44|
|Written by support group||22|
|Aimed at health professionals||22|
|Written from personal viewpoint||13|
Table 7. Importance of design measures when choosing a health information website
|Keeping personal details secure and confidential||71|
|Easy to find information||67|
|No special software required||64|
|Easy to access||62|
|No personal details required||53|
|Quick to download||51|
|Attractive appearance|| 2|
|Uses graphics|| 0|
Table 8. Frequency of use of standard quality criteria when using a health information website
|Sources of information used by author|| 7|
|Conflicts of interest|| 7|
|Author's name and employer|| 4|
|Sources of finance|| 2|
Table 9. Level of concerns about Internet health information
|Information produced by cranks||51|
|Too much information|| 7|
|Difficulty finding useful sites|| 4|
Table 10. Help required in judging the quality of Internet health information
|Site recommendations by health professionals||71|
|Guidelines from support groups||71|
|Site recommendations by support groups||69|
|Guidelines from health professionals||66|
|Guidelines on websites||58|
|Training by support groups||53|
|Guidelines in a leaflet||53|
|Training by health professionals||51|
|Site recommendations on other websites||47|
|Training by schools for children||47|
|Guidelines publicised in magazines||46|
|Training by schools for parents||46|
Table 11. Involvement with support groups
Alhough the Internet is the most common source of information, many other sources are also used (Table 1). Respondents’ use of the Internet is not restricted to information searching. Interactive facilities are also used (Table 2). Most respondents use the Internet at home, with nearly half using it at work. Other locations are little used (Table 3).
Experience of Internet searching was ranked: high 40%; fair 36%; low 18%; none 6%. Experience of setting up a website was ranked: None 71%; low 15%; fair 9%; high 6%. Most respondents had a good level of experience of Internet searching, although about a quarter had little experience. Results for setting up websites were the complete reverse.
The highest preference was for professional organization sites and support group sites (Table 4). Far fewer respondents preferred commercial organization sites and individual persons’ sites.
A site produced by a reputable/trustworthy organization was thought more important than a site produced by a known organization (Table 5). Use of site recommendations was thought far less important. This should be compared with the results for types of help required (Table 10), where recommendations by both health professionals and support groups were highly ranked.
Regarding content, authorship was not the highest ranked factor (Table 6). The most important factor was unbiased information. Patient-friendly information was also regarded as important.
For design of a website, privacy was the most important factor, with usability and accessibility the next most important. Appearance was the lowest in importance (Table 7).
The most commonly used criterion was the date the information was written, followed by details of the organization producing the site, details of the site's sponsors and the author's qualifications (Table 8). The other criteria were far less commonly used. However, the percentages of respondents ‘always’ checking criteria are low overall. Between 4 and 47% of respondents ‘never’ checked specific criteria. Most commonly respondents ‘sometimes’ checked for these criteria (20–44%).
Respondents shared the general concerns about the correctness and safety of Internet health information (Table 9). Issues about searching for sites were of far less concern.
The respondents were clearly interested in receiving some form of help with judging Internet health information quality (Table 10). These findings endorsed the project's proposed aim of producing quality guidelines. However, they also endorse site recommendations and training provided by both health professionals and support groups.
Most of the respondents were involved with support groups in some capacity (Table 11). This involvement can be very complex, e.g. one respondent was involved with several support groups; another respondent had had all levels of involvement with their support group at one time or another.
Producing the guidelines
The guidelines were written to reflect the requirements of health consumers and support groups articulated from the focus group and questionnaire data. Draft guidelines were piloted with health consumers, support groups and organizations active in health information provision. The final version of the guidelines was promoted via a website ‘Judge: Websites for Health’ (http://www.judgehealth.org.uk). Two sets of guidelines were produced: to help health consumers make informed choices about websites; to assist support groups to produce good quality websites. The guidelines comprised the sections shown in Table 12.
Table 12. Guidelines resulting form the Judge project
|1. How to search the Internet for health information:|
|• information searching|
|• search engines|
|• search tips|
|• confidence building|
|2. How to judge the quality of a website:|
|• trust and reputation|
|• who produced the site|
|• purpose of the site|
|• funding sources|
|• how the information is written|
|• descriptions of conditions and treatments|
|• medical research|
|• personal experiences|
|• foreign sites|
|• interactive facilities|
|• other quality guidelines|
|3. How to manage information in partnership with health professionals|
|Support Group Guidelines|
|1. How to produce good quality information:|
|• trust and reputation|
|• details of the organization|
|• purpose of the organization|
|• associated organizations|
|• funding sources|
|• the way the information is written|
|• non-english language material|
|• descriptions of conditions and treatments|
|• medical research|
|• personal experiences|
|• links to other sites|
|• data protection|
|• terms and conditions|
|• interactive facilities|
|• other quality guidelines|
|2. How to design good quality websites:|
|• principles of good design|
|• accessibility—fonts, colours, graphics and audio|
|• accessibility—use of technology|
|• testing and evaluation|
|• maintenance and development|
|3. How to market websites:|
|• website address|
|• search engine listings|
|• linking to your site|
|• traditional methods|
|4. How to help consumers use health information:|
|• promoting the value of information|
|• using gateways|
|• using search engines|
|• search tips and confidence building|
|• judging the quality of information|
|• managing information in partnership with health professionals|
The Judge website
The Judge website has passed accessibility and quality tests, i.e. Bobby, W3C, Visionary Design and the HON code. In 2003 it received a ‘Getting the Message Across’ Award from the National Information Forum. Monthly usage figures in the first part of 2004 were running at 7659 hits and 4458 pages.
Producing guidelines met the Judge project's aims. However, the findings from the focus groups have highlighted a number of additional issues that needed addressing, though this was beyond the scope of the Judge project. Firstly, communication and information exchange between health consumers and professionals and official organizations needs to be improved. The ideal of an equal partnership between health professionals and patients, with some patients and carers adopting the role of the ‘expert patient’, is not always happening in reality. Secondly, the value of support groups is not being fully exploited as many health consumers are unaware of them. Should support group contact details be provided to patients by their doctors as part of their ‘prescription’ or treatment? Thirdly, the wealth of knowledge and information at a local level needs to be well publicised and made more accessible. This suggests the idea of developing a ‘Local Web of Trust’, with voluntary and statutory organizations linking to each other's sites, reflecting their knowledge of each other's work and information quality based on working relationships and partnerships. An individual patient or member of the public placed in contact with one of these sites will therefore be able to link on to other quality assured sites. Future research is planned to pilot the development of a local Web of trust.
I would like to thank Diane Barnett, Information Officer, Contact a Family (joint developer of the project idea and joint project manager) and Kay Humphrey, previously Web Editor, CHIQ (technical design and development of the Judge website).