Readability and Suitability Assessment of Patient Education Materials in Rheumatic Diseases

Authors

  • Rennie L. Rhee,

    Corresponding author
    • VA Medical Center and University of Pennsylvania, Philadelphia
    Search for more papers by this author
  • Joan M. Von Feldt,

    1. VA Medical Center and University of Pennsylvania, Philadelphia
    Search for more papers by this author
  • H. Ralph Schumacher,

    1. VA Medical Center and University of Pennsylvania, Philadelphia
    Search for more papers by this author
    • Dr. Schumacher has received fees for his role as Section Editor for UpToDate.

  • Peter A. Merkel

    1. VA Medical Center and University of Pennsylvania, Philadelphia
    Search for more papers by this author
    • Dr. Merkel has received fees (less than $10,000 each) for his role as a contributing author for the American College of Rheumatology, UpToDate, the Vasculitis Clinical Research Consortium, and the Vasculitis Foundation.


Division of Rheumatology, University of Pennsylvania, 3400 Civic Center Boulevard, 8 Penn Tower, Philadelphia, PA 19104. E-mail: rennie.rhee@uphs.upenn.edu

Abstract

Objective

Web-based patient education materials and printed pamphlets are frequently used by providers to inform patients about their rheumatic disease. Little attention has been given to the readability and appropriateness of patient materials. The objective of this study was to examine the readability and suitability of commonly used patient education materials for osteoarthritis (OA), rheumatoid arthritis, systemic lupus erythematosus, and vasculitis.

Methods

Five or 6 popular patient resources for each disease were chosen for evaluation. Readability was measured using the Flesch-Kincaid reading grade level and suitability was determined by the Suitability Assessment of Materials (SAM), a score that considers characteristics such as content, graphics, layout/topography, and cultural appropriateness. Three different reviewers rated the SAM score and means were used in the analysis.

Results

Twenty-three resources written on the 4 diseases were evaluated. The education material for all 4 diseases studied had readability above the eighth-grade level and readability did not differ among the diseases. Only 5 of the 23 resources received superior suitability scores, and 3 of these 5 resources were written for OA. All 4 diseases received adequate suitability scores, with OA having the highest mean suitability score.

Conclusion

Most patient education materials for rheumatic diseases are written at readability levels above the recommended sixth-grade reading level and have only adequate suitability. Developing more appropriate educational resources for patients with rheumatic diseases may improve patient comprehension.

Introduction

Inadequate health literacy is an independent risk factor for poor health outcomes, including higher hospital admission rates ([1]), medication nonadherence ([2]), lower use of preventative services ([3]), and mortality ([4, 5]). Health literacy is defined as the degree to which individuals obtain, process, and understand basic health information and services needed to make appropriate decisions about their own health ([6]). Studies have shown that more than one-third of adults have inadequate proficiency in health literacy tasks ([7]).

More attention is being placed on health care professionals and health systems to provide usable health information ([6]). Printed and web-based educational resources are often used to supplement and reinforce information provided by clinicians. New standards are being established to enhance the quality of patient education materials, such as recommendations to write educational materials at a sixth-grade reading level ([8]).

Readability and suitability are tools used to evaluate appropriateness of written information. Readability refers to the reading difficulty of a resource and uses word and sentence length to determine a score. Focusing on readability, however, is not enough for improving comprehension. Suitability measures how well the material can be understood and accepted by the reader.

There is growing interest in rheumatology in the importance of patient health literacy ([9, 10]). Prior studies have shown that a large number of patients with rheumatic diseases are health illiterate and that the reading level of these patients is lower than the educational materials written for them ([9, 10]). Little attention has been paid to suitability of materials for rheumatic diseases. The objective of this study was to examine readability and obtain pilot data on suitability of commonly used patient education materials for rheumatic diseases and determine whether readability and suitability vary among diseases of different complexities.

Box 1. Significance & Innovations

  • Poor health outcomes are associated with limited health literacy, and national campaigns are focusing on improving the quality of patient education materials.
  • This is the first study to examine patient materials on several rheumatic diseases and assess both the readability and suitability of these materials for the general English-speaking population.
  • Results from this study indicate that resources are not meeting the recommended standards for readability and suitability, raising concerns about the effectiveness of these materials in educating patients.
  • This study provides a detailed assessment of current patient literature, bringing attention to areas in need of improvement and indicating the needs for further research to explore methods to improve patient comprehension.

Materials and methods

Osteoarthritis (OA), rheumatoid arthritis (RA), systemic lupus erythematosus (SLE), and vasculitis were selected to provide a spectrum of complexity of rheumatic diseases. Complexity was determined by familiarity to the public, based on disease prevalence, and features such as autoimmunity and multisystem involvement, which are concepts that are more difficult to translate to the general patient population. Several highly popular web sites that include patient resources were chosen for evaluation, including the American College of Rheumatology (ACR), Arthritis Foundation (AF), Mayo Clinic Health Information, National Institute of Arthritis and Musculoskeletal and Skin Diseases (NIAMS), UpToDate Basics, UpToDate Beyond the Basics, Vasculitis Clinical Research Consortium (VCRC), and Vasculitis Foundation (for references, see Supplementary Appendix A, available in the online version of this article at http://onlinelibrary.wiley.com/doi/10.1002/acr.22046/abstract). All materials were obtained between December 2011 and May 2012. All resources are available for free via the internet. These resources were chosen based on the authors' judgment of what was regarded as reputable, accurate materials that were representative of the resources available to the public.

A total of 23 resources were used: 6 resources each for OA, RA, and SLE (ACR, AF, Mayo Clinic, NIAMS, UpToDate Basics, and UpToDate Beyond the Basics) and 5 resources for vasculitis (ACR, Mayo Clinic, UpToDate Beyond the Basics, VCRC, and Vasculitis Foundation). To assess readability, the Flesch-Kincaid readability test was applied to every resource. The Flesch-Kincaid test is a valid and reliable language readability formula that is now widely used in many fields ([11]). The formula is embedded in Microsoft Word word processing software, allowing easy accessibility and usability. Readability was measured after removing all illustrations, tables, captions, footnotes, and web links. Subsequently, disease and medication names, all forms of “rheumatology,” and defined terms were removed and readability was measured again. This step was included to minimize the inflation of scores due to labels and other pronouns that are unavoidable and are often polysyllabic (e.g., “azathioprine” or “rheumatologist”).

Suitability was measured using the Suitability Assessment of Materials (SAM) created by Doak et al ([8]). The SAM is a systematic tool to assess printed health-related educational resources in a short amount of time. The SAM has been validated ([12]) and successfully used in prior studies of other printed health information ([13, 14]). The SAM consists of 6 evaluation criteria: content (e.g., “behavior information to help solve their problem”), literacy demand (e.g., “common, explicit words are used”), graphics (e.g., “simple, adult-appropriate, line drawings/sketches are used”), layout and typography (e.g., “type size is at least 12 point, no ALL CAPS for long headers or running text”), learning stimulation and motivation (e.g., “complex topics are subdivided into small parts so that readers may experience small successes in understanding or problem solving”), and cultural appropriateness (e.g., “images and examples present the culture in positive ways”).

Three readers were trained in the SAM scoring techniques. Each reviewer scored all study materials and were blinded to the source of the material. The mean SAM scores were used in the analysis. The SAM consists of a total of 21 questions and each question is rated as one of the following: superior (2 points), adequate (1 point), or not suitable (0 points). Therefore, the highest score possible is 42. The given score is divided by the total possible score to obtain a percentage. A score of 0–39% is considered not suitable, 40–69% is considered adequate, and 70–100% is considered superior ([8]).

Results

For the combined resources, the mean Flesch-Kincaid readability scores by grade level were high both before and after disease names, medications, and defined terms were omitted (Table 1). The mean reading grade level varied by resource, with UpToDate Basics having the lowest reading grade level (4.8) and the Vasculitis Foundation having the highest (12.5). Only UpToDate Basics met the recommended standard of a sixth-grade reading level for all 3 diseases examined (OA, RA, and SLE) (Figure 1).

Table 1. Mean readability and suitability scores*
 OARASLEVasculitis
  1. OA = osteoarthritis; RA = rheumatoid arthritis; SLE = systemic lupus erythematosus.
  2. aBased on Flesh-Kincaid grade level scores. Illustrations, tables, captions, footnotes, and web links were removed. Omissions refer to removal of disease and medication names, all forms of “rheumatology,” and defined terms.
  3. bBased on Suitability Assessment of Materials scoring by percentage of total possible score.
Mean readability score before omissionsa9.510.09.912.6
Mean readability score after omissionsa8.28.68.910.0
Mean suitability score, %b68565745
Figure 1.

Comparison of readability scores. ACR = American College of Rheumatology; AF = Arthritis Foundation; Basics = UpToDate Basics; Beyond = UpToDate Beyond the Basics; Mayo = Mayo Clinic; NIAMS = National Institute of Arthritis and Musculoskeletal and Skin Diseases; VCRC = Vasculitis Clinical Research Consortium; VF = Vasculitis Foundation; OA = osteoarthritis; RA = rheumatoid arthritis; SLE = systemic lupus erythematosus.

The mean SAM percent score for all sources was considered adequate: OA scored 68%, RA scored 56%, SLE scored 57%, and vasculitis scored 45% (Table 1 and Figure 2). The mean SAM percent score by resource resulted in more variability of scores, ranging from 32% (VCRC) to 72% (AF). Only the AF had a superior mean SAM score. Of the 23 total resources assessed, 5 received a superior suitability percent score: ACR for OA, ACR for SLE, AF for OA, AF for RA, and UpToDate Basics for OA. Notably, 3 of the 5 superior resources were on OA, a relatively less complex disease. Both readability and suitability scores were not significantly different among diseases, although the sample size was small.

Figure 2.

Comparison of Suitability Assessment of Materials (SAM) scores. ACR = American College of Rheumatology; AF = Arthritis Foundation; Basics = UpToDate Basics; Beyond = UpToDate Beyond the Basics; Mayo = Mayo Clinic; NIAMS = National Institute of Arthritis and Musculoskeletal and Skin Diseases; VCRC = Vasculitis Clinical Research Consortium; VF = Vasculitis Foundation; OA = osteoarthritis; RA = rheumatoid arthritis; SLE = systemic lupus erythematosus.

For 6 of the 23 resources, 2 of the 3 readers disagreed on SAM scores by a wide margin: inadequate (score 0–39%) versus superior (score 70–100%) (data not shown). The 6 individual evaluation criteria that make up the total SAM score, i.e., content, literacy demand, graphics, layout and typography, etc., were also analyzed separately to determine if one criterion contributed most to the differences in scores. The mean score was similar for each evaluation criterion when comparing different diseases.

Discussion

The purpose of this study was to assess the readability and suitability of patient education materials for 4 rheumatic diseases: OA, RA, SLE, and vasculitis. The selected resources were considered by the authors as representative of reputable, commonly-used materials available to the general public. Almost all of the resources were written at an eighth-grade reading level or higher, even after polysyllabic medication and disease names were removed and defined medical terms were omitted. Only UpToDate Basics met the recommended target of a reading level below sixth grade for all 3 diseases examined, an outcome intended by the editors ([8]). Readability varied more across resource than disease, suggesting that variability in writing style contributes more to differences in readability than disease complexity does.

Suitability scores had more disparities across the 4 diseases. OA had the highest mean suitability score (68%), whereas vasculitis had the lowest (45%). Despite the differences, all 4 diseases were still within the range of adequate suitability. In addition, 3 of the 5 resources that were considered superior were on OA. These findings indicate that diseases of lower complexity may be associated with better suitability scores. Although suitability scores were not significantly different between diseases, the sample size was small, and future studies using more resources are needed to further assess these differences.

This study has several strengths. Although prior studies have examined readability for only a single disease, this study is the first to examine readability of educational materials in 4 rheumatic diseases. The advantage of including 4 diseases was to ascertain whether disease complexity correlates with poorer readability and suitability scores. Reading difficulty is just one of many factors that impact comprehension. Exploring suitability in addition to readability adds another dimension to understanding the issues that affect patient comprehension, and this study provides pilot data on assessing suitability in rheumatic diseases. More studies with a larger sample size are needed to further evaluate suitability. Other strengths to this study include the use of standardized and widely accepted measures to assess the outcomes.

However, several limitations of this study are important to consider. Although the materials chosen for review were intended to be representative of most resources, the materials may not fully represent the spectrum of available materials. Although the readers were trained in suitability evaluation, interrater variability in the SAM scores was noted. Since suitability of patient reading materials is increasingly utilized, the subjectivity of scoring may be minimized in the future.

A possible solution to improve the quality of patient education materials is to adopt a standard reading level requirement for all educational materials. Doak et al ([8]) suggested the use of a sixth-grade reading level in order to reach a majority of the population. Even patients with adequate health literacy prefer simpler reading materials ([15]). The Agency for Healthcare Research and Quality within the Department of Health and Human Services has developed the Health Literacy Universal Precautions Toolkit, which sets forth guidelines for primary care providers to improve written communication ([16]).

Alternatively, resources may be written at varying degrees of difficulty, allowing physicians to customize the resource to the reading level of the patient. UpToDate has bypassed the issue of oversimplifying patient material in order to improve reading level by creating 2 versions of patient education material. However, this stratification by readability assumes that the provider is aware of the patient's reading level, which does not necessarily correlate with educational background or self-report ([17]). Physicians tend to overestimate what their patients can understand ([18]), and directly questioning the patient about his or her literacy level may not be effective because patients frequently struggle with shame and embarrassment caused by illiteracy ([19]).

The implications of this study are extensive. During a time of rising health care costs and growing attention to systematic errors, improving patient education materials may enhance a patient's understanding of their condition and treatment, thereby reducing inefficiencies such as patient noncompliance and unnecessary hospital admissions. Various programs are currently in development and being investigated to determine optimal methods of educating patients ([20, 21]).

Supplemental materials may alleviate some of the burden placed on physicians who have limited time to spend in the office on counseling and education. This study suggests that patient resources in rheumatology are still lacking and may not be appropriately communicating the requisite information to patients. Addressing this problem will require providing writers with guidelines and standardized tools to create effective patient materials.

In accordance with the US Department of Human and Health Services' campaign to improve health literacy ([6]), more attention needs to be placed on the methodology used to disseminate information to patients. This study concludes that patient materials for rheumatic disease are still inadequate to educate patients. A more detailed inspection and further revision of current rheumatology materials are needed in order to improve patient understanding and ultimately patient care.

AUTHOR CONTRIBUTIONS

All authors were involved in drafting the article or revising it critically for important intellectual content, and all authors approved the final version to be published. Dr. Rhee had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study conception and design. Rhee, Schumacher, Merkel.

Acquisition of data. Rhee, Von Feldt.

Analysis and interpretation of data. Rhee, Von Feldt, Schumacher, Merkel.

Acknowledgments

The authors thank Drs. R. Michelle Koolaee, Sophia Li, and Ghaith Noaiseh for their contribution to this study as the SAM reviewers.

Ancillary