- Top of page
- SUBJECTS AND METHODS
Health education and self-management (SM) programs for persons with arthritis and other chronic diseases have become prominent in North America, Britain, Australia, and other regions (1, 2). These programs are becoming widely endorsed by clinical and community groups as well as governments. However, systematic reviews reveal small to moderate effects for biologic outcomes (e.g., glycosylated hemoglobin, blood pressure, asthma attacks) and small to no effects for questionnaire-based assessments, including subjective patient-reported outcomes such as pain or disability in individuals with arthritis (3).
Traditional evaluation of health intervention programs relies on classic pre- and postintervention assessment, focusing on changes from baseline to followup. The outcomes assessed often include proximal outcomes (e.g., change in knowledge, self efficacy) and more distal outcomes such as health-related quality of life, pain, or disability. The assessment of these outcomes relies on eliciting participant self-perceived status pre- and postintervention. These assessments assume that participants do not change the way they perceive their health over the period of the program. The assumption is that participants have the same perspective of their disease and their wellbeing both before and after the program and, therefore, answer a followup questionnaire from the same point of view as they held for the baseline questionnaire.
The definition of response shift (RS) is a change in the meaning of one's self evaluation of a target construct as a result of recalibration (a change in a person's internal standards of measurement), reevaluation or reprioritization (a change in a person's values), and reconceptualization (a change in the way a person defines a target construct; for example, quality of life) (4). We define RS as having 3 forms of expression: negative (i.e., after the course, participants report they now realize they were worse before the course than they thought they were), positive (i.e., participants now realize they were better than they thought they were), or absent (no change).
RS is a phenomenon that was explored and developed in the fields of education and organizational change (5). It has recently been applied to medical and health-related psychosocial research where self-reported data are collected (4). In these areas, RS has primarily been examined with respect to individuals who experience long-term, often terminal illness and their caregivers (5–7). In this study, RS was investigated with regard to arthritis and other chronic disease SM programs that were of short duration (6 weeks).
Pre- and postintervention assessments are generally not designed to detect participants' changed perceptions of their health. However, as participants learn (and use) new and more effective strategies for coping with their chronic disease and observe others who are in better and worse conditions than themselves, their internal standards of measurement, their values, and their definition of health and quality of life may change. If individuals have a changed perception of symptoms, then they may answer questions in a systematically different way at followup. That is, participants may respond differently to questions posed after an SM program than they did to the same questions posed before an SM program. It may be possible for this to happen even if they have no physical improvement or decline. It is this change of perspective, or RS, that may confound outcome assessment of SM programs where an intervention involves social comparisons or cognitive behavioral interventions. It is also important to recognize that RS may be a genuine and intended (albeit implicit) treatment effect. The aims of the present study were to determine the extent to which RS occurs, and to determine if a paper-based questionnaire, the Health Education Impact Questionnaire (HEI-Q) Perspective, can accurately identify the presence of RS.
- Top of page
- SUBJECTS AND METHODS
The questionnaire was sent to 353 individuals. A total of 137 participants returned the questionnaire and, of these, 121 had usable responses for all items (34% response rate). The data from the 121-person subset are reported. The mean age was 63 years (range 33–83 years), 84% were women, and approximately one-third of the sample reported arthritis as their main health problem. The demographics of the interview subset (Table 1) were similar to the overall sample.
Table 1. Demographics of participants*
|Characteristic||Total sample||Interview subsample|
|Age, mean ± SD years||63 ± 11.9||59 ± 11.0|
|Female||112 (84)||34 (87.0)|
|Main health problem|| || |
| Musculoskeletal|| || |
| Arthritis (osteoarthritis and arthritis)||35 (29)||12 (31)|
| Fibromyalgia||9 (7.4)||5 (13)|
| Osteoporosis||12 (9.9)||1 (3)|
| Pain syndrome||5 (4.1)||4 (10)|
| Rheumatoid arthritis||8 (6.6)||4 (10)|
| Other musculoskeletal||12 (9.9)||4 (10)|
| Affective disorder||1 (0.8)||0 (0)|
| Diabetes||3 (2.5)||1 (3)|
| Chronic obstructive airways disease/asthma||4 (3.3)||1 (3)|
| Cardiovascular disease||10 (8.3)||2 (5)|
| Other||12 (9.9)||5 (13)|
| Missing||10 (8.3)||0 (0)|
| Total||121 (100.0)||39 (100.0)|
A total of 39 interviews were conducted. The interviews revealed that all items showed some RS, with RS occurring in approximately half the replies to items (Table 2). Across items, ∼31% had a negative RS, 20% had a positive RS, 32% had no RS, and in ∼17% of the items the presence or absence of RS could not be established. The purposeful sampling for interviews was intended to create a balanced sample across strength of endorsement of response options. Across items, 37% (range 23–59%) scored 0, 38% (28–54%) endorsed either −1 or 1, and 25% (10–38%) endorsed the ends of the scale (−3, −2, 2, or 3). The whole sample that was assessed for RS by questionnaire demonstrated similar proportions. Overall, 87% of participants had RS in at least 1 item.
Table 2. Frequency of occurrence of response shift elicited through questionnaire and through cognitive interview*
| ||Questionnaire answer for subgroup (n = 39)||Interview answer for subgroup (n = 39)||Questionnaire answer for total sample (n = 121)|
|Positive and Active Engagement in Life||38||28||33||36||23||31||10||28||47||25|
|Skill and Technique Acquisition||36||23||41||33||18||28||21||31||38||31|
|Constructive Attitudes and Approaches||44||38||18||41||31||8||21||35||50||16|
|Self-Monitoring and Insight||38||33||28||44||28||18||10||35||37||28|
|Health-Service Navigation– Knowledge||36||46||18||31||44||13||13||36||45||19|
|Health-Service Navigation– Relationships||26||59||15||15||54||8||23||16||69||16|
|Social Integration and Support||26||49||26||15||49||21||15||19||57||24|
Negative RS was most frequent in the Self-Monitoring and Insight item (44% of interviews), implying that, because of participation in the SM program, individuals now realize that they had a much poorer understanding of their health before the program than they thought they did. Typical comments include: “I didn't realize how badly it [rheumatoid arthritis] was affecting me … I was really fearful of what was going to happen in 10–15 years' time. Now I think, ‘That’s not now.' I'm doing things I'd convinced myself I could never do again. I can now live the life I want with some modification”; “I understand now that my only limitations are what I put on myself. The course helped with different techniques and not to dwell on what you can't do but what you can do.”
Another item with a frequent negative RS was Constructive Attitudes and Approaches (41%), implying that because of the SM program, these participants now realize that they had fewer constructive attitudes and approaches to their health before the program than they thought they had. Participant comments include: “I realize now I did let my health problems affect my life; I feel much stronger now because I learned how to manage the pain.”
The 2 items that related to relationships (Social Integration and Support, and Heath Service Navigation–Relationships With Health Professionals) had the lowest negative RS endorsement (both 15%) and among the lowest positive RS endorsement. Conversely, these items had the highest absent RS. A typical absent RS comment for the Social Integration and Support item was, “Basically, my husband and my family … have always been supportive. I knew that anyway.” A typical absent RS comment for the Health-Service Navigation item was, “Yes. Never have any trouble talking to them or discussing things.”
The Constructive Attitudes and Approaches item also had a low positive RS endorsement. The item with the highest positive RS endorsement was the Positive and Active Engagement in Life item (31%), suggesting that approximately one-third of participants found that the program enabled them to see that before the program, they were more positively and actively engaged in life than they thought they were. Typical positive RS comments from this item include: “Well, I never used to think I was doing very much, but when I did the course I realized I was doing a lot”; “When I saw the other people on the course who were far worse than me and they were dealing with things, I realized I wasn't so badly off.”
Closely behind Positive and Active Engagement in Life with high positive RS endorsements were the Skill and Technique Acquisition item and the Emotional Wellbeing item (both 28%), suggesting that approximately one-quarter of the participants found that the program enabled them to see that before the program, they already had a variety of skills and techniques, and that they were less anxious, worried, or depressed than they thought they were. Typical positive RS comments from the Skill and Technique Acquisition item include: “I realized I was doing exercises and not sitting and thinking about the arthritic disease”; “Some of the things we were talking about I realized I already had done that.” Positive RS comments from the Emotional Wellbeing item include: “I really did think that I was anxious or depressed and it's been acknowledged, but oh I wasn't as badly depressed as some of the others. I came away thinking, ‘my lot’s not that bad'”; “Some women in the course were very depressed; they isolate themselves and I am a go-go person. It made me realize that I wasn't depressed at all, just getting older and can't do things as quick as I used to.”
The Health-Directed Behavior item had the largest number of interview outcomes (28%) where it was unclear if RS had occurred. Across interview items, the average number of unclear responses was 17% (Table 2). Unclear RS answers included: “I used to play sports; a lot of bike riding, played in a brass band; but now it [the illness] has slowed me down a lot. In for a major operation. I had started to slow down before the course; had to give up many activities”; “I've done a bit more for myself.”
Across all items in the full sample of 121 participants, only 11 (13%) reported no RS across all items, 18 (22%) reported RS in 1–2 items, 20 (24%) reported RS in 3–4 items, 39 (47%) in 5–6, 28 (34%) in 7–8, and 5 (6%) reported RS across all items (Figure 1). Using the interview as the gold standard, the most accurate items were Positive and Active Engagement in Life, Self-Monitoring and Insight, and Social Integration and Support (overall accuracy score 0.85). These were followed closely by Health Service Navigation–Knowledge, with an overall accuracy score of 0.82. The least accurate item was Health-Directed Behavior (overall accuracy 0.69). The average accuracy across all items was 0.79 (Table 3).
Figure 1. Frequency distribution of endorsement of response shift (RS) items (0 = no evidence of RS across all items, 9 = all 9 items indicate RS) in the total sample (n = 121).
Download figure to PowerPoint
Table 3. Sensitivity and specificity of the Health Education Impact Perspective questionnaire versus interview assessments
|Item||True positive||True negative||False positive||False negative||Sensitivity||Specificity||Overall accuracy|
|Positive and Active Engagement in Life||24||9||4||2||0.92||0.69||0.85|
|Skill and Technique Acquisition||23||7||7||2||0.92||0.50||0.77|
|Constructive Attitudes and Approaches||16||12||8||3||0.84||0.60||0.72|
|Self-Monitoring and Insight||22||11||4||2||0.92||0.73||0.85|
|Social Integration and Support||14||19||2||4||0.78||0.90||0.85|
- Top of page
- SUBJECTS AND METHODS
The measurement of outcomes of health interventions (i.e., pre- and postintervention assessment) relies on participants maintaining a stable perspective from which to rate their status on quantitative questionnaires. This study indicates that ∼87% of participants may have an altered internal standard of measurement on at least 1 relevant outcome of SM courses, which could confound classic pre- and postintervention assessments. This study has also demonstrated that RS can be measured reasonably accurately by questionnaire. The concordance between questionnaire and cognitive interview is high, indicating the HEI-Q Perspective is able to detect negative, absent, and positive RS in participants of SM programs.
The presence of RS therefore has important implications for assessing the benefits that individuals might receive from participating in SM programs. Individuals who have a negative RS (“I now realize that I was worse than I thought I was”) are likely to underrate the severity of their condition in their baseline questionnaire. This could result in a preintervention/postintervention questionnaire-reported underestimation of the treatment effect, thus obscuring a true no treatment effect, or a true positive treatment effect. Participants who have a positive RS (“I now realize I was much better than I thought I was”) are likely to overrate the severity of their condition in their baseline questionnaire. This could result in a preintervention/postintervention questionnaire-reported overestimation of the treatment effect, thus obscuring a true no treatment effect or a true positive treatment effect. When no RS is present, the preintervention/postintervention questionnaire response is likely to be a true reflection of the treatment effect.
RS-related error would reduce the accurate signal (i.e., true change), increase the amount of noise, and make the demonstration of quantitative differences between pre- and postprogram assessments difficult. In settings with substantial RS, larger sample sizes would therefore be required. It may be for this reason that previous studies of SM programs have demonstrated small to no effects for patient-reported outcomes such as quality of life and related issues (3). Warsi et al (3), using a meta-analytical approach across diseases, found that SM programs led to significant outcomes where objective measures were used, i.e., for individuals with hypertension (reduced systolic blood pressure), diabetes (reduced glycosylated hemoglobin levels), and asthma (fewer attacks), but no significant reduction in subjective patient-reported outcomes such as pain or disability for individuals with arthritis. The data from our study suggest that assessment of subjective outcomes through questionnaire may be substantially confounded by RS. The results imply that preintervention/postintervention assessments in a subgroup of persons with absent RS will report accurate patient-reported outcomes (unconfounded by RS); however, for the remainder of the individuals, preintervention/postintervention assessments will be enlarged or diminished (through RS), introducing additional measurement error and difficulty in demonstrating overall group effects.
An important strength of this study is the use of cognitive interviews to validate the HEI-Q Perspective. Adamson et al advocate the importance of combining quantitative and qualitative methods to demonstrate “the complexity of subjective views and [to try] to elucidate the meanings behind recipients' answers to items” (9). Participants in that study completed a standard questionnaire followed by 2 interviews: a structured and then a semistructured interview. The research team's aim was to “investigate perceptions and experiences of health, illness, and healthcare among random samples of white, working-class residents of 3 British cities” (9). Adamson et al call the approach “questerviews,” and their article explores the importance of this method when seeking to understand health and the health care system. Questerviews have parallels, they say, with cognitive interviewing methods to validate questionnaires because both are “concerned with how participants interpret and comprehend questions and the process behind making a response” (9).
Cognitive interviewing is used primarily to identify problems with questionnaires that cannot be identified through statistical psychometric approaches. Harris-Kojetin et al used cognitive interviewing in their Consumer Assessment of Health Plans study: “The intent of cognitive testing is to examine the question-answering process to identify and address errors being introduced into the process. This knowledge then can be used to develop better survey questions, thereby improving the accuracy and reliability of survey responses” (10).
In our study, we used the cognitive interviewing method to determine if the HEI-Q Perspective questionnaire has items that make sense, can be understood by a wide range of the population, is interpreted as intended, and, most importantly, can accurately detect RS. The interviews revealed lower overall accuracy of the Health-Directed Behavior item (“I did fewer/more healthy activities or behaviors”). This item also had the highest number of unclear interview answers, which may be an indication of ambiguity in, or misinterpretation of, the question, and it may not have been detected without the cognitive interviews.
The analytical approach taken in this study is highly conservative. The RS items were scored using a 7-point scale from −3 to 0 to 3. However, we collapsed the results into dichotomous outcomes (presence or absence of RS) to examine test accuracy. The interview approach was simple and passive and was intended to allow spontaneous quotes that might reveal RS. If clear quotes were not elicited, a participant's score was categorized as unclear, and their score contributed to false positive and false negative components of the analyses. The overall specificity (0.74) was somewhat lower than the sensitivity (0.79). Although the overall accuracy is reasonable, the conservative approach of only ascribing RS absence when a clear qualitative statement was revealed may deflate the sensitivity score (i.e., the ratio of the number of individuals correctly identified as not having RS and the total number of individuals without RS). A more proactive interview process and the use of the 7-point scale are likely to demonstrate even higher sensitivity and specificity of the scale and are the subject of ongoing research. Taken together, these results indicate that the HEI-Q Perspective questionnaire has high face validity and high construct validity. Future work will need to include stability and reliability.
Several potential limitations of this study require discussion. The cognitive interview first involved reminding respondents of their initial answer, and then passively asking them, “Can you tell me more about that answer?” Although the structure of the open question partially protects against confirmation bias, the unblinded interview cannot exclude reporting bias. Importantly, across most items, respondents appeared to understand what the question was intending to communicate. In this study, we did not intend to identify the source of RS (e.g., whether or not it related to social comparison). Furthermore, the specific type of RS was not sought after in the interviews. Despite this, the interview content suggested that recalibration was occurring (e.g., “It made me realize that I wasn't depressed at all”). It is possible that the cognitive nature of the items (e.g., fewer/more/poorer) was more likely to elicit recalibration RS rather than reevaluation or reconceptualization RS.
There was also variability in the lengths of time between the end of the course, completion of the questionnaire, and the interviews. Although this provides a potential for recall error, participants seemed confident and clear about their answers. Had the questionnaires been sent out and interviews conducted within a shorter period, the results may have been more evident, with fewer responses in the unclear category. A further potential limitation of this study, as with all quality of life studies, is that a true gold standard does not exist, and therefore a surrogate was used. Although the cognitive interview approach has appealing face validity, replication of the findings of the present study is important. A final potential limitation of the study is the sample used. Participants were self selected and chose to promptly return a somewhat complex questionnaire to researchers. Only approximately one-third of the original respondents returned the questionnaire. This was not surprising because there were no incentives or active followup (reminder letters or phone calls) and no tracking of individuals who might have changed addresses. This may have biased the sample to those who were stronger endorsers of the course, i.e., those with greater negative, greater positive, or greater true change as a result of the course. The distribution of RS in unselected samples may well be different, and replication in other populations is required.
Overall, the results of this study suggest that RS may introduce substantial error into preintervention/postintervention assessments. We have identified a large proportion of participants who had a change of perspective in critical areas affected by the program. For studies where objective disease outcomes (e.g., glycosylated hemoglobin, blood pressure, or asthma) are used to assess pre- and postintervention effects, the influence of RS is nonexistent. For programs that are assessed through subjective outcomes (e.g., pain, disability, and psychological states), the presence of RS may introduce substantial measurement error into preintervention/postintervention assessments. This change, or RS, is detectable by the HEI-Q Perspective and may represent intellectual and emotional shifts that permit individuals to better cope with their disease, seek additional care, and be better equipped to manage themselves and their environment. This would suggest that RS is an important and valuable outcome of SM courses. Ongoing research is required to empirically explore if adjustment for positive, negative, and absent RS does affect the substantive conclusions from health-education program evaluations. Until RS is accounted for in outcome assessment, many effects of SM programs may be obscured, leading to false conclusions that the intervention has small or no effects.