SEARCH

SEARCH BY CITATION

Abstract

  1. Top of page
  2. Abstract
  3. Engagement Research in Advertising
  4. Objectives
  5. Method
  6. Results
  7. Physical Activity
  8. Preventing Falls in the Elderly
  9. High Blood Pressure Screening
  10. Nutrition
  11. Tobacco Cessation
  12. Colorectal Cancer Screening
  13. Talking to Children About Drugs
  14. Reliability
  15. Factor Structure
  16. Predictive Validity
  17. Discussion
  18. References

A scale for measuring the engagement properties of eHealth content was adapted from commercial advertising research. We define engagement as the process of involving users in health content in ways that motivate and lead to health behavior change. Complete responses were obtained from 230/260 participants exposed to health content from seven content areas in online remote testing. After viewing each of three randomly assigned health content areas, the participants completed two questionnaires. The first one assessed the appropriateness, applicability, motivation, and intentions to change or engage in health behaviors relevant to the set of content components displayed for that health topic. The second questionnaire was the eHealth Engagement Scale in which participants rated each of 12 descriptors on a 5-point Likert scale. Internal reliability of each of the two multi-item subscales of the Engagement Scale was .878 for Involving and .805 for Credible. A 4-factor solution eliminating three of the original 12 word descriptors was found to be the superior in the subsequent analysis of predictive validity. The eHealth Engagement Scale may prove to be an important mediator of user retention of information, intentions to change, and ultimately efforts to undertake and achieve behavior change.

“eHealth” is a broad term use to refer to an array of existing and evolving digital resources and practices to support health and health care. eHealth resources are described as those tools that “enable consumers, patients, and informal caregivers to gather information, make healthcare decisions, communicate with healthcare providers, manage chronic disease, and engage in other health-related activities.” Examples of these tools include ones that provide access to and use of health information, support health behavior change and disease prevention, allow individuals to monitor and manage their own health, online communities, various types of decision-making aids, methods to monitor and manage diseases and chronic conditions, and electronic and personal health records (Baur, Deering and Hsu, 2000; US Department of Health and Human Services (HHS), 2006). A comprehensive review of eHealth tools found substantive evidence that eHealth tools can be effective. However, evidence regarding people's access to these tools; their availability, appropriateness and acceptability; and the applicability of the content for leading to actual behavior changes are less clear (HHS; 2006).

An assessment of 273 health websites that targeted seven health behaviors used the 5As approach (Ask, Advise, Assess, provide Anticipatory Guidance and Arrange Follow-Up; Cummins et al, 2003) to determine how well each of them met current behavior change science benchmarks. The investigators found 25% used three or more of the 5 As and very low (<20%) participation rates among the general population on these types of sites (Evers et al, 2005). The authors called for research both to improve participation rates and use of eHealth internet sites and also to enhance the quality of the eHealth interventions to promote changes in health behaviors.

The research described in this report focuses on a methodology to assess a parameter that may underlie participation, use and impact of eHealth tools: engagement. Engagement is a term that has gained currency in the advertising research field as researchers and advertisers attempt to improve the efficiency and effectiveness of advertising. A working definition of engagement proposed by the Advertising Research Federation is: Engagement is turning on a prospect to a brand idea enhanced by the surrounding context (Creamer, 2006). Dr. Joe Plummer, chief research officer of the Advertising Research Foundation, has stated;

“… engagement starts from the vantage point of the customer - either current or potential. By starting there and thinking about whether they are engaged, how can we engage them, and how can we help them achieve their desires, goals or needs, then the risk of not being invited in goes away because you become more relevant. It's about providing messages, services and advertising storytelling in a way that resonates” (Kalehoff, 2006).

In the eHealth and health behavior change arena we can transform this notion into one where we define engagement as: the process of involving users in health content in ways that motivate and lead to health behavior change. This process is one that can be influenced by a number of variables including the information architecture and usability of a web site (Danaher, McKay and Seeley, 2005), the structure and format of the content itself and various user characteristics and motivations that are brought to the task.

Engagement Research in Advertising

  1. Top of page
  2. Abstract
  3. Engagement Research in Advertising
  4. Objectives
  5. Method
  6. Results
  7. Physical Activity
  8. Preventing Falls in the Elderly
  9. High Blood Pressure Screening
  10. Nutrition
  11. Tobacco Cessation
  12. Colorectal Cancer Screening
  13. Talking to Children About Drugs
  14. Reliability
  15. Factor Structure
  16. Predictive Validity
  17. Discussion
  18. References

In the commercial sector, advertisers have been exploring ways to improve audience responses to advertisements by selectively placing them in TV programs that are in a property described as program involvement or engagement. Yet, as Copernicus (2006) points out, “despite the evidence of a positive and potentially profit-producing relationship between program involvement and advertising response, a standardized approach for identifying lower/higher involving programs has not been developed.”

Work conducted by Copernicus has addressed this gap through a comprehensive review of industry and academic research on relationships between involvement and advertising effectiveness. This led to a second phase of research to explore alternative measures of TV program involvement and generate a state-of-the-science measure for use in subsequent research. Based on their literature review, the investigators used both word descriptors such as ‘Funny,’‘Entertaining,’ and ‘True-to-life’ and statements (including ‘Favorite characters in program seem like old friends' and ‘Always thinking about what would happen next') for respondents to rate TV program engagement. An initial list of 50 words and 41 statements were subjected to psychometric evaluation of item homogeneity, response variability and scale range, factor analysis, and their convergent and predictive validity. Their analysis led them to offer both a 12-item word descriptor list and a battery of 10 statements that reflect four underlying content dimensions or factors that were consistent with the literature. These four factors and their associated descriptor items were Involving (Absorbing, Attention-Grabbing, Stimulating, Surprising, Suspenseful, Thought-Provoking and Clever), Credible (Convincing, Balanced and Believable), Negative Feelings (Not Dull) and Amusing/Friendly (Hip/Cool). Both measurement instruments were found to be statistically valid and predictive measures of program involvement and highly predictive of advertising response–the immediate effects of most interest in their work.

Objectives

  1. Top of page
  2. Abstract
  3. Engagement Research in Advertising
  4. Objectives
  5. Method
  6. Results
  7. Physical Activity
  8. Preventing Falls in the Elderly
  9. High Blood Pressure Screening
  10. Nutrition
  11. Tobacco Cessation
  12. Colorectal Cancer Screening
  13. Talking to Children About Drugs
  14. Reliability
  15. Factor Structure
  16. Predictive Validity
  17. Discussion
  18. References

The current research sought to measure the engagement properties of eHealth materials within the larger context of efforts to redesign a health information web site (healthfinder®; www.healthfinder.gov). We were interested in determining whether a process for measuring engagement in the commercial sector could be validly used for eHealth materials. This report presents data on the psychometric properties of the engagement scale, particularly:

  • the variability of results achieved across numerous stimulus conditions (i.e., whether it is sensitive to variations in user perceptions of different materials),

  • the internal consistency of the scale,

  • its underlying factor structure, and,

  • its association with immediate outcomes (intentions and self-efficacy to change related health behaviors after viewing content).

Method

  1. Top of page
  2. Abstract
  3. Engagement Research in Advertising
  4. Objectives
  5. Method
  6. Results
  7. Physical Activity
  8. Preventing Falls in the Elderly
  9. High Blood Pressure Screening
  10. Nutrition
  11. Tobacco Cessation
  12. Colorectal Cancer Screening
  13. Talking to Children About Drugs
  14. Reliability
  15. Factor Structure
  16. Predictive Validity
  17. Discussion
  18. References

The eHealth engagement scale was constructed to test a prototype for presenting health content on the Office of Disease Prevention and Health Promotion (HHS) healthfinder® website (http://heatlhfinder.gov) that would be useful for people who are interested in changing specific health behaviors. Nine health topics were covered by the content: overweight/obesity, physical activity, preventing falls in the elderly, high blood pressure screening, influenza immunization, nutrition, tobacco cessation, colorectal cancer screening, and talking to children about drugs. Details of how the content was developed, background studies that led up to the prototype evaluation, and the complete methodology are available in a technical report (Z-Tech Corporation, 2007).

For the present study we chose to employ the 12-item descriptor list developed in the Copernicus research. We reviewed their battery of 10 statements and judged that they were too tailored to television watching to allow for meaningful use in viewing and responding to the static eHealth content (screen shots) used in this study (although we note that the statements might be useful in assessing the engagement properties of video eHealth content and other formats with narrative elements). The 12 items included were: Absorbing, Attention-Grabbing, Stimulating, Surprising, Suspenseful, Thought-Provoking, Clever, Convincing, Balanced, Believable, (Not) Dull and Hip/Cool.

The prototype testing included the recruitment of 260 individuals to participate in an online study of prevention content on the Internet. Participants were recruited by telephone and e-mail solicitation to achieve an even distribution across the 10 Health and Human Service regions in the United States. Study eligibility was determined by a respondent answering “yes” to the following questions:

  • Have you ever used the Internet?

  • In the past 12 months, have you used the Internet to send an e-mail message?

  • Are you between the ages of 18–65?

A stratified sampling strategy was also employed with the following parameters:

  • A 60:40 ratio of women to men to reflect documented gender differences in health information-seeking behaviors

  • 10–15 percent African American

  • 10–15 percent Hispanic/Latino

  • 3–5 percent Asian or Pacific Islander

  • 10 percent high school, some high school, or GED

  • 30 percent some college

  • 30 percent college graduate

  • 30 percent college graduate plus additional training (i.e., more college, advanced degrees, certifications, continuing education)

Participants in the study received an e-mail invitation containing the URL of the Web application and a unique ID with which to log on to complete the study. Once logged on, participants read a short introduction explaining that the study was being conducted on behalf of the US government. They were then presented with the consent form and asked to enter their unique ID again as proof of consent. All study procedures and questionnaires were reviewed and approved by the Z-Tech Corporation Institutional Review Board and the Office of Management and Budget.

As depicted in Figure 1, each participant was randomly assigned to review content from three (3) of the nine health topic areas. The presentation order for topics was randomly predetermined to control for any order effects. An a priori hypothesis was that people may differ in their assessment of health content based on their reasons for searching for it. Thus, each participant was asked to respond to their assigned health topics from the perspective of one of the following three user segments:

image

Figure 1. Study Protocol for Each Participant

Download figure to PowerPoint

  • Segment 1: Those seeking information about a health problem for themselves or someone they know.

  • Segment 2: Those wanting to find out if they or someone they know has a health problem or reason to be concerned about a health problem.

  • Segment 3: Those seeking information to prevent the onset of a health problem.

To aid in the identification with the randomly assigned user segment, participants were then exposed to a screen containing a scenario representative of their assigned audience segment. The scenario was designed to provide a specific perspective from which participants would answer questions about the health information associated with a particular topic. For example, participants assigned to the second user segment, those wanting to find out if they or someone they know has a health problem or reason to be concerned about a health problem, were exposed to the following scenario for the high blood pressure screening topic:

Let's look through some Web pages about High Blood Pressure Screening.

Imagine that you need to find out if you should be screened for high blood pressure.

As you look through these pages, remember,

“I want to find out if I should be screened for high blood pressure.”

This scenario was reinforced by the Web application, which displayed the following message at the top of each content component page of the high blood pressure screening topic: As you explore this page, remember, “I want to find out if I should be screened for high blood pressure.” [Preliminary analysis of the usefulness and overall engagement of content by user segment showed no significant variation among the three for any of the health topic areas. Subsequently, segment assignment was ignored and all responses were combined in the data reported here.]

The content for each health topic was presented in three screen shots that presented content specific to that health area from three of nine health information components developed in our earlier work. These components included:

  • Understanding the problem

  • Importance of preventing the problem

  • Risk factors: what makes people more at risk for the problem

  • Assessment of personal risk

  • Overcoming barriers

  • Motivators

  • Strategies for preventing the problem

  • Community Resources/Locator

  • Guidelines/Standards/ Recommendations

The three information components that were addressed in each content area were determined in a formative study by a card-sorting task done by 81 participants not included in this study. These participants ranked the 3 “most useful” and 3 “least useful” information components for each health content area using actual expert advice for each one. Participants then rank-ordered the “most useful” cards and were questioned to assess their appropriateness, acceptability, applicability, and engagement properties (Note: this also served as a pre-test of these questions for inclusion in the current study).

At the bottom of each screen, participants were asked to respond to two 5-point Likert scale statements before proceeding to the next component (RQ 1 and 2 in Figure 1). These statements were constructed to assess the perceived usefulness of the information and the confidence or self-efficacy one had to address the individual health topic, as shown in the following example from the high blood pressure screening topic:

  • This page provided important information about preventing high blood pressure. (1–Strongly Agree to 5–Strongly Disagree).

  • I feel more confident I can prevent high blood pressure. (1–Strongly Agree to 5–Strongly Disagree).

This sequence of presentations of each component with the two Likert items was repeated for the other two health topics.

After completing each of the three health content areas, the participants completed two questionnaires. The first one assessed the appropriateness, applicability, motivation, and intentions to change or engage in health behaviors relevant to the set of content components displayed for that health topic. The second questionnaire was the eHealth Engagement Scale in which participants rated each of the 12 descriptors on a 5-point Likert scale (1 = Strongly Agree, 5 = Strongly Disagree) for each health content area. After completing these screens, participants were offered the opportunity to provide open-ended comments about that health topic.

A final set of questions about their demographic characteristics and Internet usage and connectivity were asked after the third set of questionnaires was completed. When these questions were answered, the participants received a confirmation page thanking them for their time.

To increase the completion rate, the Web application helped the participants to complete the questionnaires by allowing participants to disconnect and login again to finish the study within 30 days, providing system prompts (error messages) if participants inadvertently missed a question, and offering Help Desk support from 10:00 a.m. to 4:00 p.m. Eastern Standard Time, Monday through Friday throughout the study.

The investigators compiled periodic reports on the participants' status that identified those who successfully completed the study and those who had dropped out. The recruitment firm remunerated participants with $50 when they completed the entire study protocol. The recruitment firm also contacted those who dropped out (non-completers) to determine if they wanted to complete the study. Those who wished to continue were provided instructions on how to re-enter the Web application and log into their existing file. If participants abandoned the Web application before completing the questionnaires, their participation did not count toward the 260 responses, and they were not remunerated.

Results

  1. Top of page
  2. Abstract
  3. Engagement Research in Advertising
  4. Objectives
  5. Method
  6. Results
  7. Physical Activity
  8. Preventing Falls in the Elderly
  9. High Blood Pressure Screening
  10. Nutrition
  11. Tobacco Cessation
  12. Colorectal Cancer Screening
  13. Talking to Children About Drugs
  14. Reliability
  15. Factor Structure
  16. Predictive Validity
  17. Discussion
  18. References

A total of 260 participants completed the remote prototype study from a group of 403 qualified participants originally solicited by telephone and e-mail (65% completion rate). The other 143 eligible participants failed to log in or, once logged in, did not complete the study and were replaced. Of these 260 completed protocols, 14 were subsequently removed from further analyses as their time to complete the protocol exceeded 3 SDs from the mean response time and we were concerned as to what confounding variables may have been introduced during these sessions. Another 15 cases were removed when an item response check found a response set where each respondent had checked the identical response to all questions. Finally, one case was eliminated due to an inadvertent exposure to the same topic area twice. A chi-square analysis of the demographic characteristics of the original sample of 260 with the final sample provided no evidence of any significant change in their composition. Thus, the analyses are based on 230 (88%) of the original 260 completed protocols.

The total time to complete the online protocol ranged from 3:01 minutes to 1322:52 minutes (Median = 23 : 42 minutes). Table 1 shows the age and income characteristics of the final sample (N = 230).

Table 1.  Description of Participants (By Gender and Household Income Level)
Description of Participants (By Gender and Household Income Level)
Gender × IncomeAge
 18–2930–4950–66
Male (Total)164630
 <$20,000152
 $20,001–$40,0000310
 $40,001–$70,0008138
 $70,001–$100,0002114
 >$100,0005146
Female (Total)298029
 <$20,000562
 $20,001–$40,00081812
 $40,001–$70,0008234
 $70,001–$100,0005208
 >$100,0003133
Total4512659

The final sample resulted in a 60:40 split of women (n = 138) to men (n = 92), 55% of the sample was between the ages of 30-49 years old, and it skewed towards the upper levels of household income (HHI) with 19% (44) reporting HHI > $100,000 and 9% (21) reporting HHI < $20,000 (Median = $40,001 −$70,000).

Demographic characteristics were found to affect the ratings of all of the 12 items on one or more health topics. However, none of these differences occurred in any systematic manner across or within the topic areas or eHealth Engagement Scale items. Given the large number of comparisons that were made, what may be surprising is that for only two topic areas - Influenza Immunization and Obesity - were no significant differences noted among any demographic subgroups on any of these engagement items despite a broad range of ratings for them. The significant differences among demographic groups on specific eHealth Engagement Scale items were:

Physical Activity

  1. Top of page
  2. Abstract
  3. Engagement Research in Advertising
  4. Objectives
  5. Method
  6. Results
  7. Physical Activity
  8. Preventing Falls in the Elderly
  9. High Blood Pressure Screening
  10. Nutrition
  11. Tobacco Cessation
  12. Colorectal Cancer Screening
  13. Talking to Children About Drugs
  14. Reliability
  15. Factor Structure
  16. Predictive Validity
  17. Discussion
  18. References
  • 50-65 year-olds found the content less ‘absorbing’ than 30-49 year-olds.

  • People with household incomes (HHI) < $20,000 were more likely to rate the content more ‘suspenseful’ than people with HHI > $70,000.

Preventing Falls in the Elderly

  1. Top of page
  2. Abstract
  3. Engagement Research in Advertising
  4. Objectives
  5. Method
  6. Results
  7. Physical Activity
  8. Preventing Falls in the Elderly
  9. High Blood Pressure Screening
  10. Nutrition
  11. Tobacco Cessation
  12. Colorectal Cancer Screening
  13. Talking to Children About Drugs
  14. Reliability
  15. Factor Structure
  16. Predictive Validity
  17. Discussion
  18. References
  • 30-65 year-olds viewed the content as more ‘believable’ than people 18-29 years-old.

  • Participants with less than a high school education found it ‘more believable,’‘clever,’‘not dull,’ and more ‘balanced’ than those with higher educational attainment.

High Blood Pressure Screening

  1. Top of page
  2. Abstract
  3. Engagement Research in Advertising
  4. Objectives
  5. Method
  6. Results
  7. Physical Activity
  8. Preventing Falls in the Elderly
  9. High Blood Pressure Screening
  10. Nutrition
  11. Tobacco Cessation
  12. Colorectal Cancer Screening
  13. Talking to Children About Drugs
  14. Reliability
  15. Factor Structure
  16. Predictive Validity
  17. Discussion
  18. References
  • Women found this content less ‘suspenseful’ and ‘hip, cool’ than men.

  • People with high school educations rated it as less ‘dull’ than those with a college education.

  • Participants with a HHI of $20,000 - $40,000 found the content to be more ‘absorbing,’‘stimulating’ and ‘surprising’ than people with either lower of higher HHI.

Tobacco Cessation

  1. Top of page
  2. Abstract
  3. Engagement Research in Advertising
  4. Objectives
  5. Method
  6. Results
  7. Physical Activity
  8. Preventing Falls in the Elderly
  9. High Blood Pressure Screening
  10. Nutrition
  11. Tobacco Cessation
  12. Colorectal Cancer Screening
  13. Talking to Children About Drugs
  14. Reliability
  15. Factor Structure
  16. Predictive Validity
  17. Discussion
  18. References
  • Men rated the content as more ‘suspenseful’ than women.

  • People with high school educations rated it as more ‘attention-grabbing,’‘surprising’ and ‘clever’ than people with more than a college education.

Colorectal Cancer Screening

  1. Top of page
  2. Abstract
  3. Engagement Research in Advertising
  4. Objectives
  5. Method
  6. Results
  7. Physical Activity
  8. Preventing Falls in the Elderly
  9. High Blood Pressure Screening
  10. Nutrition
  11. Tobacco Cessation
  12. Colorectal Cancer Screening
  13. Talking to Children About Drugs
  14. Reliability
  15. Factor Structure
  16. Predictive Validity
  17. Discussion
  18. References
  • Participants with some college were more likely than people with a college degree or higher educational attainment level to rate the content as ‘thought-provoking’ and ‘convincing.’

Talking to Children About Drugs

  1. Top of page
  2. Abstract
  3. Engagement Research in Advertising
  4. Objectives
  5. Method
  6. Results
  7. Physical Activity
  8. Preventing Falls in the Elderly
  9. High Blood Pressure Screening
  10. Nutrition
  11. Tobacco Cessation
  12. Colorectal Cancer Screening
  13. Talking to Children About Drugs
  14. Reliability
  15. Factor Structure
  16. Predictive Validity
  17. Discussion
  18. References
  • People with a high school education rated the content as more ‘attention-grabbing’ than those with higher educational attainment.

  • Participants with HHI < $20,000 found the material to be more ‘convincing’ than did those with HHI > $100,000.

The dimensions of clever (5), suspenseful (3), and absorbing (3) accounted for the most number of significant differences among demographic groups for one or more health topics. These dimensions are closely associated with the attentive (or “involving”) properties of the materials (although “attention-grabbing” was only rated differently in two instances - once in association with “clever” and in the other with “surprising”). Thought-provoking, stimulating, and convincing were rated differently by subgroups on at least two occasions. “Believable” and “not dull” were each rated differently once by subgroups.

Reliability

  1. Top of page
  2. Abstract
  3. Engagement Research in Advertising
  4. Objectives
  5. Method
  6. Results
  7. Physical Activity
  8. Preventing Falls in the Elderly
  9. High Blood Pressure Screening
  10. Nutrition
  11. Tobacco Cessation
  12. Colorectal Cancer Screening
  13. Talking to Children About Drugs
  14. Reliability
  15. Factor Structure
  16. Predictive Validity
  17. Discussion
  18. References

In the original research by Copernicus on which the eHealth Engagement Scale is based, two primary factors were Involving (Absorbing, Attention-Grabbing, Stimulating, Surprising, Suspenseful, Thought-Provoking and Clever) and Credible (Convincing, Balanced and Believable). In the present dataset, Cronbach's alpha was calculated for each of the two subscales and was found to be .878 for Involving and .805 for Credible. To determine whether either of the two scales could have the number of items reduced to shorten administration, we further examined the item-total correlations. The best candidate for deletion was Suspenseful; its correlation with other items ranged from r = .287 to .553 and the item-total correlation = .549. The impact on eliminating the item from the Involving scale had a minimal effect on internal consistency (α = .875 from .878). Because the other two original factors (Negative Feelings, Amusing/Friendly) consisted of one item each, we did not conduct any internal reliability tests on them.

Factor Structure

  1. Top of page
  2. Abstract
  3. Engagement Research in Advertising
  4. Objectives
  5. Method
  6. Results
  7. Physical Activity
  8. Preventing Falls in the Elderly
  9. High Blood Pressure Screening
  10. Nutrition
  11. Tobacco Cessation
  12. Colorectal Cancer Screening
  13. Talking to Children About Drugs
  14. Reliability
  15. Factor Structure
  16. Predictive Validity
  17. Discussion
  18. References

A series of factor analyses were performed using the SYSTAT RAMONA module. Our attempt to conduct a confirmatory factor analysis of the original factor structure of the engagement tool as developed for advertising purposes was not very successful. The overall fit of the model was poor, χ2(50, N = 682) = 689.08, p <. 001; RMSEA = .136 (.127, .145). This was primarily due to smaller correlations of the Suspenseful item with the rest of the Involving factor indicators, as well as moderate correlations of Involving indicators with indicators of other factors - specifically between the Clever and Hip, Cool item (r = .626) and between the Thought-provoking and Convincing item (r = .625). The moderate to high correlations among factors (.383–.795) also suggested that these factors were not as distinctive in this context as suggested by the previous commercial work.

Consequently, when we deleted the Suspenseful, Thought-Provoking and Clever items from the Involving factor, the fit of the overall model improved (see Figure 2). This modified four-factor model had a reasonable overall fit, χ2(23, N = 682) = 120.63, p <. 001; RMSEA = .079 (.065, .093). By deleting these three items from the Involving factor, the Cronbach's alpha for this factor modestly decreased from .878 to .853.

image

Figure 2. Path Model for Revised 4-Factor Structure of the E-Health Engagement Tool

Download figure to PowerPoint

We also performed an exploratory factor analysis on the current data using the Maximum Likelihood extraction method with an Oblimin rotation and Kaiser Normalization using SPSS. This resulted in a two-factor solution where Factor 1 resembled the original Involving factor and included Absorbing, Attention-grabbing, Surprising, Thought-provoking, Convincing, Balanced, Believable, and Not Dull. The second factor consisted of Surprising, Suspenseful, Clever, Hip/Cool, reflecting a dimension of “Stimulating.” However, both factors contained overlapping items for which loadings are greater than .5 (Absorbing, Attention-Grabbing, Clever and Not Dull).

Predictive Validity

  1. Top of page
  2. Abstract
  3. Engagement Research in Advertising
  4. Objectives
  5. Method
  6. Results
  7. Physical Activity
  8. Preventing Falls in the Elderly
  9. High Blood Pressure Screening
  10. Nutrition
  11. Tobacco Cessation
  12. Colorectal Cancer Screening
  13. Talking to Children About Drugs
  14. Reliability
  15. Factor Structure
  16. Predictive Validity
  17. Discussion
  18. References

We then examined each of the factor solutions we had developed (the 12-item four-factor solution, the modified (9-item) four-factor solution, and the 12-item exploratory two-factor solution) to determine which one would best meet our criteria for predicting immediate outcome ratings. The three criterion items we selected from the questionnaire were:

  • The information made me feel more confident that I can do something.

  • The information made me feel more prepared to do something.

  • The information made me feel more prepared to do something in the next month.

As shown in Figure 3, the modified 4-factor structure proved to provide the best fit with the data, χ2(46, N = 682) = 183.498, p <. 001; RMSEA = .066 (.056, .076). Both the Involving and Credible factors significantly predicted the Outcome factor (the three items above used as indicators). The four factors together accounted for approximately 56% of the outcome variance.

image

Figure 3. Path Model of Revised 4-Factor Solution to Explain Immediate Outcome Variables

Download figure to PowerPoint

Discussion

  1. Top of page
  2. Abstract
  3. Engagement Research in Advertising
  4. Objectives
  5. Method
  6. Results
  7. Physical Activity
  8. Preventing Falls in the Elderly
  9. High Blood Pressure Screening
  10. Nutrition
  11. Tobacco Cessation
  12. Colorectal Cancer Screening
  13. Talking to Children About Drugs
  14. Reliability
  15. Factor Structure
  16. Predictive Validity
  17. Discussion
  18. References

The results of this study demonstrate that a measure of engagement developed for commercial television advertising viewing to predict advertising effectiveness can be used to assess user engagement with eHealth content. In addition, we have shown that a modified 9-item version of the original scale accounted for 56% of the variance in self-reported confidence and intentions to act across nine different health topics.

Although the complete set of tables are not displayed in this report (see Z-Tech Corporation, 2007 for all of the tables), we did find considerable variability of results for reported levels of engagement across the various components of content presentation and health topics. In some instances, differences in self-reported engagement with specific content were noted among various socio-demographic groups.

We also found high internal consistency for the 12-item and 9-item scales that conforms with the results found in commercial settings and adds to the level of confidence that the scale items share a common focus.

This common focus on the concept we term ‘engagement’ has an underlying factor structure that is also consistent with previous work. Analysis of our data did not exactly replicate the factor structure reported in the Copernicus (2006) research. An exploratory analysis of the current dataset suggested a 2-factor solution, and a slight adjustment of the Copernicus model led to a modified 4-factor solution. This modified 4-factor solution involved eliminating three of the word descriptors and was found in the subsequent analysis of predictive validity to be superior to the 2-factor solution.

We have chosen to emphasize the modified 4-factor solution in our interpretation of results over the 2-factor solution for two primary reasons. The first is that the context in which the data were collected for these analyses may have restricted the dimensionality of user response by the nature of the presentation materials (i.e., some factors were not adequately tapped by information presented through wire frames of static web content). This presentation format was necessitated by other demands of the research protocol, but clearly omitted use of more dynamic (audio, video) or interactive elements that might result in a different pattern of ratings. Secondly, we do not want to eliminate the opportunities to further evaluate and study the effects of the other two factors (Negative Feelings, Amusing/Friendly) when measuring engagement with eHealth sites as more and varied presentation and design elements are introduced to users. This position was reinforced by the high level of predictive validity we found for the modified 4-factor solution for self-efficacy (confidence) and intentions to change health behaviors.

This modified 4-factor model was subsequently used to analyze the engagement data for each of the nine health content areas separately. Our results (not shown here) found that the factor structure (factor loadings of the items) of the modified 4-factor solution was stable across all nine topic areas. This suggests that the 9-item revised e-Health Engagement Scale is a robust tool to operationalize this concept across a variety of health topic areas. However, the modified model should be validated using independent samples to determine if it can be generalized across various eHealth formats and media.

Even with our use of static content (screen shots) in this study, differences in engagement levels among health content areas were detected using the scale. We suggest that stronger levels of engagement should be expected and be possible with multimedia (sound and video) and interactive content. The use of other media in subsequent studies would create opportunities for user experiences that are more fun, “cool” and absorbing and provide new dimensions for the tailoring of materials. We also did not assess our participants' prior knowledge of the health content areas before their exposure to our material. We might expect higher engagement scores will be found among participants where content is relevant to them, yet relatively novel, as opposed to content that is either not relevant to their interests or needs, or is already well understood.

As research on the applicability, acceptability, appropriateness and effectiveness of eHealth continues to deepen in some areas and extend into new modalities (i.e., digital social networks, mobile devices), we offer that a measure of user engagement may prove to be an important mediator of user retention of information, intentions to change, and ultimately efforts to undertake and achieve behavior change. We look forward to the discussion and research as to how levels of engagement might be enhanced through various means. Such influences on user engagement may include how websites use different types of information architecture design, the value of tailoring and targeting content, scheduling of homework tasks and the tracking of progress, roles of media and interactivity, structure and value of community and social components (e.g., Web forums, peer ratings of content), and impact of email and/or mobile phone features (c.f., Danaher, McKay & Seeley, 2005).

References

  1. Top of page
  2. Abstract
  3. Engagement Research in Advertising
  4. Objectives
  5. Method
  6. Results
  7. Physical Activity
  8. Preventing Falls in the Elderly
  9. High Blood Pressure Screening
  10. Nutrition
  11. Tobacco Cessation
  12. Colorectal Cancer Screening
  13. Talking to Children About Drugs
  14. Reliability
  15. Factor Structure
  16. Predictive Validity
  17. Discussion
  18. References