SEARCH

SEARCH BY CITATION

Abstract

  1. Top of page
  2. Abstract
  3. Introduction
  4. Literature Review
  5. Methodology
  6. Results
  7. Analysis
  8. General Discussion
  9. Limitations and Future Work
  10. Conclusion
  11. Acknowledgements
  12. References
  13. Appendix A: Items from Existing Instruments Incorporated into Engagement Survey Instrument

Increased emphasis on user experiences with technology demonstrates that systems must be not only usable, but engaging. Engagement, defined as a quality of user experience, is a multidimensional construct characterized by aesthetic appeal, novelty, perceived challenge, feedback and control, attention, motivation, and affect. To measure engagement, we developed a multidimensional instrument and surveyed 440 online shoppers to assess its reliability and construct validity. Results of exploratory factor analysis showed that engagement is comprised of six distinct factors: perceived usability, aesthetics, focused attention, involvement, novelty, and endurability.


Introduction

  1. Top of page
  2. Abstract
  3. Introduction
  4. Literature Review
  5. Methodology
  6. Results
  7. Analysis
  8. General Discussion
  9. Limitations and Future Work
  10. Conclusion
  11. Acknowledgements
  12. References
  13. Appendix A: Items from Existing Instruments Incorporated into Engagement Survey Instrument

More than a decade ago, Brenda Laurel wrote that ‘when we sit back and contemplate the complexity involved in creating first-person experiences [with technology], we are tempted to see them as a luxury, not a necessity’ (Laurel, 1993, p. 118). Today creating applications that go beyond being merely functional is imperative and gaining increasing amounts of attention (Overbeeke, Djajadiningrat, Hummels, Wensveen, & Frens, 2003). This is not surprising considering that technology pervades all aspects of our lives, from online banking and shopping, education, travel, and entertainment. This is coupled with the fact that many of today's users are ‘untethered,’ accessing online information from mobile devices such as cell phones and PDAs (Madden, 2007). In addition, phenomena such as You Tube are propelling society towards a ‘clip culture’ (The Economist, 2006) where experiences take place in short bursts in front of a screen. The proliferation of technology and our dependence on it, as well as our decreasing attention spans, put an onus on the purveyors of technology to ‘make things engaging’ (Overbeeke et al., 2003). While there has been increased emphasis on creating and understanding engagement (Hull & Reid, 2003), little focus has been placed on assessing engaging experiences with technology. Specifically, how do we know that the engagement we have tried to foster with an application has actually taken place? The goal of our research was to develop an instrument to evaluate the level of engagement with online shopping that encompassed the system (e.g. usability), user (e.g. satisfaction), and interface (e.g. aesthetics) aspects of engagement from the users' perspective. This paper describes the process of constructing and evaluating the reliability and construct validity of a metric for assessing engagement.

Literature Review

  1. Top of page
  2. Abstract
  3. Introduction
  4. Literature Review
  5. Methodology
  6. Results
  7. Analysis
  8. General Discussion
  9. Limitations and Future Work
  10. Conclusion
  11. Acknowledgements
  12. References
  13. Appendix A: Items from Existing Instruments Incorporated into Engagement Survey Instrument

Engagement is a quality of users' experiences with technology and is comprised of attention, affect, aesthetics, novelty, interest, control, feedback, challenge, and motivation. This definition is based on recent work that explored users' impressions of their engagement with one of four applications: Web searching, online shopping, video games, and educational webcasts (O'Brien &Toms, 2008), which extended the scope of previous research regarding the composition of engagement (Jacques et al., 1995; Webster & Ho, 1997; Skelley et al., 1994). This work also posited that engagement fits within McCarthy and Wright's (2004) ‘threads’ of experience framework. According to the compositional thread, engagement is a process whereby users pass through stages of becoming engaged, staying engaged, disengaging, and re-engaging. This process is situated in a specific spatio-temporal context, i.e. time and place. The emotional and sensual threads recognize the affective, aesthetic, and interactive attributes of engagement. These attributes likely vary in intensity during the experience depending on the interaction between the user, system, and task at a given point in time. Based on the ‘threads’ of experience and our previous findings, we undertook a longitudinal process to construct a measurement of engagement that was holistic in and multidimensional in nature.

Measuring Engagement

Since engagement is a quality of user experience, the evaluation of engagement hinges on users' perceptions of their performance, the system, the task, and the context. The ‘[user experience] is about technology that fulfils more than just instrumental needs in a way that acknowledges its use as a subjective, situated, complex and dynamic encounter’ (Hassenzahl & Tractinsky, 2006, p. 95). As such, an appropriate method evaluation must tap into users' internal states. In this case, the usual objective measures (e.g. time on task, number of websites visited) would not suffice because they do not measure users' perceptions of their performance or experience. Based on this rationale, an instrument was constructed to extract user-centric data.

There have been few attempts to measure engagement, but of these efforts, surveys have been the most prevalent technique. Webster and Ho (1997) investigated university students' engagement in the classroom with two types of presentation software. Their instrument, comprised of seven items rated on a Likert scale, was also used in studies of web disorientation (Webster & Ahuja, 2004; Webster, Trevino & Ryan, 1993) and multimedia learning and training software (Chapman, 1997; Chapman, Selvarajah, & Webster, 1999). The items pertained to intrinsic interest, attention, novelty, curiosity, and positive affect, but did not incorporate challenge, feedback, control, motivation, or aesthetics. Chapman (1997) examined some of the latter attributes in his study of students' interactions with multimedia software. He illustrated engagement's relationship to software format, but did not dissect the construct itself. Said's (2004) study asked children to circle a face (ranging from happy to sad) that depicted how they felt at various times during an interaction with the video game The Sims. Her approach brought emotional aspects of engagement (as motivation for continuing to play the game) to the forefront.

Overall, past attempts to measure engagement are limited in terms of the breadth and scope with which they define engagement; there is also a lack of replication outside of a specific domain, such educational multimedia or gaming. Certainly there have been no instruments designed to measure engagement in search contexts. Thus, we set out to construct a survey instrument that could be generalized to a variety of search environments. The search environment selected for this research was online shopping.

The Online Shopping Domain

To develop a sound measure of engagement, we concentrated on one information searching domain, with the idea of testing its generalizability it to other application areas in future. Online shopping, defined as ‘the consumers' adoption of the WWW as a means to purchase’ (Shang, Chen & Shen, 2005) was the domain selected for this research for several reasons.

First, online shopping websites must meet minimal usability criteria, otherwise they would be obsolete, but must also be engaging. Similar to physical storefronts, e-commerce websites must attract shoppers and make their experiences engaging enough to return in future. Secondly, research has been carried out on website design and content, user interface features, and user attitudes and behaviours in the online shopping environment with regard to trust, service and product quality, purchase decision making, and shopping motivations and satisfaction (Van der Heijden et al., 2003; Haubl & Trifts, 2000; Zhou, Dai, & Zhang, 2007; Monsuwé, Dellaert, & de Ruyter, 2004). Little is known about engagement in this environment, but, like other technologies, there is increasing emphasis on entertainment in the design of shopping websites (Shang, Chen & Shen, 2005).

Table 1. Comparison of Consumer Decision Process (CDP) with Information Search Process (ISP)
Consumer Decision Process (CDP)Kahlthau's Information Search Process (ISP)
Identify need for productTask Initiation; Topic Selection
Search for informationExploration
Evaluate alternatives by product attributes or factors such as costFormulation
 Collection
PurchaseSearch Closure
Post-purchase evaluationPresentation

Lastly, since we were interested in generalizing the survey instrument to other search environments in future, we needed a domain in which searching was intrinsic to the task. Shopping and information searching share common processes. Specifically, consumer decision making (Te'eni, Carey & Zhang, 2006) resembles basic information search models such as Kuhthau's Information Search Process (ISP) (Kuhlthau, 1991). Table 1 compares the consumer decision process (CDP) with ISP. Although ISP is a complex intermingling of cognition, affect, and behaviour, it is similar to CDP in that it breaks the search process down into stages. In both, the consumer or user must identify a need, be it for a product or bit of information; actively seek information; and evaluate what they find in order to make a decision (e.g. purchase a product, redefine their information need, conclude they have ‘enough’ information). Both processes conclude the act of searching in order to move on to other activities (e.g. make a purchase, begin writing a paper) and include evaluation of the final product or process.

In summary, the study of online shopping engagement represents a domain in which ordinary people conduct purposeful searching. Online shopping is an arena in which engagement has not been examined. Constructing our instrument in this domain made sense in terms of generalizing the scale to other search environment in future, namely because interface design and affect are salient in both general searching and shopping environments, and the process of shopping is akin to basic search models, such as ISP.

Research Goals

The aims of this study were: (a) to construct a survey instrument to measure engagement that included sub-scales for all of the proposed attributes of engagement attention, affect, aesthetics, novelty, interest, control, feedback, challenge, and motivation; and (b) to assess the reliability and construct validity of resulting instrument. This latter goal involved multiple analyses.

Methodology

  1. Top of page
  2. Abstract
  3. Introduction
  4. Literature Review
  5. Methodology
  6. Results
  7. Analysis
  8. General Discussion
  9. Limitations and Future Work
  10. Conclusion
  11. Acknowledgements
  12. References
  13. Appendix A: Items from Existing Instruments Incorporated into Engagement Survey Instrument

We employed a multi-stage process to develop and to evaluate the reliability and validity of our engagement survey instrument. This involved an extensive procedure to develop items for each of the subscales and to assess their internal consistency using Cronbach's alpha. For example, do the items purported to measure the aesthetics measure ‘hang together.’ Finally, we examined the construct validity of the instrument as a whole through exploratory factor analysis. In other words, would the sub-scales for each attribute remain distinct, or would the items from the sub-scales combine to create broader constructs?

Item development

First, a sub-scale was developed for each attribute of engagement: aesthetics, affect, challenge, motivation, interest, novelty, feedback, perceived control, and attention. To systematically develop these sub-scales, existing scales and instruments were examined that have been used in the research literature; items were also derived from the interview transcripts from a previous study (O'Brien & Toms, 2008).

With regard to existing measures of the attributes, this investigation spanned the marketing, psychology, information science, and human-computer interaction literature. Appendix A shows, for each of the sub-scales, the source from which items were derived. In total, 100 items were considered for inclusion in the final scale. These items varied in focus and physical make-up. For example, some were developed to study users' interactions with computers (e.g. users' aesthetic appraisals of websites [Pace, 2004]), whereas others were intended for general environments that may or may not involve computers (e.g. The Subjective Leisure Scale [Unger & Kernan, 1983]). For instruments designed for use in human-computer interaction, some were specific to a particular domain. For instance, the novelty attribute was measured by Mathwick & Rigdon, 2004 and Novak, Hoffman & Yung, 2000 in online shopping, while Huang, 2003; Zhang & von Drann, 2000 assessed it in general web searching. Other measures were not domain specific (e.g. The Situational Motivation Scale [Guay, Vallerand & Blanchard, 2000]). The focus of some existing measures was on people's psychological states, (Witmer & Singer, 1998) or personality traits (Litman & Speilberger, 2003; Reio, 1997), but other items were specific to the computer interface. For instance, Schmidt, et al. (2003) looked at features of computer interfaces that drew or detracted from attention, such as advertisements or pop-up windows. In some cases, the attribute was the sole phenomenon being investigated, as with Lavie and Tractinsky's (2004) aesthetics instrument. Measures of feedback (Schmidt, Bauerly, Liu, & Sridharan, 2003; Zhang & von Drann, 2000), affect (Ghani, Supnick, & Ryan, 2001; Mathwick & Rigdon, 2004; Pace, 2004; Novak, Hoffman, & Yung, 2001), and attention (e.g. Choi & Kim; 2004; Huang, 2003; Novak, Hoffman, & Yung, 2000; Webster & Ho, 1997; Webster, Trevino, & Ryan, 1993), and challenge (Mathwick & Rigdon, 2004; Pace, 2004; Novak, Hoffman, & Yung, 2001) were located in scales that purported to measure broader constructs, such as play and flow.

Since overall impressions of system use influence perceptions of other attributes of experience (DeLone & McLean, 1992), we incorporated items pertaining to ‘loyalty’ to an application (e.g. Choi & Kim, 2001) and ‘intention to use’ (Webster & Ahuja, 2004) into the overall scale. We also felt it was important to acknowledge previous attempts to measure engagement and incorporated Webster & Ho's (1997) seven items.

The words of users were considered to have merit as potential items (Kelly, 2005). Selected phrases and statements were extracted from the interview transcripts of a previous study (O'Brien & Toms, 2008). This procedure resulted in 350 statements.

In total, 450 items (100 from existing instruments,350 from the interview transcripts) were included. Items were initially screened according to their potential to be used in multiple computer domains. First, those specific to one domain and duplicate items were removed. Second, items needed to be uniform in their formatting. Some existing measures and interview citations consisted of one-word descriptions. For instance, Park, Choi, & Kim (2004) used adjectives (e.g. balanced, complex) to derive 13 aesthetic dimensions, while Huang (2003) plotted users' perceptions of websites along a continuum of opposite pairs of terms (e.g. nice-awful or entertaining-weary). As might be expected with human speech, interview statements were not always complete sentences: e.g. ‘not all clumped together’ (referring to interface layout) and ‘just engrossed in what I was doing.’

The items were further examined with respect to their wording and formatting. Ambiguous terms (e.g. could, should, or might) and vague quantifiers (e.g. occasionally, most, or very) were removed to avoid confounds based on different interpretations by respondents. In addition, the ‘tone’ of the questions was checked to ensure the presence of both negatively and positively phrased items to avoid the position effect (DeVellis, 2003; Peterson, 2000). This latter step was carried out in conjunction with an independent coder who also judged the saliency and semantic clarity of each item. This process reduced the set to 124 items.

Survey Construction

The resulting items were presented in a multi-page online survey created using Perseus Software Solutions. The introductory screen consisted of information about the survey and researchers' names and affiliations. This was followed by a demographic questionnaire. The next ten pages contained the survey questions, with 10–14 questions per page. The instructions for this section asked respondents to think about their most recent online shopping experience and to answer the following questions. The last page thanked respondents for their participation.

A five-point Likert scale ranging from ‘strongly disagree’ to ‘strongly agree’ was selected to measure the intensity of responses; there was also a category called ‘not applicable.’ We used a Likert scale because binary scales may limit users' responses and result in a lack of variance, and, since the items were statements rather than adjectives, the semantic differential scale was also rejected (Peterson, 2000). The survey questions were randomized to prevent order effects.

Recruitment

The survey was posted for two weeks (May 15th–31st, 2007) and recruitment was ongoing during this time. Respondents were recruited using in-person and online methods, including visiting academic classrooms, and posting recruitment notices on physical bulletin boards and electronic listservs. As incentive, prize draws for $100 cash or of 30 $20 gift certificates for an online bookstore were offered.

Participants

A total of 440 individuals completed the online survey. There were 305 females (69.3%), 131 males (29.8%). Respondents ranged in age: 18–25 (n=98, 22.3%); 26–35 (n=172, 39.2%); 36–50 (n=113, 25.8%); and 51–70 (n=41, 12.3%). Participants used email (99.5%) and Web browsers (98%) on a daily basis. One quarter were undergraduate or graduate students (n=104). A number of respondents stated they were retired (n=10), self-employed (n=1), unemployed (n=7), or stay-at-home parents (n=5). Others were employed at universities as professors (n=39), or administrators (n=8), or in the business and financial (n=40), health (n=14), K-12 education (n=16), information technology (n=37), information management (n=81), or retail (n=9) sectors. The reminder of those sampled (n=54) held a variety of occupations (e.g. engineers, custodian, chef, journalist, etc); 14 people did not state their occupation.

Procedure

The online survey was pre-tested in-person with three individuals. All of the information gleaned during the pre-testing was used to address issues regarding the wording of items, the format and presentation of the survey, and iron out any technical glitches.

Print and electronic recruitment notices contained a URL that directed participants to the online survey. Respondents proceeded through the screens by hitting the ‘next’ button located in the bottom left corner of each screen. A progress bar allowed them to keep track of their proximity to completion. The final page of items contained a ‘submit’ button. Clicking on this took them to a page that thanked them for their participation. Here they were invited them to enter the prize draw. Names and email addresses provided here were kept in a database separate from the data and were used only to contact prize recipients.

Results

  1. Top of page
  2. Abstract
  3. Introduction
  4. Literature Review
  5. Methodology
  6. Results
  7. Analysis
  8. General Discussion
  9. Limitations and Future Work
  10. Conclusion
  11. Acknowledgements
  12. References
  13. Appendix A: Items from Existing Instruments Incorporated into Engagement Survey Instrument

Data Analysis

Results were analysed using SPSS statistical software. First, 37 of the 124 items were reverse coded. Second, the value of 6 on the Likert scale that indicated that the question was ‘not applicable’ was recoded to reflect missing data. The frequencies of valid responses were examined to identify items that received a low response rate. All were retained.

The means of the items were examined to ensure there was sufficient variability among the items (DeVellis, 2003). Given that a five-point Likert scale was used, items with means greater than 4 or less than or equal to 1 were eliminated (n=12). Inter-item correlations of the items associated with each attribute were examined in detail. Questions that correlated negatively with other items (after reverse scoring) were eliminated (n=12).

Reliability of Sub-scales

The remaining items were tested for internal consistency by calculating the coefficient alpha for each sub-scale (i.e. aesthetics, attention, challenge, etc.). Alpha values of .70–.90 were considered optimal (DeVellis, 2003). Working with items in one sub-scale at a time, items with the lowest item-total correlations were eliminated iteratively until optimal alpha values were achieved. This procedure resulted in 49-items. Thus, the goals of this stage of analysis, which were to retain only the most parsimonious items and determine that the sub-scales were internally consistent, were achieved.

Construct Validity of the Engagement Instrument

Exploratory factor analysis was conducted to evaluate the construct validity of the resulting 49 items. The Kaiser-Meyer-Olkin Measure of Sampling Adequacy (KMO) was used to examine the pattern of correlations among the variables. The KMO test result (0.94) indicated that factor analysis was an appropriate technique to use and should result in distinct, reliable factors (Hutcheson & Sofroniou, 1999, pp. 224–225). Bartlett's Test of Sphericity, which indicates whether there are relationships among the variables, was highly significant (κ2=10562.56, df=1128, p<0.001).

Principal components factor analysis with varimax rotation was used to examine the convergent validity of the items. Principal components extraction was selected to maximize the variance extracted and because it is recommended when reducing a large number of variables into a more parsimonious set (Tabachnick & Fidell, 2007, p. 635). Varimax rotation was used to simplify the factors. The cut-off value of 0.45 was chosen for interpreting the eigenvalues (Tabachnick & Fidell, 2007).

Six rounds of factor analysis were performed in which items with eigenvalues less than 0.45 or that loaded significantly on multiple factors were eliminated. This resulted in 33 items loading on six distinct factors. Table 2 shows the number of items that loaded on each factor, the range of item loadings, and the total variance explained for each factor.

Table 2. Exploratory Factor Analysis of Items
Factor# of ItemsItem Loading RangeVariance Explained
Attention90.57–0.859.81 (29.7%)
Usability80.57–0.755.16 (15.6%)
Aesthetics50.70–0.802.43 (7.4%)
Endurability50.56–0.741.20 (3.7%)
Novelty30.51–0.651.14 (3.5%)
Involvement30.50–0.751.02 (3.1%)

Reliability Analysis of Resulting Factors

The internal consistency of the resulting factor structure was examined. Reliability for the factors was computed using Cronbach's alpha and DeVellis' (2003) guidelines. Again, the item-total correlations were examined for all of the items contained on each factor to achieve optimal alpha values. This further refined the scale to 31 items (2 items were removed from the flow factor) and indicated that the six factor structure was reliable. Table 3 shows descriptive statistics and correlations for the variables retained after factor analysis. Correlations among the factors were significant (p<0.01) and ranged from low to moderate.

Table 3. Descriptive statistics and internal consistency values of factors Analysis
 FactorsMSDα12345
1Attention1.89.5610.9 1.00   
2Usability3.14.3310.8440.18   
3Aesthetics3.53.6840.890.190.41  
4Endurability3.84.7080.8430.130.580.41 
5Novelty3.39.7530.7300.470.360.400.43
6Involvement3.51.7010.7230.370.460.470.510.51
α Cronbach.s index of internal consistency for revised scale.
Note: Listwise N for correlations =394; Correlations all significant at the 0.01 level (2-tailed)

Analysis

  1. Top of page
  2. Abstract
  3. Introduction
  4. Literature Review
  5. Methodology
  6. Results
  7. Analysis
  8. General Discussion
  9. Limitations and Future Work
  10. Conclusion
  11. Acknowledgements
  12. References
  13. Appendix A: Items from Existing Instruments Incorporated into Engagement Survey Instrument

A large pool of items (n=450) were generated from previous research and interviews conducted with computer users. This was strategically reduced to 124 items that represented engagement attributes (e.g. aesthetics, affect, etc.) were administered to a general sample of online shoppers. Through statistical analysis, the sub-scales showed high reliability. Exploratory factor analysis resulted in six distinct factors that were statistically reliable: Attention, Usability, Aesthetics, Endurability, Novelty, and Involvement.

Interpretation of the Factors

Factor analysis identifies the relationships among the items by grouping them into factors. However, it does not explain why the items are related. We must apply theory to the interpretation of the factors. It must be reiterated that the items measured user perception. Therefore the label assigned to each factor could be preceded by the term ‘perceived.’

Focused Attention

These items related to users perceptions of time passing and their degree of awareness about what was taking place outside of their interaction with the website. The remaining items pertained to users' ability to become absorbed and lose themselves in the shopping experience. The concept of flow is defined as a condition ‘in which people are so involved in an activity that nothing else seems to matter’ (Csikszentmihalyi, 1990, 4). The items in this factor pertain to four of the nine core components of flow, namely: (a) distorted sense of time; (b) concentration and focus; (c) absorption in the activity; and (d) loss of self-consciousness. Csikszentmihalyi terms the latter two components as ‘action awareness merging’. The components of flow not seen in the items loading on this factor are clear goals, feedback, challenge, perceived control, and intrinsic reward.

Usability

These items pertained to the emotions experienced by respondents when completing their shopping task, i.e. ‘annoyed’, ‘frustrated’, ‘stimulated’, and ‘discouraged’. Items also tapped into the effort required to shop on the website (i.e. ‘taxing’), and their perceptions of the navigation as being ‘confusing’. Items also measured whether users felt they could perform the tasks they wanted to do, and their perceived control over the interaction. Overall these items assessed users' perceived effort in using the website, their ability to accomplish their shopping tasks, the navigation and organization of the website, and the emotions evoked by using the website. Thus ‘usability’ was appropriate.

Aesthetics

This set of items pertained to specific features of the interface, such as the screen layout and graphics/images, and to respondents' overall aesthetic impressions of the website's attractiveness and sensory appeal. ‘Aesthetics’ is an appropriate name for this factor.

Endurability

The items contained within this factor assessed respondents' likelihood to recommend the shopping website to others. Items also evaluated respondents' perceptions of whether the shopping experience met their expectations of being ‘successful’, ‘rewarding’, and ‘worthwhile’, and working out as planned. Overall these items measured respondents' willingness to return to the shopping website directly in terms of whether they would shop on the website again or indirectly with regards to whether they would recommend the website to others and their overall evaluations of the experience. This factor is therefore labelled ‘endurability.’

Novelty

Items for this factor spoke to the curiosity evoked by the shopping task, or participants' interest in the interaction. Stimulating respondents' curiosity indicates that the shopping website or experience contained surprising, unexpected, or new information at some points. We have called this factor ‘novelty’.

Involvement

Items loading on this factor pertained to respondents' perceptions of feeling drawn into and involved in the shopping task, and their overall assessment of the experience as ‘fun’. Involvement is a ‘need-based cognitive…psychological identification with some object’ that is based on an individual's salient needs and perception that the object will satisfy those needs (Kappelman, 1995, p. 66) This label was adopted for this factor because perceptions of involvement and fun in engagement are based on the level of importance, significance, or relevance (Kappelman, p. 66) given to an object by a user.

General Discussion

  1. Top of page
  2. Abstract
  3. Introduction
  4. Literature Review
  5. Methodology
  6. Results
  7. Analysis
  8. General Discussion
  9. Limitations and Future Work
  10. Conclusion
  11. Acknowledgements
  12. References
  13. Appendix A: Items from Existing Instruments Incorporated into Engagement Survey Instrument

The initial pool items (n=124) consisted of 54 items derived from prior research and 70 items generated from interview transcripts. The final 31 items of the engagement instrument consisted of only 8 items from pre-existing scales; the remainder were new items to this study.

This study evaluated the reliability, construct validity, and multidimensionality of the engagement survey instrument. Our analyses established that the ten sub-scales were reliable and that the items comprising each sub-scale represented ‘hung together.’ However, the exploratory factor analysis found only six distinct factors. Of the initial sub-scales, only Aesthetics and Novelty were retained as constructs after factor analysis. Items from the remaining eight sub-scales converged into four factors. The Usability factor incorporated items from the challenge, feedback, control, and affect sub-scales. It is interesting to note that this factor contained only negative items derived from the affect sub-scale. Items from the attention and engagement sub-scales merged to create one factor, Focused Attention. However, there were also items from the engagement sub-scale, as well as items of affect and motivation that made up the Involvement factor. The Endurability factor contained more items from the motivation sub-scale, and also one item from each of the control and re-engagement subscales.

The clustering of the items into these distinct groups demonstrates that users distinguish the functionality of the system (Usability) from the visual appearance (Aesthetics) of the system's interface. The fact that motivational and affective items loaded on more than one factor corresponds with the notion that emotion is embedded in all aspects of experience (Nahl, 2007; McCarthy & Wright, 2004). The results also delineated focused attention and involvement as different constructs. Attention consisted of items pertaining to focused attention and distorted perceptions of time passing, whereas the Involvement factor included fun and feeling drawn in and interested in the task. Engagement has been called a ‘subset’ of flow (Webster & Ahuja, 2004), and this study suggests that some of characteristics of focused attention, time perception, and awareness that make up flow are components of engagement.

Endurability fits with our conceptual framework of engagement, as well as the evaluative components of the CDP and ISP, because it indicates that users reflect on their experiences. The factor analysis validates that experience is comprised of system-driven variables (usability, novelty, and aesthetics) that make the user want to engage and, perhaps, re-engage. The task precipitates the user becoming involved and entering into a state of focused attention. User traits, such as curiosity, aesthetic sensibilities, motivations, and perceived control and challenge are woven throughout the interaction with the system and task (O'Brien & Toms, 2008). This is concurrent with research in other areas of search, such as relevance (Toms, O'Brien, Kopak & Freund, 2005).

Limitations and Future Work

  1. Top of page
  2. Abstract
  3. Introduction
  4. Literature Review
  5. Methodology
  6. Results
  7. Analysis
  8. General Discussion
  9. Limitations and Future Work
  10. Conclusion
  11. Acknowledgements
  12. References
  13. Appendix A: Items from Existing Instruments Incorporated into Engagement Survey Instrument

The online survey contained 124 questions and, tested prior to launch, required approximately 10–15 minutes to complete; encouragingly, only four participants abandoned the survey altogether. While some participants failed to answer every question, the items were randomized to avoid order effects. One limitation, as with all scales, is subjectivity. We intend to examine the relationship between the scale and more objective measures in future. The advantage of establishing the scale first is that the results have elucidated the underlying factors of engagement. Had we started with objective metrics, we may have measured the ‘wrong’ things. A recent article in The Economist cites the issue of so much ‘data [but] so little information’ (2007, p. 80) where web metrics are concerned. We can now work on ways to objectively assess each of the six factors, rather than speculating whether measures such as pages viewed, products purchased, time on task, etc. are useful measures of engagement.

We have identified six distinct factors that comprise engagement. Our next steps are to establish the predictive validity of the scale. In other words, will these same six factors hold true for another independent sample? In addition, we have defined the factors but not their relationships with each other. Do some of the factors predict others in order to lead to engaging outcomes? We will undertake another large scale sample in order to perform confirmatory factor analysis and path analysis and address these questions.

Another area that requires attention is the generalizability of the scale. We have made a case that online shopping activities parallel those of other search contexts, but we need to provide evidence that the scale will perform affectively in other search environments. In addition, we isolated items that we labelled usability, but there are scales, such as the System Usability Scale, that purport to assess usability. How do our usability items compare to these scales for how well they represent this construct? We would like to compare our usability items with other scales and with usability metrics.

Conclusion

  1. Top of page
  2. Abstract
  3. Introduction
  4. Literature Review
  5. Methodology
  6. Results
  7. Analysis
  8. General Discussion
  9. Limitations and Future Work
  10. Conclusion
  11. Acknowledgements
  12. References
  13. Appendix A: Items from Existing Instruments Incorporated into Engagement Survey Instrument

This research systematically constructed and tested the reliability of a multidimensional instrument to measure engagement in online shopping environments. What's more, we have defined engaging experiences as the combination of six constructs: usability, attention, aesthetics, novelty, involvement, and endurability. The contribution of this work is a concrete understanding of the composition of engaging experiences. Armed with this information, we can pursue future studies that establish the concurrent and predictive validity of our measure. On a tangible level, we have developed a survey instrument that will assist designers in assessing the engagement quotient of their products. Our results offer empirical evidence that there is more to system use than functionality. Users' perceptions of the technology and their experiences with it are modern-day necessities (Laurel, 1993).

Acknowledgements

  1. Top of page
  2. Abstract
  3. Introduction
  4. Literature Review
  5. Methodology
  6. Results
  7. Analysis
  8. General Discussion
  9. Limitations and Future Work
  10. Conclusion
  11. Acknowledgements
  12. References
  13. Appendix A: Items from Existing Instruments Incorporated into Engagement Survey Instrument

The authors wish to thank the inter-rater and individuals who participated in post-testing. We acknowledge the support of the Social Sciences and Humanities Research Council of Canada (SSHRC) Doctoral Scholarship; Killam and Eliza Ritchie Scholarships, Dalhousie University; NSERC (NECTAR), Canada Research Chairs program; and the Canadian Foundation for Innovation.

References

  1. Top of page
  2. Abstract
  3. Introduction
  4. Literature Review
  5. Methodology
  6. Results
  7. Analysis
  8. General Discussion
  9. Limitations and Future Work
  10. Conclusion
  11. Acknowledgements
  12. References
  13. Appendix A: Items from Existing Instruments Incorporated into Engagement Survey Instrument
  • Chapman, P. (1997). Models of Engagement: Intrinsically Motivated Interaction with Multimedia Learning Software. MSc, University of Waterloo, Waterloo, ON.
  • Chapman, P., Selvarajah, S., & Webster, J. (1999). Engagement in multimedia training systems. In Proceedings of ICSS, Hawaii.
  • Childers, T. L., Carr, C. L., Peck, J., & Carson, S. (2001). Hedonic and utilitarian motivations for online retail shopping behavior. Journal of Retailing, 77(511535).
  • Choi, D., & Kim, J. (2004). Why people continue to play online games: In search of critical design factors to increase customer loyalty to online contests. Cyber Psychology and Behavior, 7 (1), 1124.
  • Clip Culture: Internet video. (2006). The Economist, 379 (8475), 73.
  • Csikszentmihalyi, M. (1990). Flow: The Psychology of Optimal Experience. New York: Harper & Row.
  • Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1989). User acceptance of computer technology. Management Science, 35 (8), 9821003.
  • DeLone, W. H., & McLean, E. R. (1992). Information systems success: The quest for the dependent variable. Information Systems Research, 3 (1), 6095.
  • DeVellis, R. F. (2003). Scale Development: Theory and Applications (2nd ed.). Sage.
  • Ghani, J. A., Supnick, R., & Rooney, P. (1991). The experience of flow in computer mediated and in face to face groups. In Proceedings of the International Conference on Information Systems, NY.
  • Govern, J. M., & Marsch, L. A. (2001). Development and validation of the situational self-awareness scale. Consciousness and Cognition, 10, 366378.
  • Guay, F., Vallerand, R. J., & Blanchard, C. (2000). On the assessment of situational intrinsic and extrinsic motivation: The Situational Motivation Scale (SIMS). Motivation and Emotion, 24 (3), 175213.
  • Hassenzahl, M., & Tractinsky, N. (2006). User experience – a research agenda. Behaviour and Information Technology, 25 (2), 9197.
  • Haubl, G., & Trifts, V. (2000). Consumer decision making in online shopping environments: The effects of interactive decision aids. Marketing Science, 19 (1), 421.
  • Huang, M. (2003). Designing website attributes to induce experiential encounters. Computers in Human Behavior, 19, 425442.
  • Hull, R., & Reid, J. (2003). Designing engaging experiences with children and artists. In M. A.Blythe, A. F.Monk, K.Overbeeke & P. C.Wright (Eds.), Funology (pp. 179187). The Netherlands: Kluwer.
  • Jacques, R., Preece, J., & Carey, T. (1995). Engagement as a design concept for multimedia. Canadian J. of Educational Communication, 24 (1), 4959.
  • Kelly, E. (2005) Context and Leadership in the Remote Environment. Doctoral Dissertation. Saint Mary's University, Halifax, N.S.
  • Kuhlthau, C. C. (1991). Inside the search process: Information seeking from the user's perspective. JASIS & T, 42 (5), 361371.
  • Laurel, B. (1993). Computers as Theatre. Addison-Wesley.
  • Lavie, T., & Tractinsky, N. (2004). Assessing dimensions of pervieved visual aesthetics of web sites. International J. of Human-Computer Studies, 60, 269298.
  • Litman, J. A., & Spielberger, C. D. (2003). Measuring epistemic curiosity and its diverse and specific components. J. of Personality Assessment, 90 (1), 7586.
  • Liu, Y. (2003). Developing a scale to measure the interactivity of Web sites. J. of Advertising Research, 43 (2), 207216.
  • Madden, M. (2007). Online Video. Washington, DC: Pew Internet & American Life Project. Many ways to skin a cat: Web metrics. (2007). The Economist, 385 (8557), 80.
  • Mathwick, C., Malhotra, N., & Rigdon, E. (2001). Experiential value: Conceptualization, measurement and application in the catalog and internet shopping environment. J. of Retailing, 77(39-56).
  • Mathwick, C., & Rigdon, E. (2004). Play, flow, and the online search experience. J. of Consumer Research, 31 (2), 324332.
  • McCarthy, J., & Wright, P. (2004). Technology as Experience. Cambridge, Mass: MIT Press.
  • Monsuwe, T. P., Dellaert, B. G. C., & de Ruyter, K. (2004). What drives consumers to shop online? International J. of Service Industry Management, 15 (1), 102121.
  • Nahl, D. (2007). The centrality of the affective in information behaviour. in D.Nahl & D.Bilal (Eds.), Information and Emotion. Medford, New Jersey: ASIS & T.
  • Novak, T. P., Hoffman, D. L., & Yung, Y. F. (2000). Measuring the customer experience in online environments: A structural modeling approach. Marketing Science, 19 (1), 2242.
  • O'Brien, H. L., & Toms, E. G. (2008). What is user engagement? A conceptual framework for defining user engagement with technology. JASIS & T, 59 (6).
  • Overbeeke, K., Djajadiningrat, T., Hummels, C., Wensveen, S., & Frens, J. (2003). Let' s make things engaging. in M. A.Blythe, A. F.Monk, K.Overbeeke & P. C.Wright (Eds.), Funology (pp. 717). The Netherlands: Kluwer.
  • Pace, S. (2004). A grounded theory of the flow experiences of Web users. International J. of Human-Computer Studies, 60, 327363.
  • Park, S.-E., Choi, D., & Kim, J. (2004). Critical factors for the aesthetic fidelity of web pages: Empirical studies with professional web designers and users. Interacting with Computers, 16, 351376.
  • Peterson, R. A. (2000). Constructing Effective Questionnaires. Thousand Oaks, CA: Sage.
  • Reio, T. G. J. (1997). Effects of curiosity on socialization-related learning and job performance in adults. Unpublished Doctor of Philosophy, Virginia Polytechnic Institute and State University, Falls Church, VA.
  • Said, N. S. (2004). An engaging multimedia design model. In Proceedings of the 2004 Conference in Interaction Design and Children, College Park, Maryland.
  • Särkelä, H., Takatalo, J., Komulainen, J., Nyman, G., & Häkkinen, J. (2004). Attitudes to new technology and experiential dimensions of two different digital games. In Proccedings of NordiCHI, Tampere, Finland.
  • Schmidt, K. E., Bauerly, M., Liu, Y., & Sridharan, S. (2003). Web page aesthetics and performance: A survey and an experimental study. In Proceedings of ICIE, Las Vegas.
  • Schneiderman, B., & Plaisant, C. (2005). Guidelines, principles and theories. In Designing the User Interface (4th ed., pp. 60106): Pearson/Prentice Hall.
  • Shang, R.-A., Chen, Y.-C., & Shen, L. (2005). Extrinsic versus intrinsic motivations for consumers to shop on-line. Information & Management, 42 (3), 401413.
  • Skelly, T. C., Fries, K., Linnett, B., Nass, C., & Reeves, B. (1994, April 24–28). Seductive interfaces: Satisfying a mass audience. Paper presented at the Conference on Human Factors in Computing Systems, Boston, MA.
  • Stone, D., Jarrett, C., Woodroffe, M., & Minocha, S. (2005). User Interface Design and Evaluation. London: Morgan Kaufman.
  • Tabachnick, B. G., & Fidell, L. S. (2001). Using Multivariate Statistics (Fourth ed.). Needham Heights, MA: Allyn & Bacon.
  • Te'eni, D., Carey, P. & Zhang, P. (2006). Human-Computer Interactio. Wiley.
  • Toms, E.G., O'Brien, H.L., Kopak, R., Freund, L. (2005). Searching for relevance in the relevance of search. in Crestani, F. & Ruthven, I. (Eds.) CoLIS, Volume 3507. Glasgow, UK: Springer-Verlag, p. 5978.
  • Unger, L. S., & Kernan, J. B. (1983). On the meaning of leisure: An investigation of some determinants of the subjective experience. The J. of Consumer Research, 9 (4), 381392.
  • van der Heijden, H., Verhagen, T., & Creemers, M. (2003). Understanding online purchase intentions. European J. of Information Systems, 12, 4148.
  • Webster, J., & Ahuja, J. S. (2006). Enhancing the Design of Web Navigation Systems: The Infuence of User Disorientation on Engagement and Performance. MIS Quarterly, 30 (3), 661678.
  • Webster, J., & Ho, H. (1997). Audience engagement in multimedia presentations. The DATA BASE for Advances in Information Systems, 28 (2), 6377.
  • Webster, J., Trevino, L. K., & Ryan, L. (1993). The dimensionality and correlates of flow in human-computer interactions. Computers in Human Behavior, 9, 411426.
  • Witmer, B. G., & Singer, M. J. (1998). Measuring presence in virtual environments. Presence, 7 (3), 225240.
  • Zhang, P., & von Dran, G. M. (2000). Satisfiers and dissatisfiers: A two-factor model for website design and evaluation. JASIS & T, 51 (14), 12531268.
  • Zhou, L., Dai, L., & Zhang, D. (2007). Online shopping acceptance model: A critical survey of consumer factors in online shopping. J. of Electronic Commerce Research, 8 (1), 4162.

Appendix A: Items from Existing Instruments Incorporated into Engagement Survey Instrument

  1. Top of page
  2. Abstract
  3. Introduction
  4. Literature Review
  5. Methodology
  6. Results
  7. Analysis
  8. General Discussion
  9. Limitations and Future Work
  10. Conclusion
  11. Acknowledgements
  12. References
  13. Appendix A: Items from Existing Instruments Incorporated into Engagement Survey Instrument
Table  .  
AttributeDefinitionSources of Items
AestheticsThe visual appeal of an application or interfaceChoi & Kim, 2004; Lavie & Tractinsky, 2004; Mathwick, Malhotra, & Rigdon, 2001; Pace, 2004; Schmidt, Bauerly, Liu, & Sridharan, 2003; Sarkela et al; Zhang & von Drann
AffectEmotional responseChilders, 2001; Davis, Bagozzi, and Warshaw, 1992; Sarkela et al
AttentionAttention, the concentration of mental activity (Matlin, 1994), pertains to users level of focus or distraction when performing tasks.Govern & Marsch; Mathwick, Malhotra, & Rigdon, 2001; Novak, Hoffman, & Yung, 2000; Pace, 2004; Webster, Trevino, & Ryan, 1993; Witmer & Singer, 1998
ChallengeThe amount of physical or cognitive effort users' believe they are expending on a task (Webster & Ahuja, 2004)Novak, Hoffman, & Yung, 2000; Unger & Kernan, 1983
ControlComputer users need to feel they are in control of their interactions (Schneiderman & Plaisant, 2005), and that they possess the ability to overcome any challenges posed by an activity (Mandel, 1997).Ghani, Supnick, & Rooney, 1991; Liu, 2003; Mathwick & Rigdon, 2004; Sarkela et al; Webster, Trevino, & Ryan, 1993; Witmer & Singer, 1998; Zhang & von Drann
EngagementA quality of user experienceWebster & Ho, 1997
FeedbackSystems convey visual, auditory, and tactile feedback to users about the actions and results that have occurred (Stone, Jarratt, Woodroffe & Minocha, 2005).Choi & Kim, 2004; Novak, Hoffman, & Yung, 2000; Sarkela et al; Witmer & Singer, 1998; Zhang & von Drann
MotivationMotivation concerns people's external and internal reasons for engaging in an activity.Childers, 2001; Guay, Vallerand, & Blanchard; Mathwick, Malhotra, & Rigdon, 2001; Unger & Kernan, 1983
NoveltyThe tendency to be attracted to new, interesting, or unusual elements in one's environment (Huang, 2003Huang, 2003; Menon & Khan; Novak, Hoffman, & Yung, 2000; Unger & Kernan, 1983; Webster, Trevino, & Ryan, 1993; Zhang & von Drann
Re-engagement McKinney, Yoon & Zahedi