Beyond Happy-or-Not: Using Emoji to Capture Visitors ’ Emotional Experience

Museums are emotionally driven sites. People visit museums to feel and their emotions in ﬂ uence how the museum and its artefacts are perceived. Thus, evaluating emotional states are increasingly important for museums. However, evaluating visitors ’ experiences is increasingly challenging, especially with the introduction of new and emerging technology. Moreover, people ’ s behaviour is not strictly objectiveand rational.Whileemotionalstates aresubjectiveandhardtoverbalizeorobserve,emoji are often used to express emotions on mobile and smartphone messaging applications. In this paper we investigate whether emoji can capture emotional states elicited by museum experiences, supporting traditional methods such as interviews. While other non-verbal self-report methods have been used to evaluate emotions, this is the ﬁ rst tool of this kind designed speci ﬁ cally to measure emotions elicited by museum experiences. We designed a set of 9 emoji illustrating a variety of emotional states beyond happy-or-not. Then, we con ﬁ rmed that participants understood our emoji ’ s intended concept using a word association task. Finally, we used our 9 emoji to evaluate an interactive museum experience. We also run interviews and we investigated the correspondence between participants ’ comments and the emoji they chose. Through this study we gained a better understanding of how the emoji can be deployed to capture a range of visitors ’ emotional experiences. Our ﬁ ndings suggest that emoji can capture which emotional states participants felt beyond the happy-or-not dichotomy, but that they should be complemented with traditionalmethodssuchasinterviewstounderstandwhyspeci ﬁ cemotionswerefelt.


INTRODUCTION
In the past couple of decades, emoji have been growing in types and number (G€ uls ßen, 2016), becoming part of everyday language and everyday life (G€ uls ßen, 2016;Oxford Dictionaries, 2015).Emoji are text-based pictographic characters illustrating facial expressions and abstract concepts such as emotions (Rodrigues et al., 2017).According to Rodrigues et al. (2017), emoji are used to help us to visually express our emotions, especially in social networks (Kelly & Watts, 2015;Vidal, Ares, & Jaeger, 2016).While there are multiple types of emoji, ranging for example from animals to objects, emoji illustrating facial expressions are particularly popular in everyday life to express emotions (Barbieri, Ronzano, & Saggion, 2016;G€ uls ßen, 2016).
Face emoji have been increasingly deployed to capture users' satisfaction, both online and offline.For example, emoji have been used to evaluate consumers' emotions regarding the design of new products or brands such as food and drinks (Desmet, Overbeeke, & Tax, 2001;Jaeger et al., 2017).An increasingly common example of this practice is the 'smiley terminal' or 'satisfaction kiosk' often seen in airports.
Emoji have recently been introduced in museums too to capture visitor satisfaction, but they are far less common (Figure 1).These satisfaction kiosks usually display 3 to 5 emoji ranging from happy to unhappy, so are limited to expressing the dichotomy of happy-or-not.While this may be enough to judge whether a user is satisfied with the service in an airport bathroom or in a shop, museum visitor experiences are more complex as they are very personal and influenced by a number of factors, including emotions (Brent Ritchie et al., 2011;Norris & Tisdale, 2013;de Rojas & Camarero, 2008).
As visitors are more and more interested in 'feeling' rather than 'learning' (Munro, 2014), it is becoming critical for museums to understand which emotions their public feel, where, and why.Once museums have a better understanding of visitors' emotional experiences, they can plan for narratives that are more engaging, meaningful, and ultimately satisfying (Galani et al., 2011;Hansen, Kortbek, & Grønbaek, 2012;Simon, 2016).For example, memorials and history museums may wish to elicit empathy with historical events and with victims of atrocities (Savenije & de Bruijn, 2017).Local museums may want to connect with their communities at a deeper emotional level (Munro, 2014).
While understanding visitors' emotional experience is such a key factor in improving visitors' satisfaction, traditional methods such as questionnaires, interviews and observations can have limitations when it comes to capturing emotions (Foster, 2008; "Visitor Evaluation Guidelines" 2015) (see A Visitor-Centred Museum).Emotions are not easy to verbalize or to observe (Desmet, Overbeeke, & Tax, 2001;Hein, 1998;Mehrabian, 1995).Moreover, the aforementioned methods tend to be time consuming both for the museum and the visitors.Image-based methods illustrating, for example, facial expressions are often more reliable in capturing emotional states.However, these methods also have limitations (see Evaluation of Visitors' Experiences).For example, they may present too many emotions and require visitors too much time and effort to use.Emoji may offer an approach that is quicker but still effective but there is very little formal research to support whether or not emoji can effectively capture visitors' emotions.
This paper explores the potential of a range of emoji specifically designed for evaluating visitors' emotional experience beyond simply happy-or-not.We present an initial investigation of an emoji-based method tailored specifically to museums, to evaluate whether museum visitors are satisfied (i.e.happy) or not with their experience.The method must be rapid and intuitive, so it should include a limited number of emoji.Thus, we designed a set of 9 emoji icons illustrating key emotions elicited during a museum visit (see Design Process for Emoji).Here, we use the term emoji to identify icons that are pictorial (i.e.not textual) and are not necessarily digital.Since emoji may be misinterpreted (Caicedo & Van Beuzekom, 2006), we set out to validate both the method and the emoji themselves.First, we evaluated whether the emoji we designed presented the intended meaning (see Emoji Validation: Word Association Task).Then, we further validated emoji as a method to evaluate visitors' emotional responses through an emoji-based evaluation of an augmented reality sandbox developed for the UK's National Trust (see Validation of an Emoji-based Tool to Capture Emotional Reactions).Based on findings from this study, we discuss how emoji can capture visitors' emotional experiences beyond happy-or-not (see Discussion).

BACKGROUND Defining Emotions
The question 'what is an emotion?' is still very complex to answer (Solomon, 1993).Emotions are often confused with other terms such as feelings or other affective states such as attitudes and moods that impact human behaviour less but last longer (Scherer, 2005).Nevertheless, researchers usually agree that emotions are prompt responses to events or stimuli.Indeed, while feelings and moods have long term effects, emotions are more immediate and can change rapidly (Borod, 2000).For example, Borod (2000) defines emotions as changes in conscious subjective feelings in response to an evaluation of external or internal events.Scherer (2000) highlights how emotions are short affective episodes that change over time in response to internal or external events.More recently, Del Chiappa et al. ( 2014) defined emotions as affective variables elicited by an experience or by the use of a specific item.Here, when we talk about emotions, we refer to a range of emotional experiences including boredom, anger and inspired.
Researchers also agree that emotions are multidimensional.While these dimensions can sometimes vary across fields and researchers, there seems to be agreement on two main dimensions: arousal and valence (Baas, De Dreu, & Nijstad, 2008;Russell, 1980).Arousal happens when a person 'feels' something, meaning their senses are stimulated or a physiological/psychological state is awoken.In other words, it identifies the level of reactivity to an event or stimuli (Russell, 1980).Then, we interpret this arousal to determine why it occurs and what it is, often in relation to context.For example, if we are having a negative experience, we may be sad or angry.The differentiation between positive and negative experiences or events is known as valence (Barrett, Lewis, & Haviland-Jones, 2008).According to Ortony et al. (1988), every emotion has a valence, a polarity.Thus, there is no neutral emotion.However, an emotion can still be associated with different valences (e.g.surprise).Moreover, emotions can present with different levels of intensity, which is the degree to which emotions are experienced regardless of valence (van Goozen et al., 1994).Indeed, survey tools to evaluate emotions often use different levels of intensity to gain a deeper understanding of personal emotions.For example, Scherer (2005) arranges emotions with five degrees of intensity, Bradley and Lang (1994) propose a nine-point scale, and Desmet (2005) uses a three-point scale (not-felt, light, intense).

A Visitor-Centred Museum
Traditionally, museums considered their key mission to preserve human knowledge and educate (Murphy, 2007).However, recent advances in digital technologies are pressuring Daniela De Angeli, Ryan M. Kelly, Eamonn O'Neill museums into redesigning their exhibits to share knowledge through more enjoyable, entertaining experiences in order to drive visits and support traditional goals such as education (Tallon & Walker, 2008).This is happening for a variety of reasons.First, digital technologies have driven changes in society influencing how we live, communicate, learn and how we perceive the world (Bryce, 2001;Greenfield, 2014;Siemens, 2005).Museums need to maintain their relevance in a changing society as their mission is not only to preserve knowledge but also to share it (Tallon & Walker, 2008).Second, digital technology offers us so many options for our entertainment, some passive (e.g.movies) other more interactive (e.g.games) (Greenfield, 2014).While traditional object-focused exhibitions tend to limit visitors' interaction to merely passive observation, this is not how most people choose to use their free time (Simon, 2010).Hence, the simple display of facts and objects is no longer enough to attract and engage visitors.As a consequence, museums are embracing a more visitor-centred approach where it is increasingly important to understand visitors and how they interact with the artefacts in order to design narratives that are more engaging and meaningful (Galani et al., 2011;Hansen, Kortbek, & Grønbaek, 2012;Samis, Michaelson, & Baird, 2017;Simon, 2016).
According to Silvia Filippini-Fantoni, specialist in museum audience analysis and former director of interpretation, media and evaluation at the Indianapolis Museum of Art: Museums are institutions that serve the public and therefore knowing what people enjoy and don't is fundamental to helping these institutions create improved experiences for their visitors.It can result in a better appreciation of the art, better understanding of the stories that the museum is telling, more enjoyment, and ultimately a deeper and more frequent engagement with the institution.(Dindar, 2015) In particular, Filippini-Fantoni explains how a visitor-centred approach to exhibition development can increase visitors' satisfaction (Dindar, 2015).Understanding visitors is key when we consider that their museum experience is often based on personal experience rather than objective values (Gallarza & Saura, 2006).
In particular, research has highlighted how emotions strongly influence visitors' satisfaction (de Rojas & Camarero, 2008).For example, Del Chiappa et al. (2014) interviewed 410 visitors at the National Museum of Archaeology "G.A. Sanna" in Sardinia (Italy) and discovered that higher positive emotions reported corresponded to a higher level of satisfaction.
While the evolution of digital technology has strongly influenced museums and their approach to exhibition design, the evaluation of visitors' experience and satisfaction is crucial regardless of the nature of the exhibition.Whether an exhibition is digital or not, interactive or passive, the evaluation of visitors' emotional response to it is key to understanding how they perceive the museum and ultimately to increase engagement.However, evaluating the impact of emerging technologies can be particularly challenging as both their implementation and impact are less known than long established tools (Damala et al., 2013).Although museums usually have well-established methods to evaluate traditional passive visits, they may not yet have a standard protocol to evaluate new and emerging technologies: They have yet to cultivate standard protocol for measuring the success of the technologies they deploy.Exacerbating this challenge is the notion that evaluation should occur both before and after technologies are implemented; staff must have thorough understanding of how the tools correspond with the museum's mission and goals prior to being embraced at scale.Unfortunately, there are not always concrete precedents for the use of new technologies in the cultural heritage sector, and museums that are early adopters often gamble when trying them (Johnson et al., 2015, 28) In the next section we explore which methods and strategies museums are deploying to understand their visitors' experience, whether with traditional or emerging tools and technologies, and whether these methods are effective in providing relevant and useful data.

Evaluation of Visitors' Experiences
Recent evaluation guidelines from cultural institutions such as the Smithsonian ("Visitor Evaluation Guidelines" 2015) and the East of England Museum Hub (Foster, 2008) indicate that the most used methods to evaluate visitors' experiences were and still are observation, questionnaires and interviews.For example, visitors' experience with the exhibition The Hague and the Atlantic Wall: War in the City of Peace at Museon, in the Netherlands, was evaluated using a mixed methodology of observations, questionnaires, and interviews (Damala et al., 2013).
However, subjective and subtle emotions are difficult to articulate and measure with verbal questionnaires (Desmet, Overbeeke, & Tax, 2001;Mehrabian, 1995).Moreover, standard scales such as Likert tend to gain positive, noncandid responses (Benedek & Miner, 2002).While longer questionnaire may provide richer amount of data, they would also be time consuming for visitors; for example, a questionnaire used to evaluate visitors' experience at Museon included four A4 pages of questions (Damala et al., 2013).
Observational methods such as direct observation have also been used in museums, but it is difficult to acquire a deep understanding of visitors' feelings just by observing them.Indeed, "researchers . . .are limited by what they can actually see" (Hein, 1998, 47:101).Often only strong emotional reactions are clearly visible (e.g.very angry), and people express and experience emotions differently accordingly to their cultural or personal background (Scherer, 1988).
Emotions can also be measured through interaction with gesture, gaze and auditory stimuli.For instance, Ramanarayanan et al. (2015) used a variety of equipment and software tools including Microsoft Kinect to evaluate the quality of public presentations in relation to speech, face, emotion and body movement.Lu and Petiot (2016) used a set of auditory stimuli to convey and assess a set of emotions such as funny, serious, relaxed, and depressed.One advantage of these techniques is that they are unobtrusive as they do not require users to verbalize their feelings.However, the technology used to sense non-verbal behaviour often has limitations.For example, Mueller and Bianchi-Berthouze (2015) noted that gesture recognition technology is unpredictable and the set of movements and gestures cannot be predefined.Moreover, such methods cannot usually measure mixed emotions and their range is limited to few basic emotions (Desmet, 2005).Finally, it is typically not feasible for museums to buy, install or persuade visitors to wear devices such as body trackers, brain or temperature sensing devices.
Visitors' behaviour and interactions may also be video recorded.This method can be unobtrusive and can provide rich data about visitors' behaviour and interactions with the exhibition.A video may subsequently be analysed in detail in order to collect and interpret information about visitors and their experience.Nevertheless, this method remains little used, often due to ethical concerns or the difficulties in installing appropriate recording equipment within the museum (Damala et al., 2013).
All of these methods have their merits and limitations, meaning that combinations of different techniques (i.e.mixed methods) are often preferred to evaluate visitor experiences.By using combinations of methods for data collection, museums can cover a wider range of people and data.For example, interviews are often used to support both direct observations and questionnaires, providing more in-depth information.For example, the Smithsonian uses a mix of qualitative and quantitative methods including interviews, surveys, and direct observation to support their visitors' evaluation ("Visitor Evaluation Guidelines" 2015).Still, combinations of traditional methods provide limited data about emotions and, as discussed above, visitors' experience is strongly influenced by their emotions, which are not easy to verbalize (Reijneveld et al., 2003) or to observe (Hein, 1998).

Methods to Evaluate Emotions
As noted in the previous section, traditional methods such as interviews and direct observation may struggle to capture visitors' emotional experiences.However, there are other fields such as consumer evaluation where emotions have been successfully evaluated.For example, the Geneva Emotion Wheel is a well-known method to evaluate emotions (Scherer, 2005).Participants indicate which emotion(s) they are experiencing from a wheel-shaped emotion scale (Scherer, 2005).Emotions are arranged in this circular pattern according to two major dimensions: control/ power and valence.Each emotion is represented by a different colour and five circles of different size indicating five degrees of intensity, so participants can also indicate the intensity of their emotions.Plutchik's Wheel of Emotions is also well known and has been used for example to detect emotions on social media (Tromp & Pechenizkiy, 2015).It includes 8 basic emotions that are considered key to our survival.These emotions come in pairs and are located opposite to each other: Joy and Sadness; Acceptance and Disgust; Fear and Anger; Surprise and Anticipation.This arrangement is due to Plutchik's belief that opposite emotions cannot be felt at the same time (Plutchik, 1980).
However, the time taken to apply these tools including multiple emotions and intensity levels can be challenging (van Goozen et al., 1994) and too time consuming in some contexts such as museums where evaluation methods should not disrupt the visit.Moreover, the outcomes of verbal methods rely on people's ability to express their emotions (Fox, 2008).A way to overcome this limitation of language-based methods is to visualize emotions as images rather than words (Foglia, Prete, & Zanda, 2008).Indeed, there have been adaptations of emotion wheels using images such as cartoons or emoji rather than text (Figure 2).In general, image-based methods tend to be more fruitful to capture emotions than verbal ones.For example, a widely accepted image-based tool is Bradley and Lang's (1994) graphical version of Mehrabian's PAD which used Manikins (graphical characters) combined with a ninepoint scale to make the tool more intuitive to participants.However, it uses graphic characters to represent emotions that are not familiar to the general public, and they may require too much time and effort from visitors to understand.Moreover, this method cannot measure differentiated emotions (Caicedo & Van Beuzekom, 2006).
Emoji have become increasingly familiar to the general public, notably through mobile messaging systems.Consequently, they are increasingly used to express and capture emotions in a variety of fields including for example food consumption (Jaeger et al., 2017;Vidal, Ares, & Jaeger, 2016) and wellbeing (Fane et al., 2018).The 'Emoji-ometer' uses emoji to evaluate children's experiences with technology (Read & MacFarlane, 2006).'Emoji-face' assessment scales have also been deployed in museums to augment traditional questionnaire scales (Loizeau, K€ undig, and Oppikofer, 2015;Mittelman and Epstein, 2009).Similar emoji systems are often seen in kiosks at airports as a method to empower customers and measure their level of satisfaction (Dickinson, 2018).On one hand, these tools are a very rapid way to capture users' basic experience.On the other hand, they usually evaluate users' satisfaction simply in term of happiness and unhappiness.As a result, they capture a very limited range of emotions, while people's emotional experiences are far more complex and go beyond happy or unhappy.
Pictorial representations of facial expression, such as cartoons and photos, have been used effectively to communicate a wider variety of emotions and have been researched as evaluation tools in a variety of fields (Bradley and Lang, 1994;Desmet, 2005).For example, Microsoft tested a questionnaire using pictures of six faces as stimuli to get user input on intangible properties such as "desire" and "fun" (Benedek & Miner, 2002).Emofaces uses a series of female and male faces to represent emotions ranging from pleasant to unpleasant, and intense to calm (Posner, Russell, & Peterson, 2005); Emocards ask people to choose the cartoon face that best identifies their experience (Desmet, 2005).
One of the most successful non-verbal self-report methods is PrEmo (Product Emotion Measuring Instrument), which uses a three-point scale (not-felt, light, intense) with 12 cartoon animations to represent emotions (Desmet, 2005) (Figure 2).Each of these emotions was selected to represent emotions elicited by consumer products: desire, satisfaction, pride, hope, joy, fascination, disgust, dissatisfaction, shame, fear, sadness and boredom.Each cartoon was animated and includes sound effects designed to facilitate their interpretation.However, this could potentially disrupt visitors' experience within the museum.It also means that PrEmo requires the use of a computer or mobile device, with corresponding oversight and maintenance requirements if deployed in museums.Even if the cartoon were instead printed on paper, PrEmo was designed exclusively to measure emotions elicited by consumer products.While it can be used in other fields by selecting only relevant emotions, emotions relevant to museums (e.g.achievement and social engagement) are missing from this set.The purpose of our emoji collection is not to evaluate the product satisfaction of a generic consumer but to understand museum visitors' satisfaction beyond simply happy-or-not.In the next section we describe the reasoning behind the design of each emoji in our set and the emotional states they represent.

DESIGN PROCESS FOR EMOJI
The overall quality of the museum experience is often based on personal experience rather than objective values (Gallarza & Saura, 2006).Personal experience includes social context, personal motivation, education and expectations (D.Bryce et al., 2014;de Rojas & Camarero, 2008).Visitors' perceptions of the museum experience are affected by whether their expectations are met, they feel engaged and involved and they perceive the museum service as adequate (Bride, Disegna, & Scuderi, 2014;Bryce et al., 2014;Lu, Chi, & Liu, 2015).
We draw on Thurley's (2005) account of the heritage experience as a cycle of understanding, valuing, caring and enjoying.If the museum narrative is clearly communicated, then visitors are more likely to enjoy it and find it relevant.If visitors find the narrative is valuable to them, they may also learn and feel a sense of achievement.If visitors feel a sense of achievement and that the content they are interacting with is relevant, then they feel more involved and they are more inclined to care for what the museum has to offer.Involvement implies some kind of social participation (Chen & Chen, 2010), where people feel part of the museum, and the museum part of the community.To close the circle, if visitors feel involved and a sense of achievement, then they will probably enjoy their visit.Thus, in this paper we argue that visitors feel satisfied not only if they have received an excellent service but also if they: • Enjoy their visit; • Achieve and/or acquire new knowledge; • Feel inspired; • Feel socially involved; • Feel entertained/engaged; • Find the narrative clear, communicated clearly Accordingly, we designed a set of 9 emoji to mirror emotional states directly related to the above principles of visitors' satisfaction or indicative of dissatisfaction (Figure 3): • Basic enjoyment is illustrated by the emoji Happy, Sad, and Angry.
• Learning outcome is represented by the emoji Achieved; • Feeling overwhelmed by information is illustrated by Tired; • Inspiration is illustrated by the emoji Inspired; • Lack of engagement is illustrated by the emoji Bored; • Feeling involved is illustrated by the emoji Socially Engaged; • Lack of clarity is represented by the emoji

Confused;
The emoji illustrating happiness, anger and sadness were the easiest to design, possibly because they were the most familiar ones.For example, they are often seen in smiley terminals at airports.The design of other emoji, such as the ones representing social experience or achievement, was more challenging.We researched existing emoji including those used by Apple iOS, Facebook Messenger and Skype in order to facilitate our design but also to make them more recognizable.While these icons were already familiar to many people as they are proliferating on mobile messaging apps, they were also protected by copyright, encouraging us to design our own emoji for use by researchers and practitioners.The designs were produced using a Wacom Bamboo graphics tablet and Adobe Illustrator.

EMOJI VALIDATION: WORD ASSOCIATION TASK
Pictograms such as emoji do not always clearly depict a specific emotion (Caicedo & Van Beuzekom, 2006).Moreover, they can be open to interpretation because social networks and mobile messaging apps such as Facebook and WhatsApp render these icons differently (Miller et al., 2016).In order to address this issue, we ran a validation study to investigate whether participants recognised our emoji's intended concept or if instead they misinterpreted the emotion we intended to depict with a given emoji.We developed an online word association test using Google Forms.The survey included standard demographic questions such as gender and age.Then, the emoji were displayed one after another.Each emoji was followed by a text box where participants were invited to type the first word that occurred to them in response to the stimulus.
Our method was inspired by Prada et al. ( 2016) and Rodrigues et al. (2017) who asked participants to state the first meaning or emotion that came to mind related to an emoji.Moreover, as in Pejtersen (1991), we used word association to identify the meaning of images.Word association is a well known method (Jung, 1910;Nielsen & Ingwersen, 1999) which consists of presenting a stimulus and the participant answering as quickly as possible with the first word that occurs to her.The responses create a cluster of associative representations of the stimulus (Nielsen & Ingwersen, 1999).Word association is used to collect information on people's perceptions, emotional states, mental models and vocabulary (Nielsen & Ingwersen, 1999;Roininen, Arvola, & L€ ahteenm€ aki, 2006), and has been used to capture the meaning of icons for graphical user interfaces (Pejtersen 1991) and food (Roininen, Arvola, & L€ ahteenm€ aki, 2006).
Participants were recruited through Amazon MTurk.The survey took about 5 minutes to complete, for which each participant received $1.30.We had 121 participants, 77 males, 43 females, and 1 who preferred not to say.Participants were mostly 26-35 years old (63 The responses were recorded in a spreadsheet which was then imported into NVivo.We used NVivo to calculate word frequency and to group responses into categories.Synonyms (e.g.angry, anger, and annoyed were included in the same category) and singular/ plural forms (e.g.idea, ideas) were included in the same category.Some responses could not be related to others, so they were not included in any category (e.g.yellow, eye, and carrot).The main results illustrating the most frequent words associated with each emoji are summarized in Table 1.
The icons illustrating anger, happiness, sadness, and confusion were clearly associated with one specific category.For example, emoji A was associated with the category including words such as anger and angry (#anger) 96 out of 121 times.Emoji B (#happiness) was associated with happy 116/121 times.Emoji C (#sadness) with sad 114/121 times.Emoji E (#confusion) was most frequently associated with confusion, 90/121 times.Other emoji were associated with more than one category.This was true in particular for the emoji illustrating the concept of social experience, which was associated with words such as friend (33/ 121) and love (46/121).Emoji D (#boredom) was frequently associated with both bored (66/ 212) but also with tired (31/121), which may link to mental tiredness.While emoji I (#tiredness) was mostly associated with tired (34/121) and sick (23/121), which may relate to a physical tiredness.Emoji G (#inspired) was described using words such as idea (70/ 121) while emoji H (#achievement) with educated (83/121).
Having validated the emoji as conveying their intended meanings, we used them to capture emotional aspects of a visitor experience: an augmented reality (AR) sandbox developed for the National Trust in the UK.We ran this study to investigate whether the emoji can indeed be used in practice to evaluate visitors' emotions.We also wanted to confirm which emotional states each emoji can capture to further validate how the meanings of the emoji are perceived.Can complicated concepts such as personal achievement and social engagement be represented through emoji?; and are they selected by participants when there has been some educational outcome or positive social experience?

VALIDATION OF AN EMOJI-BASED TOOL TO CAPTURE EMOTIONAL REACTIONS
We further validated the perceived meaning of these emoji by designing an emoji-based survey.The survey was used to evaluate how people perceived an interactive sandbox that the authors developed for the UK's National Trust to commemorate the tercentenary of the landscaper Capability Brown in 2017 (Figure 4).The system was devised to illustrates how Brown implemented his landscapes and is based on the AR sandbox developed by Reed et al. (2014), which allows users to create topographic models by shaping sand.The system augments the sandbox by projecting a topographical map onto the sand using a projector connected to a Microsoft Kinect 3D camera.When the sand is moved around, the Kinect senses the changes in the sand's elevation and changes its projection accordingly.For example, if someone digs a hole in the sand, the system projects a blue surface representing water in that location.In our study we used two versions of the sandbox: (1) Reed et al.'s (2014) digital version augmented by Microsoft Kinect and a projector; (2) a more traditional 'analogue' version without depth sensing and projection, where participants could create a landscape with props such as little houses and signs illustrating trees and water.By using two versions of the same sandbox, we could evaluate our emoji-based tool with both digital and analogue experiences.

Participants
We had 24 participants in total: 12 participants interacted with the digital version (4 male and 8 female) and 12 with the analogue one (6 females and 6 males).Participants were aged 22 to 34 years.We were particularly interested in this age range because the National Trust identified this as a gap in their audience.The introduction of new technologies is seen as a way to attract a younger audience that is currently not visiting their properties.As we were looking specifically for young adult participants who were not necessarily visitors to the National Trust, we recruited participants on campus, through university mailing lists and word of mouth.All participants were students and members of the staff at the University of Bath.

Method
An interactive sandbox was installed in a room at the University of Bath so that we could collect data in a controlled environment.Each participant was given a printed copy of a topographical map.First, they were asked to use the sandbox to replicate the landscape in the map.If they were interacting with the digital version, they simply moved the sand around and the projection on to the sand automatically changed to match the landscape they created.If they were using the analogue version, they moved the sand and placed the props to add landscape elements such as trees and houses.They were then asked to make changes of their choosing to the landscape they had created.For example, they could move the sand around to create a hill or dig a lake, or change the position of the props.Each participant interacted with the sandbox for about 15 minutes.
At the end of the session, a researcher asked participants to complete an emoji-based survey and carried out a semi-structured interview.Each participant was assigned an ID that was associated with their survey.The interviews were audio recorded and saved with the ID of the participant.
First, participants completed a emoji-based survey by selecting the emoji that best illustrated their experience.The emoji were presented in a paper survey.The survey was initially designed with the emoji in a circular patter opposite each other similar to the Geneva Emotion Wheel (K.R. Scherer, 2005).However, we did not want to imply an opposite valence.Rather, we acknowledged that different emotions can happen at the same time and do not necessarily exclude each other.Thus, we displayed the 9 emoji (Figures 1  and 5) with 3 levels of intensity in order to keep the survey short and rapid.PrEmo (Desmet, 2005) successfully used a 3 point scale where participants could pick low, medium or high intensity for a specific emotional state.Hence, we decided to use a simple three intensity levels scale for our survey.Similarly to other imagebased tools such as the Geneva Emotion Wheel, intensity levels were displayed as circles of different sizes (Figure 5).
After completing the survey, participants were interviewed for about 15 minutes.Each interview was guided by a list of questions to evaluate visitors' experience and satisfaction with the sandbox.Questions ranged from learning outcomes to usability of the system, including: Did you enjoy the experience?What did you like/not like?Was the system easy to use? Do you have any questions?Did you need any help?Would you like to play it with your friends?

Data Analysis
The interviews were transcribed into a Word document by the researcher who ran the interviews.The participant's ID was also included in the document.Meanwhile, a different researcher (i.e. an analyst) organised the emoji selected into a spreadsheet: the document included participant ID, emoji selected and at which level.The analyst then carried out a qualitative content analysis of the interviews to gain an understanding of the participants' experience with the sandbox.Comments related to emotional states and subjective experiences were identified.Using participants' ID as a reference, these comments were then compared with the selected emoji to investigate the correspondence between what participants said during the interview and the emoji they chose.

Results
The most frequently selected emoji were those representing Happiness, Achievement and Inspired, chosen respectively 21, 19 and 14 times out of 24 (Figure 6).This general positive attitude was confirmed during the interviews where all participants declared that they had a pleasant experience.Pt 14 selected happiness and during the interview described the sandbox as a magical experience: "so funny . . .easy to use.I have enjoyed it".Pt 16 selected both Achievement and Inspired, feelings that were confirmed by the interview, in which the participant explained how a textbook can give you more knowledge "but you cannot really know what a topographical map is [from a book]".
While traditional surveys tend to attract only positive responses, our emoji were also able to capture negative emotions.3 participants had very negative feelings.Pt 11 selected both Confused and Angry, stating: "It is not clear, I am really confused", adding later: "It is boring . . . it is just sand . . . it is like homework".The fact that this participant "felt tired because it is boring" was also confirmed by the emoji survey where maximum level of Boredom was selected but not Tiredness.Pt 20 was also not happy with the overall experience, selecting Boredom and Sadness rather than Happiness and Achievement.During the interview, this participant stated that the sandbox could be "slightly more interesting" and a textbook would be better, more educational.Pt 20 also complained that the sandbox was too small and about the consistency of the sand.Pt 6 and 4 also selected Boredom.Pt 6 was "not very excited . . .little bit tired" and selected both Tiredness and Boredom.Pt 4 said that the sandbox was "just sand" and could be improved.While participants felt confused almost equally with both the digital and analogue versions of the sandbox, the second version attracted the majority of negative emotions.Indeed, the emoji Boredom, Angry, Tired and Sadness were selected only by participants who interacted with the analogue sandbox.
Participants also selected different levels of intensity.In some cases, positive emotions were selected together with other positive ones.For example, usually people who selected the maximum level of Happiness choose exclusively other positive emoji such as Educated and Inspired.However, participants also selected positive and negative emotional states at the same time, a mixed experience that was confirmed during the interview.For example, participants 2, 4 and 22 selected maximum Happiness together with a minimum level of Confusion or Boredom.During the interview, Pt 4 stated that the sandbox "is fine and a good learning tool, but the colour coding (of the map) is confusing" and could be improved to better indicate "heights".However, the emoji representing boredom was selected instead of the confusion emoji.This could be because the interaction itself was clear and easy to understand but that the colour mapping was slightly annoying and made the experience less engaging.Pt 2 liked the experience and selected happiness, probably because it "helped me understanding different altitudes", however, "I did not know the exact altitude of each colour" and could not "understand what the different colours mean".This was reflected in the emoji survey as the participant selected Confusion.Pt 9 argued that the sandbox was "quite interesting" but that a textbook would be more useful to gain a deeper understanding of the subject.At the end, pt 9 was "neither tired nor excited".These mixed feelings were also reflected on the selection of maximum intensity of Happiness together with average intensity of Confusion.Pt 22 also selected Confusion and during the interview confirmed the participant found the tasks not so clear.Pt 21 considered the sandbox a "more direct way to understand (topographical maps)" but expressed a wish for more specific guidelines.Again, the emoji representing Confusion was also selected.Pt 19 found the experience more interesting than a textbook but initially found the colour mapping a "little bit confusing".By the end, that was "not a problem" and did not affect the overall experience.Indeed, the emoji indicating Confusion was not selected.

DISCUSSION
Our study suggests that emoji can be used effectively to capture visitors' subjective experiences beyond a simple dichotomy of happiness and unhappiness.It does so by providing an understanding of why emoji were selected, how they were perceived and how they may be deployed as a survey tool.Moreover, the digital and analogue sandboxes were clearly experienced differently, a point that was successfully recorded by both interviews and emoji.In particular, emoji clearly displayed how one version was more enjoyable that the other.This suggests that emoji can capture emotions elicited by a range of different museum experiences, and also that they can capture both positive and negative emotions, giving candid results.Furthermore, our findings demonstrate how emoji can capture mixed emotions as in some cases participants selected negative emotions (e.g.Bored and Confusion) at the same time and at different levels.
The experiences described during the interviews further validated how the emoji are perceived.For example, the emoji illustrating Confusion was usually selected by participants who were frustrated with an interface or who did not understand some content, e.g.users who expressed usability or clarity issues with the sandbox during the interview.Participants typically selected Boredom if they found the sandbox not engaging or interesting.Tiredness was selected when a participant felt physical, rather than mental, fatigue.During the validation process, this emoji was also associated with sickness, so it could be interesting to test whether it might be also used as part of evaluating whether an experience is likely to make visitors physically sick, such as motion sickness in virtual reality experiences (LaViola, 2000).Our results also clarify the perception of the emoji illustrating Social experience.During the validation study (section 4), this emoji was associated with words such as friends, friendship, love and happiness.The participants in our case study selected this emoji when they thought the sandbox was potentially fun to play with friends.This suggests that this emoji could be used to investigate whether the visitor had a playful and happy experience together with others.
Only 3 participants interacting with the analogue sandbox selected Anger, at levels 1, 2 and 3 respectively.However, participants did not describe such a negative experience during the interview.Participants found the analogue sandbox particularly boring and they did not like the fact they could not wash the sand from their hands at the end.The results from the validation study indicate a very clear association between this emoji and the concept of anger.One possibility is that interviewees were trying to be polite during the interviews but more fully expressed their emotions through the survey.It is also important to remember that the emoji survey was completed immediately after participants interacted with the sandbox.By the time they did the interview, their memory of the experience and their related emotions may have already changed somewhat.
The study raised questions around the optimal number and individual distinctiveness of emoji.The set of 9 emoji enabled greater expressive 'bandwidth' than the simple happyor-not dichotomy, while retaining sufficient distinctiveness between similar or related emotions to facilitate participants' selection of the emoji that most closely matched a given emotion.For example, the study confirmed the relationship between Achievement and Inspired.Participants who completed the tasks with the sandbox felt a strong sense of achievement and selected Achievement, while Inspired was selected when participants also felt they had learned something new.In principle, creating an even larger set of emoji is appealing since the further increased bandwidth could represent an even wider range of emotions and allow for representing more fine-grained distinctions between similar or related emotions.However, there are at least two potential problems with adding more emoji.The first is simply that a key requirement for our development of the emojibased approach was speed and ease of use in the museum visitor setting.This requirement would become harder to meet as the set of emoji grew, almost inevitably leading to less participation and therefore less visitor experience data collection in practice.
The second challenge exists with any set of emoji and becomes even more challenging if we attempt to use additional emoji to represent more fine-grained distinctions between similar or related emotions.Any two emoji must be sufficiently distinct that they are reliably recognised and distinguished from each other.The work reported here validated that this requirement was met for the set of 9 emoji.
However, as with any representational system, there is a trade off between the set of symbols, their semantics, their expressive power.Having only 9 emoji facilitated the rapid use of the survey but came at the expense of some expressiveness.For example, participants often selected Sadness to express dissatisfaction rather than sadness or grief per se.This makes sense since the sandbox was not designed to provoke strong emotional states or to upset visitors.This finding suggests that we may need more emoji than this basic set of 9 as users could not distinguish between sadness and dissatisfaction.However, such an extension of the set of emoji would require further careful design and validation, and is likely to necessitate creating visually very distinct emoji even for similar emotions.For example, we might propose emoji depicting Thumbs up and Thumbs Down to illustrate satisfaction/dissatisfaction in order to maintain a reliable distinction from sadness.Further work is needed to determine the extent to which using a wider range of emoji would adversely affect the method's reliability as well as one of its main strengths, its speed.
Through the case study we gained a better understanding of how the emoji can be deployed to capture visitors' emotional experiences.In particular, our results provided insights on the use of polarity (i.e.opposite valence of emotions) and intensity levels.Emotions are dynamic and can present at different intensities (van Goozen, van de Poll, and Sergeant 1994), which is why emoji surveys should allow visitors to select multiple emotions and different intensity levels.While established smiley terminals usually allow users to select only one icon (e.g.happy, neutral or angry), our participants could select multiple emoji.Indeed, our survey recorded a variety of emotional responses, positive and negative, as well as their intensity level.Not only did participants often choose more than one emoji, they also selected combinations of emotions of different valence, such as happiness and confusion.Apparently, participants enjoyed the sandbox despite not always regarding the system as very usable.This corroborates the claim that usability cannot fully explain users' experience without taking emotions into account (Agarwal and Meyer, 2009).Furthermore, participants selected different intensity levels, ranging from low (1) to high (3).Thus, the tool was able to record emotional layers where different emotions happened at the same time, at different levels of intensity, for example, a high level of Happiness together with low levels of Boredom, Confusion or Sadness.
Lastly, our study confirmed that while verbal methods can be used to describe an experience, they are not optimal to capture emotions (Desmet, Overbeeke, & Tax, 2001;Mehrabian, 1995;Reijneveld et al., 2003).Indeed, during the interview participants often talked about their general experience with the sandbox rather than their emotional state during the interaction.The emoji-based survey can provide a clear indication of which emotions were felt, which is something interviews can struggle to do.However, the interviews helped to understand why emotions were felt.Thus, we suggest than emoji and other traditional methods such as interviews are used together so that they can complement each other.Data from the interviews can be mapped to the emoji selected.These combined methods can provide a rich understanding of which emotional states were felt and why.

CONCLUSION AND FUTURE WORK
In this paper we tested the ability of emoji as a tool to evaluate visitors' emotional experience.While other non-verbal self-report methods have been used to evaluate emotions, this is the first tool of this kind designed specifically to measure emotions elicited by museum experiences.Before this study, there was little formal research to support whether emoji can effectively capture visitors' emotions.Hence, we designed a set of 9 emoji: Happy, Sad, Angry, Confused, Achieved, Inspired, Bored, Tired, and Socially Engaged.We validated their meaning and confirmed that these emoji were indeed able to capture the relevant emotions in a controlled environment.This helped us to test the method, ensuring it was rapid, intuitive, and effective before deploying it in the field.Our emoji are freely available under Creative Commons license from: https://drive.google.com/open?xml:id=1onitMwbFF9echTCBb0QHBBcqil vzwp1w We are currently developing an online survey editor that will allow museums to create their own emoji survey.We intend this tool to be generalizable and applicable to a wide range of museums and visitors.While emoji are a global phenomenon and are often interpreted in the same way across different cultures and languages (Barbieri, Ronzano, & Saggion, 2016), people may still understand images and facial expressions differently according to their cultural background (Jack, Caldara, & Schyns, 2012;Park et al., 2013).Thus, further studies are needed to investigate emoji validity across visitors with different cultural and social backgrounds.

ACKNOWLEDGMENTS
This work is part of EPSRC Centre for Digital Entertainment (grant EP/G037736/1) and the National Trust funded research project to investigate next-generation cultural heritage user experiences.A special thanks go to Xindan Wang who helped collecting data.Daniel J Finnegan and Malcolm Holley who helped to develop and installing the sandbox.Eamonn O'Neill's research is partly funded by CAM-ERA, the RCUK Centre for the Analysis of Motion, Entertainment Research and Applications (EP/ M023281/1).

Figure 1 .
Figure 1.Visitor satisfaction kiosks from a museum in Spain (left and center) and Germany (right).Photographs taken by Daniela De Angeli in 2018.

Figure 3 .
Figure 3.The emoji icons used to capture visitors' emotional experience.

Figure 4 .
Figure 4. Photo (left) and schematic (right) of the augmented reality sandbox.

Figure 5 .
Figure 5.The survey with nine emoji and three levels of intensity each.

Table 1 .
List of emoji (ID) with their intended meaning.The table also includes the words most frequently associated with each emoji (i.e.Associated meaning) and world clouds displaying all the words associated with each emoji