Using interactivity to help users understand the impact of spam filter settings



Interactivity as a way to help users understand and/or explore complex information issues has been utilized successfully in information science areas such as information retrieval. In this paper we report experiences with utilizing interactivity in a different albeit related context: informing users about spam filtering processes. Spam filters that prevent spam from showing up in inboxes by filtering incoming messages according to their suspected level of “spamminess” are readily available. However, the complexity of modern spam filters aggregating evidence from various sources means few non-expert users of spam filters actually understand what their spam filters are doing. It also remains unclear how different spam filter settings would impact on what they see in their email inboxes. In this paper, we describe work into using interactive exploration as a way to increase users' understanding of how spam filters work. Lab-based evaluation of a prototype suggests a considerable increase in spam filtering understanding occurred even among subjects that stated they had prior spam filtering expertise.


Spam is widely regarded as having negative social and economic impacts, and these impacts are escalating. According to Ward (2003) some commentators believe that the alarming growth in spam may jeopardize the future use of the email system.

Email users are concerned about what is happening to their email. Based on interviews conducted in the U.S., Fallows (2003) reported that 30% of email users surveyed were concerned their email filters might filter genuine incoming email and 23% of users were concerned email they send to others may be filtered.

Modern spam filters can be considered fairly reliable (see eg Cormack and Lynam 2007) but there is also anecdotal evidence that overly ambitious spam filters may discard genuine emails falsely classified as spam (so-called false positives) every now and then. The problem can be traced back to a lack of objective criteria that spam filters could employ for determining “(un)solicitedness” of emails. Neither “unsolicited” nor “unwanted” are objective, measurable aspects of emails (Lueg 2005).

Explaining to non-expert users how spam filters actually determine “spamminess” is extremely difficult because of the complexity of some of the algorithms involved (e.g. term distribution). Furthermore, multiple spam filters/modules may be used simultaneously to scan for spam characteristics; each of these can contain different ways of assessing spamminess (see for example the vast number of rules offered by SpamAssassin which is a very popular and freely available spam filter). All this complexity can make it difficult for end users to understand how certain spam assessment criteria can actually affect what messages they see in their inboxes.

In this work, we looked at interactive exploration as a way of educating non-expert users about the spam filtering process. Specifically, the research was designed to assess the use of interactive information exploration for illustrating the effect of certain spam assessment criteria and/or thresholds on a dataset of emails.

Why Exploring Interactivity in a Spam Filtering Context?

The idea to appropriate interactive exploration (‘exploring and browsing’ in Preece et al's 2007 terminology) was inspired by Ahlberg and Shneiderman's (1994) work on starfield displays developed to assist users in exploring large data sets.

In information science, interactivity has been used successfully to work around the problem that users find it difficult to describe their information needs. Interactive information retrieval (eg Koeneman and Belkin 1996), for example, utilizes interactivity in conjunction with relevance feedback. In the human-computer interaction field, Lueg (1998) explored interactivity and implicit relevance assessments as a way of coping with large volume conversational data as experienced in high volume Usenet newsgroups.

Experimental Setup and Results

The aim of this project was to determine if interactive information visualization helps users understand the effects that certain spam filter modules or thresholds may have on their email. We used an existing spam classifier (SpamAssassin) with no additional modifications; auto learning was disabled. As data set we used a mix of real emails consisting of a random sample of emails obtained from the publicly available Enron corpus (available from plus an additional selection from a local junk mail folder. The information was presented using a custom-designed application shown below.

original image

We recruited subjects among currently enrolled university students, mainly by email. The average age of subjects was early to mid-20s with 66% of users being male. 73% of subjects were from a computing related degree; the remaining 27% from Faculty of Arts. In order to gauge pre-existing experiences, concerns and general spam awareness, but also to obtain a good understanding of the bias possibly introduced by recruiting among computing students, we presented subjects with twenty questions prior to using the custom-designed application. Nine of the questions used a Likert Scale with a range from one (low) to five (high), and the remaining questions required an individual choice from a selection of preset options. After using the test application users were asked an additional twelve Likert Scale questions.

Results show that users initially believed they had an average understanding of how spam filters worked and a high level of concern in relation to spam problems; importance of email was rated even higher. 20% of users tested stated they were aware of the criteria spam filters use to classify email; the remaining 80% were not aware or only had a partial understanding which was an interesting find as 73% of the test base was from a computing based degree where one would assume a higher level of understanding.

The application received good feedback in relation to its interface and ease of use which was a pleasant surprise as usability was not a priority when developing the application. The experimental session as a whole also received good feedback regarding how beneficial it was. Even subjects who had stated in the pre-session questionnaire that they had a very good understanding of how spam filters work reported that their understanding increased by a moderate amount.

Improvements of understanding were found in all areas including understanding why legitimate email may be blocked, why spam may slip past the filters as well as an overall understanding of the spam filtering process.

Discussion and future research

The data we collected suggests that overall understanding increased by 18%. We believe this is a very promising outcome considering that the application was a prototype and the duration of individual experimental sessions was limited to 15 minutes. A certain bias has to be taken into account as subjects were recruited among students in their early-mid20ies which means they are used to internet technologies and also used to ‘learning’.

Future work in relation to the custom-designed application includes improving the user interface with more aesthetically pleasing elements; converting the spam classifications routines to a multi-threaded architecture to increase the response time of the application. We are also exploring ways to allow users to dig deeper into the visualizations and pull up additional information about why certain SpamAssassin scores were calculated they way they were. Revealing what SpamAssassin rules fired would allow to explain in great detail why individual emails were classified as spam.