Improving the user experience of professional researchers: Applying a user-centered design framework in archival repositories
User-based evaluation of archives has evolved significantly over the past three decades. However, existing frameworks rely almost exclusively on surveys and questionnaires, which address users' self-reported perceptions of quality of user experience in the archives. In this paper we propose a user-centered design (UCD) framework for the systematic improvement of the in-person archival user experience for professional researchers that borrows from service design, user experience design and usability. Using existing archival literature on use and users as our point of departure, we will discuss building professional researcher personas, the application of usability performance metrics to encourage greater efficiency for infrastructural archival activities, and UCD tools for enabling professional researchers to take on active roles in the re/design process as experts of their experience.
Archives contain slivers (Harris, 2002) of the historical record – evidence of events or processes that can be re-constructed and configured to give users of the archive, as well as its indirect beneficiaries such as clients of a law firm or viewers of a documentary film (Pugh, 2005), a window through which they can view the past. Archives also perform the vital task of long term preservation of that evidence. Because archives lie at the intersection between preservation and interpretation of primary source materials, they play a key role in providing the means with which members of a civil society can engage in an informed dialogue about political and cultural identities.
According to Pugh (2005), professional researchers are “direct users linking archival sources to many indirect users.” (p.46) In other words, the consumer information product (film, television program, new stories, legal argument), produced by professional researchers facing time and financial constraints, is often the mechanism by which the public dialogue about political and cultural identities begins. Because their most valuable resource is time, measuring professional researchers' user experience in the archive lends itself to non-qualitative/objective HCI research methods.
Despite their important role in “processing” primary resource materials for broader consumption, up to this point there have been no large scale research activities, qualitative or quantitative, dedicated to a better understanding of professional researchers in archives. However, there has been a considerable push from user-centered research advocates including Elsie Freeman (1984), Paul Conway (1986), Shelley Sweeney (2002), and Kate Theimer (2007) to direct the archival profession in a more responsive direction. Focusing primarily on scholarly researchers, students, and genealogists, these and other authors have performed extensive user studies to determine who these archives users are, their information needs, user information-seeking behavior, and even user satisfaction.
In 2008, during the second phase of a study on user-based evaluation in archives, Wendy Duff, et al., wrote: “Despite the increase in studies of specific user groups, user-based evaluation research of archival services and systems remains limited.” (p.145) The Mellon funded study evolved into Archival Metrics Toolkit, an effort spearheaded by Wendy Duff, Elizabeth Yakel and Helen Tibbo, that provides generic, user-based evaluation tools for archives. (Archival Metrics, n.d.) However, we argue that through exclusive use of questionnaires and focus groups (Yakel & Tibbo, 2010) as evaluation tools, Archival Metrics privileges user-reported, subjective evaluations while excluding more objective measures such as task analysis, and therefore provides a narrow view of user experience in the archive. Although self-reporting methods can be useful in creating archival personas, predicting user tasks, and measuring user satisfaction, when addressing the improvement of archival systems for supporting professional researchers, these methods should be augmented with real-time measurements of users' performance, as well as creative tools that enable users to communicate their needs and expectations in an artifactual, more visible, and/or more pronounced way in order to produce holistic evaluations of the professional researcher's archival experience. (Stickdorn & Schneider, 2011)
To this end we propose a user-centered design framework (Craven, 2008) for improving user experience of professional researchers in the archive. The choice of a design-specific framework is generally appropriate for archives because the genesis of “design thinking” (Buchanan, 1992) comes from problem spaces in which acquiring enough data to isolate correlation and causation between all the dependent and independent variables is impractical. (Rittle & Webber, 1973) Design thinking and its implementations are an attempt to deal with these “wicked problems” (Churchman, 1967) in a systematic way.
Using existing archival literature on users and use as our point of departure, we will explore archival applications of methods and tools borrowed from service design, user experience design, and usability including: creation of the professional researcher persona; determining key tasks for professional researchers and using usability performance metrics such as users' success rate, time on task, error rates, and efficiency in order to evaluate archives' effectiveness in facilitating completion of those tasks (Tullis & Albert, 2008); and ‘providing users with instruments and activities (such as customer journey maps, (Stickdorn & Schneider, 2011) and card sorting) that enable them to take on roles as experts of their experience and participate in design process.’ (Tassi, 2012) These tools are not meant to replace any existing evaluation tools used in archives, but to augment existing tools and methods in order to better meet the specific needs of professional researchers. This framework is meant to be applied towards the systematic improvement of the in-person archival user experience for professional researchers in existing repositories, as well as, the design of future archives.
The content of this section focuses on translating existing research and repository-specific data regarding professional researchers into professional researcher personas. “Personas,”(Cooper, 1999) or “user profiles,”(Kuniavsky, 2003) can be used to represent a shared understanding of a class of archival user experience issues that are repository-specific and important to the success of the archival product.(Levy & Robles 1984) This technique gives archivists a relatively quick and cost-effective way to operationalize UCD using data that is already gathered by many repositories about their users, including: demographics, background, level of experience, research topic, etc. (Duff, et al., 2008) Because repository-specific data lacks generalizability and there is currently no central online repository for sharing anonymized user data, our discussion of a professional researcher persona is based on anecdotal accounts by professional researchers of their experience in archives, and user studies published in the archival literature regarding professional researchers. This is problematic because the amount of available literature on information needs, motivations, and information seeking behavior of users varies widely across identified user groups and, as previously acknowledged, there is relatively little on professional researchers. However, our motivation for spotlighting professional researchers is two-fold:
- 1Combine some of what is known about information needs, information seeking behavior, external constraints on user experience, level of experience/expertise in the archive and digital proficiency within a segment of the archival user population as a means of predicting user tasks - a prerequisite to measuring user performance (Kuniavsky, 2003)
- 2Draw greater attention to the existing gap in archival user research regarding thorough acknowledgement of the diversity of motivations, reward systems and expectations within the professional researcher user group.
Professional researchers include users that are motivated by financial compensation for their retrieval of relevant primary source materials, (including lawyers, journalists, film and television producers), and persons employed by these groups. As a professional researcher for the State of Texas, Mary Speakman, emphasizes the necessity of time-efficient check-in procedures, realistic estimations of wait times provided by the reference archivist in pre-visit phone interviews, and the availability of informational artifacts regarding repository-specific rules and procedures that she can add to her personal files in order to plan future visits. (Speakman, 1984) Time is the most valuable resource among professional researchers. In 2005, Pugh wrote that professional archival users as a group (distinct from scholarly professional researchers), typically “expect considerable reference assistance” and “often work under time constraints.” (Pugh, 2005, p.47) In 1994, Elizabeth Yakel and Laura Bost synthesized their findings from a series of interviews conducted with administrative users of university archives, (a subset of professional researchers), in which participants even admitted to sacrificing better precision for time saved by having archival staff complete their research requests. (Yakel & Bost, 1994) Meeting deadlines and operating within budget are external constraints that exert considerable influence over the specific information needs and overall user experience of professional researchers in the archives.
Other research findings in the archival literature help us to flesh out persona dimensionality. Findings from Conway's (1986) survey of presidential library users revealed that “a sizable portion of non-academic researchers” are considered “archivally inexperienced.”(p.48) In 2003, Yakel and Torres reported findings based on interviews conducted with twenty-eight research users of academic archives during an eight month period in 2001. The authors illustrated that the expectations for, and the actual archival user experience of expert and novice archival users differ in specific and significant ways. (Yakel & Torres, 2001) Beyond motivation, external constraints, level of experience, and level of subject expertise, digital proficiency is yet another dimension that can be added to a professional researcher persona. Weissman (1994), Taylor (1991/2) and Prensky (2001) have argued that digital proficiency has a significant influence on users information needs, their information seeking behaviors, their expectations for how information can be accessed, how quickly information can be accessed, and subsequently the usability of current analog archival information systems.
Described above are several dimensions isolated by prior research that can be used to capture the diversity among professional researchers. Institutional variation is also expected, as personas will be based on repository-specific user data. Personas are useful tools for predicting user-specific tasks, which are a prerequisite step to measuring user performance.
3. PERFORMANCE METRICS
Reference logs, orientation, exit interviews (Pugh, 2005), and other forms of self-reporting including surveys and focus groups have been and are used by repositories to gather data about users in the archive. However, data gathered from these measurement tools has been almost exclusively motivated by requirements to evidence use to resource allocators (Pugh, 2005), instead of improving archival systems and services to enhance the usability of the archive. Performance evaluations of reference archivists and repositories has also been encouraged as a means of evaluating the archives (Duff et al., 2008), however these evaluations are based on reports of user satisfaction which, alone, is not a sufficient evaluation of user experience in the archive.
Performance metrics can augment user-reported data, however, they are not appropriate for many of the activities that take place in the archive. Activities that are essential to the work of professional archivists cannot be evaluated using these metrics. Essential activities are those activities that rely on the archivist's expert knowledge of repository collections and the tools used to access them, in combination with their experience determining the needs of a client based on skillful probing and interpretation of the in-person context. However, performance metrics do make sense for infrastructural tasks that include check-in procedures and call-slip requests. We argue that both professional researchers and archivists can be served by applying performance metrics to infrastructural archival tasks.
Usability performance metrics are well-known tools for measuring some aspects of the effectiveness and efficiency of different products. They help identify issues for users as well as the magnitude of specific issues, which can guide decisions on prioritizing improvements. We will focus on the application of four out of five basic usability performance metrics as listed in Tullis and Albert (2008): task success (how effectively users are able to complete a given set of tasks); time-on-task (how much time is required to complete a task); errors (mistakes made during a task); and efficiency (examining the amount of effort a user expends to complete a task).
In order to measure task success, it is necessary to articulate the task the archival user or potential participant is being asked to perform in a way that decouples it from other related tasks. For example, if the goal of the repository is to measure the user success rate at finding a description of archival photocopying policies, an inappropriate phrasing of the task might be something like, “Find out more about archival policies regarding the use of materials.” This phrasing fails to provide a “stopping rule,” or a point at which users know they have completed the task. A more appropriate phrasing of the task might be, “Find the current photocopying policies for this repository.” This phrasing narrows the range of interpretation by specifying which policies (“current” and “photocopying”) and which repository (“this”). Measuring task success also involves defining what success means for that task. There are two standard forms of measuring task success: “binary” (users either complete the task or they don't) and “levels of success” (when there are apparent and reasonable shades of grey associated with completing a task). If appropriate phrasing is used, the ‘photocopying policies’ task is a binary task. Users find the current photocopying policies for the repository or they don't and additional data is provided by noting strategies users deploy in their attempt to compete the task. Alternatively, repositories can evaluate the visibility of a particular access point for photocopying (or other) information by restricting users to specific information seeking strategies. For example, “In your attempt to complete this task we request that you avoid asking the reference archivist.” (This restriction may seem problematic in light of the archivist's role as a “boundary person” (Hong in Robinson, 2000) – ‘mediating the flow of ideas across professional boundaries’ However, our application of performance metrics is limited to infrastructural activities, thereby excluding meaningful knowledge exchanges between the professional researcher and the archivist.)
One archival task that might be appropriately measured using “levels of success” is the use of finding aids. Levels of success can be measured across multiple dimensions depending on the desired outcome, the choice of participants, and the task participants are being asked to complete. One dimension for measuring levels of success is the extent to which a participant completed the task on their own. Levels of success can also be measured base on participants' experience completing the task or similar tasks in the same or similar contexts. Regardless of how task success is measured, it is important to provide participants with alternative stopping point if they encounter difficulty and frustration during their attempt to complete a task. For example, it might be helpful to say to users: “Stop whenever you come to a point at which you would normally ask for help from the reference archivist.” (Tullis & Albert, 2008)
Although task success is a useful metric for determining areas of archival systems and services that need improvement, time-on-task specifically measures the amount of time required for a user to complete a task. This metric is especially valuable considering professional researchers working under deadlines and within budget. For these users, their archival user experience is influenced heavily by the speed with which they can find and access relevant primary source material. Tullis and Albert (2008) write, “Time-on-task is particularly important for products where tasks are performed repeatedly by the user.” Reference interviews, as well as finding and accessing relevant primary source material are both much-repeated tasks, not only in the archival context generally, but even in the course of a single visit to an archive. However, each time these tasks are performed, the questions change, as do the logistical challenges of finding and locating relevant information since primary sources require significant interpretation in determining their relevance and the number of resources available vary from topic to topic. The challenge then, is to deconstruct those tasks into steps and determine which of those steps can be decoupled and measured independently from variables such as repository holdings and knowledge of the reference archivist on a particular topic. Assuming a repository has materials relevant to user's topic, one of the first steps in finding and accessing those materials is to locate and consult archival finding aids. This quantifiable step provides an appropriate platform for measuring time-on-task. Taking in to account users' levels of archival experience and expertise, evaluating time-on-task for locating finding aids can draw the attention of archivists and archival administrators to potential areas of improvement within a repository-specific information system and its ability to support the information seeking tasks of professional researchers.
While measuring the ability of a user to succeed at completing a task in a reasonable amount of time is valuable, measuring errors is important for pinpointing exactly what actions or interactions might contribute to task failure. Key to measuring errors is to first to determine what constitutes the task, and then, to determine what constitutes an error. The latter requires distinguishing between errors, on the one hand, and actions that are simply inefficient on the other. Determining what constitutes an error also requires knowing what constitutes correct action. For example, an archive wants to measure user errors for filling out a call slip request for materials. First, the archive would have to determine if measurement starts with locating the call slips, or if errors are being measured from a point at which the user has the call slip and finding aid in hand. Next, an archive would need to determine whether or not completing a call slip for each box (when the call slip provides fields for multiple boxes) constitutes an error or simply an inefficiency. Perhaps, the only actions that constitute errors are those having to do with entering information into fields (Tullis & Albert, 2008), such as putting the accession number in the box number field. When it is time to start analyzing underlying reasons for user errors, it is useful to code different types of errors. To use the call slip example, putting the accession number in the box number field might be coded as a ‘form field error.’ Tracking the number of error types per task across the user sample makes it easier to identify problem areas that require attention. If the ‘form field error’ type was found to be the most common error type for the call slip task, that outcome suggests a reexamination of the call slip form: “Do the fields map easily onto the information fields in the finding aid?,” “Is the wording unclear?,” etc.
Efficiency is measured to some extent by time-on-task, however, efficiency can also be measured by counting the number of steps it takes to complete a task. Presumably, the archivist knows the optimal path, or minimal number of steps required to complete a task. Therefore, comparing the average number of steps it takes for your user sample to complete a task against the optimal path defined by the archivist can be very revealing. Archival check-in procedures are a likely candidate for measuring efficiency because 1) they represent the first service ‘touchpoint’, or point of contact, between an archival user and an archivist and therefore have a significant impact on users' overall impressions of their experience and 2) they have been criticized in the past as overly involved and unnecessarily time consuming. (Speakman, 1984), (Brauer, 1987).
Similar to Max Evans (2007) proposal to offload item-level description to volunteers, we want to simultaneously decrease the amount of time archivists spend doing tasks that distract from the focus of their “professional mission,” and increase quality of user experience for professional researchers. The first round of usability testing across these metrics quantifies a repository's current service state, establishing a reference point from which to measure any proposed or future changes to the repository service model. Additionally, data gathered from these metrics can be plotted against independent variables addressed through surveys and interviews use during persona creation. Examples of independent variables that repositories might want to compare against usability performance metrics include the perceived value of orientation or the level of user expertise.
There may be cases where only one or two of the metrics described in this section can be realistically implemented, however, we believe that it is worthwhile to address all of them and allow individual repositories to decide which metrics they feel have the greatest application for their archival information system.
4. TOOLS FOR HARNESSING USER EXPERTISE
Archives have always been a locus for co-creation of knowledge by archivists and users. Nardi and O'Day's (1999) examination of librarians as a keystone species of the library information ecology maps (easily although not directly) on to the archival context and helps illustrate the particular value of the reference interview as a co-creative activity. It is during the reference interview, “the client and the librarian construct together a clearer picture of what the information request is about. The picture couldn't have been produced by the client or the librarian alone.” Nardi and O'Day's (1999) observation is evidence that enabling users to ‘take on more active roles as experts of their own experience’ (Tassi, 2012) does not happen as an activity outside of traditional archival activities and later applied to the improvement of those activities. Instead, the requisite relationship for successful co-creation/participatory design is formed during the course of traditional activities. Existing tools, such as reference logs, take advantage of the user input-in-context of the reference interview and do not require users to learn to use a specific set of UCD tools in order to participate in the improvement of archival services.
However, we would like to suggest several, more formal methods for operationalizing the participatory nature of user-centered design in the archive. There is a multitude of UCD-specific tools for empowering users and extracting constructive feedback about different aspects of their experience. One such tool is a customer journey map. (Stickdorn & Schneider, 2011) Used across subfields of user-centered design such as interaction design and service design, a customer journey map is a tool used to elicit and visualize the narrative of a user's experience and are typically created to reflect a particular service context, (such as ‘completing a first-time visit to a well-funded university archival repository on a campus serving 20,000+ students’). Stickdorn and Schneider's customer journey map is an appropriate tool for mapping a holistic view of a user experience because it divides the customer journey into three distinct stages: pre-service, service, and post-service. (2011) The pre-service stage of the customer journey map probes users for information regarding: ways in which the archive communicated resources and services offered; information about the repository that was accessible through social media; the information communicated to them from friends, colleagues and family about the resources and services offered by repository or general information about the repository itself; past experiences users have had with similar resources and services or similar archives; and finally, the expectations users have for archival resources and services, or towards the archivist and other service providers during their visit. The service stage addresses user touchpoints throughout their archival service journey and asks users to identify touchpoints that stood out as particularly good or particularly bad. The post-service stage elicits user feedback on how the archive follows up with its visitors; information they chose to share about their archival user experience via social media; and information they chose to share with friends, colleagues, and family about their archival experience. Another goal of the post-service stage is to prompt users to evaluate, systematically, their level of satisfaction by comparing the service expectations they identified in the pre-service stage with their actual service experiences. Stickdorn and Schneider's customer journey map can be an activity completed by service providers (archivists) using professional researcher personas based on user survey data and reference interviews. However, it is conceivable that narratives could come directly from users. The map could be broken up into stages and each stage could be administered at the appropriate time. (Stickdorn & Schnieder, 2011)
Card sorting is another tool that archivists can provide to enable users to take on a more active role in the design or redesign of archival information systems. According to Kuniavsky, the goal of card sorting “is to get perspective on how your intended audience understands your proposed information space.”(Kuniavsky, 2003, p.193) Adapted for an archival information space, an open card sort might involve asking users to sort terms or images that the repository is thinking of using on visual aids throughout the space, steps in a typical check-in process, touchpoints that users are likely to encounter during their service experience, or finding aid components. After users sort cards into groups, it is helpful for them to provide their own labels for the groupings. User-provided labels help to reveal professional researchers' mental models of the archival information space; how they group and sort both tasks and content in their heads. If groups of cards can be organized into larger conceptual groupings, users should be encouraged to do so. Analyzing the groupings can be done informally by noting trends. These trends help reveal how a representative user sample intuitively characterizes relationships between nodes within a repository's archival information system, which can inform archival education programs, incremental changes to an existing repository's information space, and the design of future archives.
In this paper we have proposed a user-centered design framework for improving user experience in the archive for professional researchers. Personas were suggested as a means of operationalizing user-centered design using data already collected by archival repositories, (or using data that can be easily collected using pre-existing methods), to predict user tasks. We examined task analysis using usability performance metrics as an objective measurement to augment user-reported data and restricted the application of these methods to archival activities supportive of, but not directly part of the research task. We feel that these infrastructural activities can significantly impact the experience of professional researchers due to this group's larger information-seeking context (time constraints, financial constraints, and information needs). Finally, we suggested augmenting traditional co-creative activities, such as reference interviews, with more visible UCD tools for participatory design, including customer journey maps and card sorting.
This is one example of Design thinking and its application to the archival context. However, future research should look toward in-depth analysis of the broader applications of Design thinking to the administration of archives. Many parallels can be found between Design research and archival research that might provide a new lens though which to examine the process, culture and structure of archives. Our goal is to provide the archival community with an alternative methodological approach that augments existing evaluation tools towards a holistic capture of the archival user experience, and addresses the specific needs of professional researchers.