Quality evaluation of health answers in social Q&A: Socio-emotional support and evaluation criteria
The quality of health information shared in the context of social and emotional support is of critical concern, but little is known about the quality of health information in community contexts and about socio-emotional factors that impact evaluation of the quality of health information. This poster reports on a study of the quality of health answers in social Q&A sites, focusing on the socio-emotional reactions of and evaluation criteria discussed by three groups of evaluators: librarians, nurses, and site users. Forty participants from each group evaluated 10 answers each drawn from the Health categories of Yahoo! Answers. Results from qualitative analysis of responses to open-ended questions identified five common expressed emotions: fear or concern, confidence (or lack thereof), surprise, trust, and empathy. Five evaluation criteria and indicators were also identified: sources, subjectivity, style, completeness, and accuracy. Our results illuminate key differences in social and emotional factors across the three groups and the need for appropriate balance in educational efforts for evaluation of and the provision of health information.
INTRODUCTION AND BACKGROUND
Social media and community channels are a popular source for people with information needs, problems, or questions about health (Fox & Jones, 2009). In many of these contexts and communities, social support plays an important role. Social support is “the resources provided by other persons” including “useful information or things … [that] may have negative as well as positive effects on health and well-being” (Cohen & Syme, 1985, p. 4). While the relative importance and presence of social support and interaction in online communities has been debated (Eastin & LaRose, 2005; Klinenberg, 2012; Marche, 2012; Wellman & Gulia, 1999), most research has found social and emotional support and social ties to be important factors in online communities. Eysenbach's (2004) meta-analysis was inconclusive on online communities' effects on health outcomes, but Burnett and Buerkle (2004), Coulson (2005), Frost and Massagli (2008), and Gooden and Winefield (2007) have all found socio-emotional and community support to be a significant part of the activities of health-related online communities. Despite this, little is known about the quality of health information in these contexts and communities and how socio-emotional factors impact evaluation of the quality of health information.
The purpose of the current study is to investigate the quality of health answers in social questioning and answering (social Q&A). Social Q&A is a free, easy-to-use Web-based service allowing people to ask and answer one another in many different topic areas, including health, benefiting from the varying expertise and experiences of others. Social Q&A has grown with incredible speed over the past few years (Gazan, 2011; Shah, 2011), with health being a popular topic. In previous studies of social Q&A sites, social and emotional support have been found to be the most important criteria desired in answers (Kim, Oh, & Oh, 2007; Kim & Oh, 2009); Kim and Oh believed the social contexts of such environments increased socio-emotional criteria's importance over traditional relevance studies (see Saracevic, 2007). Narrowing the context to health, Kim, Oh, and Oh (2009) found socio-emotional criteria to be a close second to the utility of answers.
Research findings from studies of online support groups are similar in finding users place high value in “hyperpersonal” support and communication—“strong, personal relationships and exchanges” (Turner, Grube, & Meyers, 2001, p. 232) that surpass face-to-face settings in desirability and intimacy (Walther, 1996)—and personalized sharing of information (Burnett & Buerkle, 2004; Frost & Massagli, 2008; Gooden & Winefield, 2007). The main difference between social Q&A and online support groups is in their comprehensiveness. The latter tend to focus on a certain disease, addiction, habit, or condition (Pennbridge, Moya, & Rodrigues, 1999); social Q&A embraces people who have various immediate health problems, providing responses from answerers with varied levels of expertise and experiences. The open and social nature of most Q&A sites encourages users to be active, altruistic, and empathetic members of the community who contribute informational, personal, social, and emotional content (Oh, 2011; Shah, Oh, & Oh, 2008).
As part of a broader study of the quality of health answers in social Q&A, we proposed two exploratory research questions in the context of social and emotional support:
- 1.What socio-emotional reactions do librarians, nurses, and users have to evaluating health answers from social Q&A sites?
- 2.How does socio-emotional support relate to the evaluation criteria and indicators used by librarians, nurses, and users in such evaluation?
Our study examined the quality of answers given to health-related questions on Yahoo! Answers, one of the most popular social Q&A sites (Gazan, 2011; Shah, 2011). Forty evaluators from each of three groups—Yahoo! Answers users, librarians, and nurses—participated and received compensation for doing so.1 Users were invited from a sample of users who asked health-related questions in Yahoo! Answers during May 2011. Librarians and nurses were invited via appropriate library and medical association mailing lists. Each participant, assigned alphanumeric codes based on their role (L=librarian, N=nurse, U=user), evaluated 10 answers from a group of 400 posted in the Health categories of Yahoo! Answers during May 2011 (see Oh, Yi, & Worrall, 2012, for quantitative ratings).
At the end of their evaluation, participants answered open-ended questions asking for their overall impressions of the health answers, explicit suggestions they would make for people seeking answers online, and any other comments they might have. We used an open coding approach to analyze these qualitatively. All three authors reviewed the answers we received and developed emergent codes for evaluation criteria and indicators mentioned by participants and for emotional reactions displayed in their comments. After we merged our proposed code lists together, each author coded a selection of the responses from each of the three groups. Overlap allowed for intercoder reliability checks; while subtle differences existed, we observed no significant discrepancies. The first author then reviewed and analyzed all coding together.
Across the three groups, emotions indicating fear or concern were the most frequent in participants' comments. Librarians showed the greatest levels of fear for users due to the nature of the answers they reviewed, 18 showing a degree of concern. A few librarians believed answerers were “attempt[ing] to answer with useful infonnation,” but cautioned that users should “…consult with a medical professional” (L03). This librarian also showed fear in suggesting users should “CONSULT A MEDICAL PROFESSIONAL!” and asking “why in the world would one take medical advice from a random person online? It's frightening” (L03). Other librarians also showed high levels of fear over the advice being given and the potential for users to take it without careful evaluation of its quality, accuracy, and credibility. One asked if “people really think they are getting reliable health information this way??” and continued “YIKES! I didn't know it was this bad” (L22).
Nurses showed less concern and fear, but six expressed these emotions to a degree; their concerns related to the evaluation of sources, the level of thought put into answers, and the general nature of the questions and answers. One felt “this is scary” and stated “it's one thing to write a review … but offering personal opinions on health matters is dangerous” (N40). Two users expressed concern and fear that following the suggestions in answers without careful consideration could lead information seekers to harm themselves, relatives, or close friends.
Comments about trust and mistrust were the second most popular emotion identified across the three groups. Equal numbers of nurses and librarians—six each—were identified as feeling this emotion. Of the nurses, four suggested users should mistrust social Q&A sites due to the potential for misinterpretation and varying credibility of answers. Two others were more nuanced, suggesting caution but that the information obtained could be trusted in sufficient context, “with a grain of salt” (N12).
The six librarians' concerns over trust, compared with the nurses' concerns, related more to sources and their credibility. One librarian was more positive, suggesting combining answers with other sources could “offer some dimension beyond the authoritative answer of a legitimate Web site or a doctor.” “Of course,” this librarian continued, “I also read books by actual doctors and reliable Web sites like MedlinePlus” (L06). One user raised trust in the context of answerers needing to gain the trust of those asking questions.
We identified four nurses, three users, and one librarian who discussed their level of confidence. The nurses expressed a lack of confidence in the answers and in social Q&A sites, one being “unimpressed with the majority of answers given” (N02) and another imploring users to “avoid question & answer types sites like these!” (N32). While many librarians shared in the fears implicit in the nurses' comments, only one librarian indicated a lack of confidence in the answers—“one or two were fair; the rest were rubbish” (L04)—and the evaluation skills of users, who they hoped “did not take up the advice” (L04). For the three users, confidence levels were mixed but more positive than the nurses and librarians; one termed their use of Yahoo! Answers “a supplement” for other sources that had “helped me greatly over the years” (U25), while another felt that “the experience is usually good” (U17).
Five nurses and three librarians showed surprise at the reality of the social Q&A setting and of the health answers; no users showed surprise. Nurses were “surprised at some of the answers” (N28) and “that people really turn to other folks on the Internet and trust them” (N14); a librarian “realized these are opinion answers” but still found them “a little startling” (L16).
Empathy was the least common emotion identified; three librarians and one nurse (and no users) expressed a degree of empathy for users or answerers. The nurse's concern was how users' finances would impact their ability to see a medical professional. One of the librarians showed similar empathy for the younger population, given their potential lack of financial resources and hesitation to ask for parental guidance. Another librarian expressed empathy for answerers, who “inherently want to help people and … feel useful,” but was concerned over “dangerous” implications for this “in the online medical community” (L11).
Evaluation Criteria and Indicators
Participants also raised many quality evaluation criteria and indicators in their comments. Our analysis identified five that were significant: (a) sources, (b) subjectivity, (c) style, (d) completeness, and (e) accuracy. We discuss these and the differences between the three groups on each below, focusing on comments relating to social and emotional support. In distinguishing between criteria and indicators, we try to follow the distinctions made by Stvilia, Mon, and Yi (2009) while acknowledging our data show significant interrelation between these types.
Evaluating answers based on their source and the sources they cited was the most common category of indicators of quality mentioned by nurses and librarians. Nearly two-thirds of the librarians commented on a perceived lack of credible and authoritative evidence in the answers: “very little evidence [was] involved” (L19), answers were “without complete information or credentials” (L37), and “not even in the best of these did … anyone [refer] the questioner to an authoritative source” (L37). A couple of librarians held a more positive and nuanced view, one stating “authority and accuracy is not always the necessary ingredient in a forum/discussion situation” (L24). Nearly half of the nurses mentioned source-related indicators of quality, focusing on the apparent lack of research in and opinionated nature of answers: “… most were individual's opinions… . I wouldn't have wasted my time reading those [social Q&A] sites as the credibility can vary widely” (N11). Fewer users—less than one-fifth—commented on source-based indicators of quality. Some shared the concerns of the librarians and nurses, while others felt using personal experience was “a source in itself” (U12) and its use was not a problem. Still others took a nuanced approach, believing the context of the source and identifying any underlying bias were important.
Fifteen nurses and eighteen librarians mentioned the subjectivity of answers as a quality indicator. Nurses focused their feedback on what they perceived to be “heavily opinionated” (N01) answers with “a lot of personal bias” (N12). One nurse felt a lack of professional experience—which “counts for so much” (N32)—and face-to-face interaction led to poor quality answers: “having someone lay their eyes on your body makes a big difference in the quality of your diagnosis and treatment” (N32). Most of the librarians echoed the same concerns as nurses about opinion-based, anecdotal answers. Librarians also tied subjectivity to the answerers' use or lack of sources: “no credentials other than [their] experiences were ever cited” (L14). Per one librarian, acknowledgement of the subjectivity and “ambiguity” in answers was rare (L15). A couple of librarians were accepting of personal opinions and biases in the answers, believing they could be useful when placed in context. Users raised subjectivity much less; five discussed this indicator of quality in their comments. Most were accepting of the subjective nature of the answers, stating this is what they would want out of an answer and what they believed other users would desire. Answers that “only gave websites” were not seen by one user as “how these questions should be answered” (U23).
All three groups mentioned the style of the answers. Ten nurses, while having mixed feelings, erred on the negative side; they often felt answers were too informal, lacking detail and evidence. Those who made positive comments focused on the attempted helpfulness of many answerers and praised responses “somewhat like you would get from a good friend” (N39). Eleven librarians also focused on the attempted helpfulness of answerers—“most people are polite and want to help” (L02)—but were more dismissive overall than the nurses' comments; one felt answers were “more like friends responding to one another via email than actually providing useful information” (L09).
Users raised the issue of style with greater frequency than librarians or nurses, with 15 making comments in this area; it was the most frequent criterion mentioned by users. Five made negative comments on what they perceived as the “vague” (U01), unprofessional, “not very scientific” (U02), over-opinionated, or “dull … [and] mundane” (U18) style of the answers. Seven users made positive comments on the style of answers as “clean and understandable … informative” (U14), “considerate” (U22), and “help[ing] any individual who comes looking for it” (U39). The remaining users were neutral, offering advice to others on how to ask questions, to “trust the long lengthy answers” (U11), and to “ignore the irrelevant voicings from people who are simply over-opinionated” (U15).
All three groups mentioned the completeness of answers as a criterion, at similar frequency. Of eleven nurses, all commented on the incomplete and inadequate nature of many answers; they included “only a fraction of what … would be provided in a hospital/clinic setting” (N01) and “most” did not show “any thought put into … them” (N28). While one nurse believed “some [answers] are very informative with lots of detail,” other answers were judged to be “very slack, lazy, and not helpful in the least” (N37). Fourteen librarians shared much the same concerns as the nurses on the criterion of completeness. While “most people were trying to be helpful … few put in a lot of effort in their answers” (L18); “several seemed to ignore the real, unasked questions … [and] most were incomplete” (L28). Thirteen users commented on completeness; their comments were more positive than the nurses and librarians, but still erred towards at least some answers being seen as incomplete due to short length and low effort.
Finally, discussions of accuracy were less common among librarians and users, but were more common for nurses; twelve raised this criterion. Most were concerned the majority of answers being given included incorrect information—“only one gave the most accurate answer which was to go to the doctor” (N14)—despite answerers portraying confidence: “if the answerer seemed confident, the information was equally as incorrect” (N19). Six librarians commented on accuracy, sharing similar concerns over the accuracy and authority of answers, although one argued “authority and accuracy [are] not always the necessary ingredient in a forum/discussion situation” (L24). Four users shared varying opinions of accuracy.
Our results confirm the previous findings of Kim, Oh, and Oh (2007; Kim & Oh, 2009; Kim et al., 2009); social and emotional support are important criteria on social Q&A sites. While source-related indicators were most popular, librarians and nurses often invoked the subjectivity of answers. Users were more accepting of subjectivity and instead focused on the overall style of the questions, illustrating greater consideration of the social, emotional, and community-based support they valued from the site. This may extend the previous findings of Burnett and Buerkle (2004), Frost and Massagli (2008), and Gooden and Winefield (2007) to a new setting, health-related social Q&A, although further research is needed to confirm this.
Socio-emotional factors also impact on (and are mutually impacted by) the indicators and criteria used and discussed by nurses, librarians, and users. Although completeness is often considered a content-related quality criterion in the literature, our data show complete answers indicate answerers' effort and helpfulness, elements of providing positive social support. Accuracy was valued by nurses and most librarians as an indicator of content quality, but many users valued social and emotional support over perfect accuracy. It was not always desirable for users, serving as an indicator of potentially negative social support.
Librarians appear less comfortable with users' focus on social and emotional support, believing well-cited, factual, objective, and complete responses are needed alongside such support. Nurses shared in these concerns, but were also more concerned than other groups with the lack of accuracy of the answers. While many librarians appreciated users are not always seeking factual answers, almost all nurses argued they should seek face-to-face help from a medical professional first. The lower levels of fear shown by and more positive comments from users indicate greater faith in their fellow users' ability to evaluate the answers they receive in socio-emotional and personal context.
Trust and mistrust were also emotions often felt by the nurses and librarians, but not by users. As a librarian stated, trust of multiple sources in mutual context is better than trusting each singly, as feared by many of the nurses. Users may trust answers without thinking or consulting other sources, but they may also feel greater trust in the socio-emotional, community-based setting of a social Q&A site than a one-on-one, fact-based conversation with a medical professional. Some nurses and librarians understood the “good friend” nature of the community (e.g. N39), but others felt it inappropriate (e.g. L09).
We believe our results show social Q&A sites and health-related online communities must balance providing factual, accurate information and offering socio-emotional support. Librarians, nurses, other medical and information professionals, and answerers should provide users, patrons, and patients with quality information and answers in a setting and context they are comfortable with. Users, patrons, and patients must also be educated in appropriate evaluation that takes into account both subjective and objective aspects of the information they find and the answers they receive, echoing and extending the conclusions of Bibel (2008) and Stvilia et al. (2009). While some users understand they should seek out multiple sources and consider source bias and context, others may place too much emphasis on their social and emotional needs, forgetting that accuracy, completeness, and source credibility should also be evaluated.
Further analysis will explore the overall perceptions of social Q&A by the three groups and the advice given by our participants for people who have health questions, problems, or information needs; we will also explore design implications for social Q&A sites. Future research will examine the potential for collaboration among librarians, nurses, and users of social Q&A sites in providing appropriate educational efforts and information services that encourage users to evaluate health information in light of objective and subjective criteria and indicators of quality.
We could not use data from one user in analysis because their evaluation was incomplete and we could not reach them to have them complete the missing data.