Measuring the impact of pharmacoepidemiologic research using altmetrics: A case study of a CNODES drug‐safety article

Abstract Purpose To provide an overview of altmetrics, including their potential benefits and limitations, how they may be obtained, and their role in assessing pharmacoepidemiologic research impact. Methods Our review was informed by compiling relevant literature identified through searching multiple health research databases (PubMed, Embase, and CIHNAHL) and grey literature sources (websites, blogs, and reports). We demonstrate how pharmacoepidemiologists, in particular, may use altmetrics to understand scholarly impact and knowledge translation by providing a case study of a drug‐safety study conducted by the Canadian Network of Observational Drug Effect Studies. Results A common approach to measuring research impact is the use of citation‐based metrics, such as an article's citation count or a journal's impact factor. “Alternative” metrics, or altmetrics, are increasingly supported as a complementary measure of research uptake in the age of social media. Altmetrics are nontraditional indicators that capture a diverse set of traceable, online research‐related artifacts including peer‐reviewed publications and other research outputs (software, datasets, blogs, videos, posters, policy documents, presentations, social media posts, wiki entries, etc). Conclusion Compared with traditional citation‐based metrics, altmetrics take a more holistic view of research impact, attempting to capture the activity and engagement of both scholarly and nonscholarly communities. Despite the limited theoretical underpinnings, possible commercial influence, potential for gaming and manipulation, and numerous data quality‐related issues, altmetrics are promising as a supplement to more traditional citation‐based metrics because they can ingest and process a larger set of data points related to the flow and reach of scholarly communication from an expanded pool of stakeholders. Unlike citation‐based metrics, altmetrics are not inherently rooted in the research publication process, which includes peer review; it is unclear to what extent they should be used for research evaluation.

Findings from pharmacoepidemiology studies are often relevant to a broad audience including scientists, healthcare professionals, policy makers, industry, and the public. Research funders, such as the National Institutes of Health, the European Research Council, and the Canadian Institutes of Health Research (CIHR), are also keen to understand the impact of the research they fund. 1 Although citationbased author level (eg, h-index 2 ); article level (eg, cumulative number of citations per article); and journal level (eg, journal impact factor) bibliometrics have served as the mainstay of measuring scholarly impact for decades, altmetrics are increasingly becoming recognized as a complementary measure of research impact in the age of the social web. [3][4][5][6][7][8] Alternative metrics, or altmetrics for short, trace the flow of scholarly communication across a diverse range of research outputs, among a broad audience and in essentially real time. [9][10][11] Importantly, altmetrics can capture previously hidden elements of engagement with research outputs from both scientific and nonacademic audiences. 10,12 Today, a researcher may download the PDF of an article, save it to her online reference manager, discuss the article on social media and blogs, and provide comments or formally recommend the article online postpublication in an academic social network (eg, F1000, Mendeley, ResearchGate, Academia.edu). The historical equivalent may have been to read an article in a print journal, store a copy in an office file, engage in informal hallway conversations, and perhaps comment on or endorse the article in a conference presentation. The use of altmetrics continues to grow and is becoming more prominent in some fields (eg, information, medical, and biomedical sciences) [13][14][15][16][17][18] but has been used less frequently by pharmacoepidemiologists to date. 19,20 There is limited information on how individual articles on population-level drug-safety and -effectiveness research diffuse through the web and whether the data derived can be useful in determining patterns of knowledge translation of drug-safety issues. Altmetrics may be a promising approach for better understanding, planning, and implementing efforts to translate knowledge from observational drug-effect studies to policy makers, healthcare professionals, industry, and the public. This article will provide an overview of altmetrics, including where they may be obtained from, their benefits and limitations, and their role in assessing pharmacoepidemiologic research impact. We informed the following overview by compiling relevant literature from the field of altmetrics in health research, with a particular focus on its use in pharmacoepidemiology. We worked with a librarian at Dalhousie University to develop and implement search strategies in health research databases (PubMed, Embase, and CINAHL), without restrictions on publication year and using the following terms: altmetric* OR infodemiology OR (metric* AND social media). We searched the grey literature (websites, blogs, and reports) and hand-searched journals featuring altmetrics (e.g., PLoS One altmetrics collection) and the reference lists of key sources, including those already known to us or identified in our database search. We present a case study of the altmetrics for a study conducted by the Canadian Network for Observational Drug Effect Studies (CNODES) on higher potency statins and the risk of incident diabetes 21 to further illustrate how altmetrics tools and techniques can be applied in drug safety (Text Box 1).

Box 1 A case study for using altmetrics in pharmacoepidemiology
CNODES is a nationally distributed network of researchers and data centers using collaborative, population-based approaches to study drug-safety and -effectiveness. CNODES is funded by the CIHR and is a collaborating center of the Drug Safety and Effectiveness Network. 39 CNODES' knowledge translation efforts follow a rigorous dissemination strategy to target actionable messages from its studies to various stakeholders. Altmetrics are one way of indicating impact, specifically by measuring the extent to which CNODES' research has reached and been taken up by these target audiences. To examine this uptake, we collected altmetrics data for a CNODES study 21 on October 1, 2014, 4 months following its publication, using 4 complementary approaches. Appendix S1 shows the number of times Dormuth et al 21

Key Points
• "Alternative" metrics, or altmetrics, can complement traditional citation-based measures to assess the reach, uptake, and short-term impact of drug-safety articles.
• Altmetrics may allow for a broader view of research uptake, as they process data related to the flow and reach of activity and engagement from both scholarly and nonscholarly communities.
• There are a variety of tools available to acquire altmetrics for assessing research impact; potential users should understand each tool's unique benefits and limitations.
• Pharmacoepidemiologists may use altmetrics to evaluate and better understand the extent of online attention of their scholarly work, as well as to identify potential audiences and collaborations in drug-safety andeffectiveness research.
• Further work is needed to explore data quality issues and determine the accuracy and interpretations of altmetrics within the context of drug-safety andeffectiveness research to capture meaningful impact.

Mainstream media coverage:
We reviewed altmetrics data from the mainstream media, specifically CTV News network coverage, on May 29, 2014 (http://bit.ly/2gzBiqU). This story was shared 312 times overall using the "share" icon and was recommended 711 times on Facebook. To further assess the potential reach of the original article through this CTV story, we examined how many Twitter users who shared it are considered to be "influential" themselves; in other words, those who have a large follower base and whose messages are often reposted. According to the website Topsy.com ¶ that tracks social media mentions, among those Twitter users who shared the link to the CTV story, 7 members (primarily those with a CTV-related account and/or who are health-related writers) are considered "influential". One of the tools we used to assess the context of Twitter accounts, Topsy.com, was discontinued in December 2015, reinforcing the ever-changing landscape of altmetrics tools. qualitative information about the social media content (eg, accuracy or positive and negative emotion to words). While we know that the article was accessed, we do not know if or how the information was used. 40 For example, we could not determine if patients contacted a healthcare professional as a result of media coverage or other online content. We also were not able to determine why individuals linked to our article, including whether it was related to their interest in the BMJ, statins, adverse drug reactions, or other issues. We determined the number of tweets but not the names of those who were tweeting or the number of retweets and new followers. Third, our time was limited to 4 months since the article was released, and we did not assess temporal trends such as seasonality within the period of data capture.
Lastly, we did not examine the credibility, quality, or funding source of the blogs and examined only English language sources.

| Defining altmetrics
The term altmetrics was coined on Twitter in 2010 by Jason Priem,22 who subsequently defined them as "the study and use of scholarly impact measures based on activity in online tools and environments." 10 More recently, the National Information Standards Organization (NISO) has defined altmetrics as a "broad term that encapsulates the collection of multiple digital indicators related to scholarly work … [that] are derived from activity and engagement among diverse stakeholders and scholarly outputs in the research ecosystem, including the public sphere." 23 Altmetrics are a subset of webometrics or cybermetrics in that they focus on measuring the online engagement with research products through social media, reference managers, blogs, etc. 10 Altmetrics are concerned with online-or web-based sources of measuring scholarly activities and use the technological infrastructure of the modern web (ie, Web 2.0, defined as websites that underscore usergenerated content, interaction, and collective knowledge production and exchange [24][25][26] to capture information on the social immediacy and visibility of many types of research outputs in nearly real time. They are not specific to a level of aggregation; they may be applied at the article, author, journal, funder, geographical region, subject area, or institutional level to contribute to the assessment of research impact. 27 Scientometricians and research evaluators have been using nontraditional sources of data to track scholarly impact such as acknowledgements, patents, mentorships, news articles, and use in syllabi. 10 The novelty in altmetrics data sources is not that they create new scholarly practices but that they enable formal and informal communication to occur in a traceable format over the web. Whereas citation-based metrics are centered on the peer-reviewed manuscript, altmetrics include metrics on many types of research outputs or artifacts that are traceable on the web* including peer-reviewed manuscripts; datasets; software code; blog posts; videos; presentations; and shares, likes, and posts made on social media. 9 Indeed, it is the creation of social networks on Web 2.0 that allows for communication among all knowledge users including scientists, healthcare professionals, policy-makers, and the public. 24 Citation-based metrics fail to capture this breadth of an audience, which is important, given it is estimated that only about 15% to 20% of United States-based scientists have published a peer-reviewed article 28 and a small core group of scientists is responsible for publishing a large proportion of articles. 29
News media, policy documents, library holdings, and download statistics may also be considered relevant altmetrics sources. 31 In addition to measuring "the quantity of attention received," some altmetrics data aggregators integrate the "quality of attention" (e.g., a news story counts for more than a Facebook post, and attention from a researcher counts for more than attention from an automated Twitter bot). 32 However, because of the proprietary nature of many altmetrics tools, the exact nature of the scoring algorithms is not always disclosed.
Where would a researcher start if she was interested in obtaining altmetrics for her most recent article or perhaps all her articles? One place to start is the journal publisher website. Many publishers, includ-  also include relevant references. Visitors may "like" a comment that they read on the BMJ responses tab, adding interactivity to the website and allowing readers to express their support for a comment.
Many of the altmetrics tools (Impactstory and KUDOS) have web browser-based applications whereby a researcher enters a persistent author level (ORCID) or article level (DOI) identifier and a set of digital indicators will be provided. Altmetric.com has several products for researchers including a "bookmarklet" that directly integrates into a web-browser bookmark bar and "badges" that may be used for personal webpages or curriculum vitae. They also provide access to an application programming interface (API), which enables researchers to request specific content from the Altmetric.com servers, allowing the data to be analyzed and presented directly by the researcher. For example, researchers may use Altmetric.com's API to obtain data for a research study. Although some of these tools are free, access to the full suite of many of the products and tools used by institutions and publishers requires subscription.
Researchers may also explore the uptake and spread of their work by using clustering (e.g., Carrot2.org) and visual (e.g., Touchgraph.com) search engines to identify additional web resources that mention the full article title and view the interconnections between resources.  Internet Research, highly tweeted articles were 11 times more likely to be highly cited than less tweeted articles. 35 However, the correlation between Twitter activity and citations is highly variable, and the totality of the literature does not support a significant correlation. 36 At the same time, the number of blog posts mentioning a publication has been shown to increase the likelihood of a paper receiving a new citation by nearly 37% in the field of Health Professions and Nursing. 37 Importantly, altmetrics allows tracking of research uptake beyond the peer-reviewed manuscript including op-eds, blogs, editorials, postpublication peer review (e.g., f1000.com); software (e.g., GitHub.com);

| THE UTILITY OF ALTMETRICS FOR PHARMACOEPIDEMIOLOGISTS
knowledge translation products such as drug information tools; and other online content related to research (e.g., videos, posters, and slide  researchers. 43 Moreover, the rapid rise of predatory scientific publishing entities adds further noise to online activity. 44 These predatory journals may be mistaken for legitimate journals by scholars and the public. 45 Patients engaging in social media may be targets for promotion of health products or disease-based advertising. Pharmaceutical regulators may also be interested in analyzing altmetrics for purposes of tracking online activity of their own outputs, such as advisories about new safety signals or drug product monograph updates. When interpreting altmetrics, identifying the source of online activity and classifying whether there are potential conflicts is particularly relevant in pharmacoepidemiology.

| Considerations when interpreting altmetrics
It will be important for altmetrics data sources aggregators to work with relevant academic groups, editors, and others to develop methods to identify trusted and evidence-based sources of knowledge, as well as sources with a real or perceived conflict of interest.
Third, gaming and manipulation are theoretically possible by the creation of false data through fake accounts and automation of downloads, tweets, posts, likes, etc. 6,46,47 Although the notion of false positive hits on social media sites has partially been solved by the advertising industry whereby algorithms can identify patterns suggesting manipulation, there are many potential gaming scenarios that are not easily detectable. 48  Another area where manipulation may occur is when readers vote to "like" online content. This feature is very ambiguous. What does a "like" mean for a comment that consists of several hundred words?
Does the person support the comment in principle or do they "like" an argument presented? Since anyone on the web can "like" something without registering on the website, we believe that this feature is prone to gaming. Altmetrics.com is working towards greater transparency in both this issue, as well as its scoring algorithm. 49 Fourth, there are many data quality issues surrounding altmetrics that may result in systematic error. Accuracy, consistency, and replicability of altmetrics data are cited as main issues. 50 Data quality is also dependent on understanding the type of user engaging in research through social media. Certain altmetric tools differentiate between scholarly and public engagement through stratification of data sources.
For example, scholars may tend to download PDFs, whereas the public may view HTML pages. More research is required to test the validity of these types of approaches. Ambiguity and redundancy may also occur when multiple versions of the same research output exist; altmetric data aggregators will typically not be able to distinguish between a preprint and postprint version of an article. Similarly, author disambiguation may be difficult given the lack of standard unique identifiers for specific researchers. ORCID (orcid.org) is one solution to this problem, although uptake has been slow to date.
Online behavioral patterns differ across disciplines in respect to the level of online engagement that will create disparities in the volume of altmetrics data generated. Social media behavior has also changed over time and with more recently published articles. Behavioral patterns may also differ across languages. Therefore, to allow for cross-field and -time comparisons, altmetrics data must be normalized. [51][52][53] Tested approaches have included a process for normalizing Twitter counts at the journal level, 52 field-normalized indicators based on Mendeley data, 51 and normalization of Mendeley reader counts based on an established citation-count normalization method. 53 How to best distinguish different meanings between content-rich (e.g., blog posts and Wikipedia) and content-poor (e.g., Facebook shares or likes) data is also unclear. Consistency of view is another concern among the altmetrics community. 54 Both raw counts and aggregate scores are presently used, with substantive variation in process and composition of aggregate scoring.
Our case study (Text Box 1) may have had specific features that made it highly accessed. The BMJ is one of the most highly cited journals in medicine (impact factor of 16.4 in 2013), and many blogs may focus on highly cited journals. 5  Our case study provides several lessons for the role of altmetrics in pharmacoepidemiology. We provide an example of using altmetrics to measure the short-term research impact of a drug-safety study. Our case study also illustrates the way in which altmetrics can be used for formative and developmental evaluation and to determine which organizations could be "receptor site" targets (ie, to reach specific stakeholder audiences) for future articles to quickly communicate with other researchers, health professionals, decision makers, and the public. Academics can learn to use altmetrics to complement other knowledge translation strategies, both with the public and with other researchers. 5,61,62 For example, we identified several organizations and individuals who are interested in this specific work of CNODES.
They represent important members of CNODES' broader receptor community including, but not limited to, the mainstream media, health writers and bloggers, information resources for health professionals, and patient-focused organizations such as the National Diabetes Education Initiative.

| CONCLUSIONS
Altmetrics is increasingly being used to measure the scholarly impact of research within and beyond the scientific community. Although there are many potential benefits for using altmetrics, we have pointed out several concerns which require clarification. Indeed, as altmetrics become more popular and accepted, they may no longer be considered 'alternative'. Our case study demonstrates that altmetrics, even in its current state, can complement traditional citation-based measures to assess the short-term impact of a drug-safety article. As Bornmann (2015) concludes, future studies need to also focus on the potential of altmetrics to measure broader impacts of research, beyond academia. 36 The rapid uptake and broad reach of information demonstrate its potential to provide drug benefit/risk information to many stakeholders. Further work is needed to explore data quality issues and determine the accuracy and interpretations of altmetrics within the context of drug safety and effectiveness research. Altmetrics could also be employed to document collaborations within pharmacoepidemiology research teams, such as CNODES and its alumni, as well as to determine future collaborations. Our altmetrics analysis identified which organizations and individuals are interested in this drug safety article. In future, this audience could be specifically targeted to more effectively and efficiently disseminate knowledge from future drug safety studies. Finally, we encourage pharmacoepidemiologists who are interested in utilizing altmetrics to evaluate the impact of their research to work with individuals with expertise in the information sciences and social media studies.