Exploring the relationship between traditional bibliometrics and Altmetric scores in the primary care literature

There is some evidence that Altmetric scores correlate with citations in medical research, but this is not consistent across different specialties. No previous studies have examined the association between Altmetric score and citations amongst primary care research journals. The aim of this study was therefore to describe this association. We identified the ten most frequently cited articles published in the top 15 highest impact factor primary care research journals. Article and journal metrics were extracted and summarized using descriptive statistics. We used Spearman's correlation coefficient (rs) and log–log linear regression modelling to examine the relationship between citations and Altmetric score. 150 articles were included with a median of 36.5 (IQR 20–59; range 5–811) citations. We found a positive association between citations and Altmetric score (rs = 0.519; p < 0.001). A unit increase in log Altmetric score was associated with increased log citations [0.175 (95% CI 0.091–0.259, p < 0.001)] in an adjusted linear regression model. The regression findings indicate that increasing Altmetric score by 10% was associated with a 1.68% increase in citation rate. This has implications for how authors, academic institutions and primary care research journals approach dissemination of articles.


BACKGROUND
In 2021, over 120,000 articles were published in medical research journals (SCImago, n.d.).The impact and quality of articles has traditionally been assessed with markers such as citation count and impact factor of the publishing journal.These markers can determine esteem for researchers and inform future promotion or funding allocation.Citation counts are presently used in the United Kingdom Research Excellence Framework to determine the allocation of public funding to academic institutions (Research Excellence Framework, 2019).Journal impact factor is determined by how frequently articles are cited, and is used to determine the relative importance of the journal within its field with authors gravitating to journals with seemingly 'widespread influence' (Garfield, 1996).Article, author and journal bibliometrics are calculated using citations.The exact citation count may differ based on the database used, with coverage of research output and journal title varying across major research article databases (Blakeman, 2018).These metrics are generally used to assess the scholarly impact of research.
Altmetric is an alternative measure of article impact, aiming to complement rather than replace traditional bibliometrics, which assesses the wider societal impact of an article using online attention (Altmetric, n.d.-d).Sources of attention are used to make up the 'Altmetric donut' (Fig. 1; Altmetric, n.d.-b) which illustrates the sources of attention an article has received.These include policy documents, news, blogs and social media mentions.This gives each article an 'Altmetric score' calculated by an automated algorithm, with each mode of attention given a default minimum weighting to 'reflect the relative reach of each type of source' (Altmetric, 2023).These weightings are shown in Table 1.
Mendeley readers, Dimensions citation counts and CiteULike bookmarks do not count towards the score and are not included in the donut.The quality as well as the quantity of mentions is included in the algorithm, with higher profile sources receiving more weighting (Altmetric, 2020).In order to track sources for a research output, it is required to have a unique identifier, such as a Digital Object Identifier or Pub-MedID.Mentions can then be monitored using online sources tracked by Altmetric (Altmetric, n.d.-c).
There are concerns over the use of citation-based traditional bibliometrics at article, author, journal and institutional level as tools to assess merit or quality in the academic community.These include issues around gaming the system to increase citations, editorial policies to boost impact factor, and skewed distribution of citations (Blakeman, 2018;Martin, 2016).There are therefore calls for transformation in how academic performance is measured, for example the San Francisco Declaration on Research Assessment (DORA, n.d.).Whilst Altmetric scores have some advantages over citations, traditional bibliometrics are still used most frequently by researchers and their institutions (Bosman & Kramer, 2016).However, journals increasingly use components of the Altmetric score to promote and disseminate articles (Erskine & Hendricks, 2021).Determining whether these methods of dissemination increase traditional article bibliometrics is important for editors, authors and institutions.If alternative measures of impact are associated with traditional measures, which are tied to financial rewards and viability, then it would make sense to divert resources into improving Altmetric scores for their articles.A meta-analysis of the correlation between Altmetric score and citations in health sciences research showed a weak positive correlation (r = 0.19), but there was a high level of heterogeneity due to the presence of high impact, highly influential journals which were not comparable to smaller, specialized fields (Kolahi et al., 2021).Journal impact factor and citations are heavily dependent on the nature of their field (Seglen, 1997).It is therefore necessary to examine this relationship in each medical specialty.There is no existing literature on the relationship between Altmetric scores and traditional bibliometrics in articles published in primary care research, a distinct discipline that covers an array of populations, clinical and community settings and health systems.This study therefore aims to quantify the association between Altmetric scores and traditional bibliometrics in primary care research journals.

Key points
• There is some evidence that Altmetric scores correlate with citations in medical research, but this is not consistent across different specialties.
• No previous studies have examined the association between Altmetric score and citation amongst primary care research journals.
• Using correlation coefficients and log-log linear regression modelling, this study found a relationship between Altmetric score and citations.
• This may have implications for how authors, academic institutions, and primary care journal editors approach promotion and dissemination of published articles.

Identification of research articles
The ten most frequently cited articles published in the year 2018 for each journal were identified using Scopus (Elsevier, n.d.) to provide a consistent source of citation counts.This potentially included all article types, not only the research, protocol and review-type articles used to calculate journal impact factor.We used the most frequently cited articles as our sample as this was consistent with previous research exploring the relationship between Altmetric scores and citations in other fields of medicine (Barbic et al., 2016;Floyd et al., 2021;Mullins et al., 2020;Nocera et al., 2019).A specific year was chosen as opposed to a longer time range to ensure all articles had equal time for citation and exposure.2018 was chosen to avoid the inflated citation numbers seen in COVID-19 research (Brandt et al., 2022;Ioannidis et al., 2022).We extracted the following data from identified research articles: number of citations, open access status, article type as recorded by Scopus (e.g., article, editorial, review;Elsevier., 2020), and country of the lead author's institution.Journal data extracted were journal impact factor in 2018, open access status, and whether the journal had a Twitter or Facebook account.Altmetric score for each article was obtained using the Altmetric Bookmarklet tool (Altmetric, n.d.-a).Overall score and individual component mentions were extracted.

Statistical analysis
Descriptive statistics were used to summarize article citation counts and Altmetric score.As data were not normally distributed, the association between citations and Altmetric score was quantified using Spearman's rank correlation coefficient.
In order to fit multivariable linear regression models to describe the relationship between the variables whilst adjusting for other factors, the number of citations and Altmetric score + 1 were log transformed to linearise the relationship (Fig. 2).The Pearson correlation coefficient was reported for the association between the log-transformed variables.Unadjusted linear regression was undertaken using the log of citations as the dependent variable and the log of Altmetric score + 1 as the independent variable.Multivariable adjusted linear regression was then   (Altmetric, 2023).

Journal and article characteristics
In total, we included 150 papers, ten from each of the 15 identified journals.They had a total of 8095 citations (mean 54.0

Association between citations and Altmetric score
We observed an association between citations and Altmetric score with a Spearman correlation coefficient (r s ) of 0.519   Our findings are similar to that of studies in other medical specialties.Subject-specific research in pathology (Floyd et al., 2021), urology (Nocera et al., 2019), emergency medicine (Barbic et al., 2016), general surgery (Mullins et al., 2020), and joint arthroplasty (Ramamurti et al., 2021) reported positive associations between Altmetric score and citations, with correlation coefficients ranging from 0.164 to 0.714.Three of these articles reported Altmetric scores for the top ten most cited articles published in both 2013 and 2016, and found greater cumulative Altmetric scores in the articles published later, suggesting increased use of alternative methods of research dissemination over time.In all studies that reported individual Altmetric components, Twitter was the medium with the most mentions (although weighting means it may not be the component contributing most to each individual article's score).Ownership of Twitter changed in 2022, and some in the medical Twitter community have expressed concerns about the direction of the platform (Stokel-Walker, 2022).For individual authors, social media is not necessarily a benign tool even if used exclusively for professional purposes to disseminate published articles.Comparison being the thief of joy, potentially unlimited exposure to the success and achievements of one's peers can be inspiring but also challenging.However, limited or no engagement with social media, and Twitter in particular, seems likely to affect the traditional bibliometric impact of individual authors' publications.Interactions on social media can also have implications for journal editors and need to be handled with care (Burch et al., 2023).This may be easier to navigate for academic institutions and journal social media accounts not run by named individuals.There are important ethical considerations for promotion and dissemination of research on social media.Platforms can provide an opportunity for direct public engagement and knowledge exchange, as well as democratizing access to research findings.Conversely, the volume of information can be overwhelming and there is a risk of the spread of disinformation with echo chambers closing off debate.Solutions to these issues have been proposed including careful curation of 'following' lists, combating disinformation, and linking posts to original evidence or data (Choo et al., 2015).For clinical academics in particular, adherence to guidelines on professionalism and use of social media for example (General Medical Council, 2020) is vital whether communicating with colleagues or the public.
We did not explore whether mentions by authors, institutions or journals were more likely to influence citations, only overall mentions and scores.Notably in our study, whether the publishing journal had a Twitter account was not associated with citations (though only two journals did not have a Twitter account), but we did not explore whether the account was used to promote each individual article.Future research might further explore whether there is a benefit in research promotion being performed by a named author or their institution/publishing journal.Altmetric themselves, when calculating scores, apply modifiers to tweets to adjust each mention's weighting for 'reach', 'promiscuity', and 'bias': posts 'suggesting promotional intent' are worth less than tweets from researchers unconnected to the article; posts from accounts with a large reach are worth more.Original tweets are also worth more than re-tweets or re-posts (Altmetric, 2022).This in theory reduces the ability to 'game' the system, though smaller departments and newer journals-with less resource for promotion on social media or to traditional media and a smaller reach-could potentially be disproportionately disadvantaged in Altmetric scores, subsequent citation counts and impact factor.We did not examine in detail the sources contributing to each Altmetric component for individual articles, so are unable to comment on the possible impact of frequency and intensity of author, institution or journal promotion on social media or through traditional media outlets.It is also worth considering that Altmetric scores do not distinguish between positive and negative content, so do not necessarily indicate desirable attention.This is, however, also true of citations: papers may be cited to support claims, methods or findings but they may also be cited to be criticized.To obtain an in-depth understanding of Altmetric mentions or citations for an article, one would need to review the sources and content of these metrics.This information is easily available for Altmetric scores.
Given our results suggest there is benefit in allocating resources to increasing primary care research article Altmetric scores, it is worth considering how this might affect academic practice beyond social media.All identified individual Altmetric components except Google+ had a positive association with citations.This offers several ways to increase Altmetric scores.News outlet mentions, with the highest Altmetric component score of eight, may provide an opportunity to significantly increase an article's score and social impact.Blogs, with a weighted score of three, are already being used by academic primary care institutions to promote and disseminate their work (e.g., https:// capcbristol.blogs.bristol.ac.uk/).For those institutions not currently producing blog posts, this may be a worthwhile investment to increase article exposure and potentially citations.Prioritizing areas to try to improve scores may be useful: articles in this review only had mentions from six of a possible 17 sources of attention, so these may be a reasonable starting point.There are also useful articles available on how to use Altmetric scores in CVs and grant applications (The Source, 2015).For primary care research journals, consideration of resource allocation to increase Altmetric scores of papers published appears to have dual benefits of potentially increasing citations and therefore improving the journal impact factor.Our study also adds to the existing literature that review-type articles are more frequently cited than original research in health sciences (Miranda & Garcia-Carpintero, 2018), which may influence choice of research question for authors and institutions, and publication decisions for journals.There is also evidence that research articles promoted by publishers in embargo emails to journalists obtain both higher Altmetric score and citation counts, so this could be considered by editors and publishers as another dissemination method (Lemke et al., 2022).There are a number of other factors that may be associated with citations beyond Altmetric score, including article (quality, novelty of subject, characteristics of field, methodology), journal (journal impact factor and prestige, journal language, scope and coverage), and author factors (number of authors, author's reputation and academic rank, collaboration, country, gender, age, ethnicity; Tahamtan et al., 2016).
There is a broader point to reflect on around the use of alternative measures of impact in health sciences research.Whilst we have considered whether Altmetric scores are associated with citations, the components of Altmetric scores are designed to be complementary to, and not to replace or impact, traditional bibliometrics.In theory, citations reflect academic impact and Altmetric scores reflect societal impact.However, Altmetric scores have also been shown to be associated with article quality-though less strongly than citation counts (Thelwall et al., 2023)-and, given social media mentions may well be from other researchers or scholarly bodies, the boundaries between academic and societal impact are not clear-cut.Institutions and researchers, through schemes such as the 2013 San Francisco Declaration on Research Assessment, are being encouraged to consider value beyond traditional article and journal bibliometrics when assessing research and researchers.This includes making assessments based on scientific content rather than exclusively using citation-based bibliometrics, and using a broad range of impact measures comprising a range of alternative metrics such as influence on policy and practice (DORA, n.d.).

CONCLUSION
This study quantified the association between Altmetric scores and traditional bibliometrics in primary care scientific journals.
We observed that the findings for primary care research followed a similar pattern to wider scientific research and journal publication; higher Altmetric scores were associated with more citations.
For journals, institutions and authors considering ways to increase the impact and citations of primary care research, resource allocation and focused attention to the components within the Altmetric score could be considered.This paper shows that efforts and planning around research dissemination through social media and mainstream media could not only increase the wider public impact but can have a positive impact on citation counts.Despite the traditional structure of medical research, embracing social media and engaging with alternative methods of dissemination of primary care research is likely to have beneficial impacts on both the traditional measure of citation count and the more novel Altmetric score.

AUTHOR CONTRIBUTIONS
AB conceived the project, all authors developed the methodology.AB and DB collected and curated the data, AB and OU analysed the data.AB wrote the original draft, all authors offered critical review and revision of the manuscript prior to submission and following peer review.
of Dimensions and Web of Science citations 0 undertaken adjusting for journal impact factor, journal open access status, whether journal had a Twitter profile, and article type.

TABLE 1
Altmetric attention score default weightings

Table 2 .
Lead authors were most commonly based in institutions in the United States (40.5%), followed by the United Kingdom (11.5%),Australia (10.1%) and Canada (8.8%).Table3shows the full summary of Altmetric component mentions.Included articles were mentioned by six of a possible 17 components.

TABLE 2
Summary of journal data.

TABLE 3
Summary of Altmetric component mentions.p<0.001).Associations between individual components of Altmetric score and citations are shown in Table4.There was also an association between citations and journal impact factor (r s = 0.674, p < 0.001).This may have implications for how authors, academic institutions and primary care journal editors approach promotion and dissemination of published articles.To our knowledge, this is the first study to examine the association between Altmetric score and citation count within primary care research.A strength of this study is the statistical analysis: previous studies in this area in other speciality literature have Note: Included articles were mentioned by six of a possible 17 components.(DISCUSSIONThisstudy examined the association between Altmetric scores and citations amongst primary care research journals.The estimated regression coefficient (0.175) from the adjusted log-log linear regression model can be interpreted as indicating that a 10% increase in Altmetric score was associated with a 1.68% (95% CI: 0.87%-2.50%)increase in citations.are missing; although our sample did include a wide range of citation numbers from five to 811.Alternative methods of sampling include selecting papers randomly or selecting papers with the highest Altmetric scores as opposed to citations; both of these methods also have limitations.Sampling by highest citation count is the method most frequently used in other studies in this area and we therefore felt this to be the most appropriate methodology, but this does mean we do not have any studies with fewer than five citations included.This may limit the applicability of our findings to these studies.Whilst we chose 2018 as the sample year to avoid COVID-19 literature and allow time for papers to be cited and shared, this may miss changes that have occurred in the way publications are disseminated since then.It does, however, avoid including recently published papers that may have generated a high Almetric score, with fast and reactive components of online attention, whilst not yet having been cited to a reasonable level.We could not take into account the effect of news or media cycles on a particular topic at a given time.As this is a cross-sectional study, we are unable to comment on how the

TABLE 5
Adjusted linear regression of log citations.

TABLE 4
Correlation between Altmetric components and total citations.