The relative citation ratio (RCR) as a novel bibliometric among 2511 academic orthopedic surgeons

Objectively measuring research output is important for grant awards, promotion, and tenure, or self‐evaluation of productivity. However, certain shortcomings limit common bibliometric indicators. The time‐ and field‐independent relative citation ratio (RCR) was proposed to overcome these limitations. The objective of this study was to determine whether the RCR correlates with academic rank, gender, and PhD degree status among US academic orthopedic surgeons. Full‐time faculty surgeons at Accreditation Council for Graduate Medical Education‐accredited orthopedic surgery residency programs were included in this study. Mean (mRCR) and weighted (wRCR) RCR scores were collected from the National Institutes of Health iCite database to quantify scholarly “impact” and “production,” respectively, and were compared by academic rank, gender, and PhD status. All information was collected from publicly available faculty listings on departmental websites. A total of 2511 orthopedic surgeons from 132 residency programs were assessed. Overall, the median (interquartile range) mRCR score was 1.56 (1.05–2.12) and the median wRCR score was 27.6 (6.97–88.44). Both metrics increased with each successive academic rank, except for department chairs. There was no difference in mRCR between male and female surgeons. Among assistant professors, males had higher wRCR scores. Both metrics were higher among surgeons with a PhD degree. The RCR offers key advantages over other indices, which are reflected in differences in score distributions compared with the widely used h‐index. Nevertheless, implementation of the RCR should be preceded with careful consideration of its own limitations.


| INTRODUCTION
Research output is an important measure of an author's productivity within their field and is generally regarded as an important factor when considering candidates for grant awards, job offers, or promotion and tenure. 1 Numerous bibliometric methods have been proposed to assess the scholarly contributions of scientific authors. The once commonly used impact factor, a journal-level metric, evaluated authors based on the journals in which they published. This approach has been largely abandoned due to overvaluing of less influential papers in popular journals and undervaluing of highly influential papers in lesserknown journals. More recently, the author-level Hirsch index, better known as the h-index, has become a highly prominent metric. [2][3][4] The original h-index proposed by physicist J. E. Hirsch in 2005 was defined as the number of papers (h) produced by an author that have at least h citations, excluding those papers with fewer than h citations. 5 For example, an author who produces exactly five papers which have each acquired five or more citations would earn an h-index of five.
Despite its adoption by the scientific community since it was first introduced, the h-index is limited by several shortcomings. As an author-level metric, the h-index is largely driven by the number of articles authored, potentially favoring individuals who publish more frequently, even if those articles are less impactful. 6 Similarly, the h-index may also demonstrate an age bias favoring older authors with a large repertoire of publications accrued over their careers, putting younger physicians at a disadvantage. 5,7-10 Furthermore, the h-index is not field-normalized. The citation potential of any study is inherently dependent on the size of the audience for which it is published, and directly comparing papers from different fields based on citation count alone would be unfair. As such, the h-index is not an adequate metric for comparison of authors belonging to different fields or different age brackets. 2,7,9,[11][12][13] In 2015, the National Institutes of Health (NIH) created a new article-level metric called the relative citation ratio (RCR) that was designed to overcome these issues with the h-index. 13 The RCR is defined as the total number of citations per year earned by a publication divided by the average citations per year received by NIH-funded articles in the same field (as determined by a cocitation network algorithm). In essence, the algorithm predicts the presence of a distinct academic community by identifying groups of articles that cite one another (also known as a "cocitation network"). For example, an article published in a pediatric orthopedics journal would likely cite several pediatrics studies and hopefully be cited by future pediatric studies-therefore, it would be part of a "network" of pediatrics orthopedics literature. The RCR for any given article is then adjusted to the citation potential of similar articles within its cocitation network. This cocitation network is a critical feature of the RCR, enabling field normalization and accurate comparison of papers across different scientific disciplines. 14,15 Individual publications, therefore, earn their own RCR. Two author-level derivatives of the RCR, the mean (mRCR) and weighted (wRCR) RCR, are calculated as the average and sum, respectively, of the article-level RCR scores of all publications produced by a given author. By averaging scores, the mean is independent of publication volume and more directly measures how frequently an authors' works are cited or their scientific "impact." Conversely, the wRCR is heavily influenced by the total quantity of publications and is more appropriate for measuring scholarly contributions over time, or "production." Over the last 6 years, utility of the RCR has been studied broadly, [15][16][17] as well as in the context of individual medical and surgical specialties, including radiation oncology, 8 ophthalmology, 18 and neurological surgery. 9 The principal objective of establishing benchmark data is to confirm that new metric behaves in the "expected" manner and

| Study design
Cross-sectional cohort study.

| Oxford center for evidence-based medicine level of evidence
Level II, prognostic.

| Inclusion and exclusion criteria
A list of orthopedic surgery residency programs in the United States was compiled using the Accreditation Council for Graduate Medical Education (ACGME) public-use residency program database. 20 Individual departmental websites for each ACGME-accredited residency program were accessed from July-September 2021. Nonphysicians, nonacademic faculty, and part-time or volunteer employees (i.e., adjunct professors, lecturers, and so forth) were excluded from this analysis. Gender, academic rank, and additional degrees were determined using surgeon profiles on departmental websites. Programs or faculty members with missing academic rank information were excluded from analyses involving academic rank but were retained for other analyses. Programs without complete faculty listings or those where it was unclear which faculty were full-time (n = 31) were excluded from the study altogether.

| RCR score collection
Academic orthopedic surgeons were individually indexed by name using the online NIH iCite database, 21 which includes PubMed-listed articles from 1980 to 2019 (calculation of the RCR is subject to a 2-year latency period). 16 mRCR and wRCR scores were collected for each surgeon in October 2021.   Figure 1B). Assistant professors had a lower wRCR than all other academic ranks including chairs (p < 0.0001). Associate professors had a lower wRCR than full professors and chairs (p < 0.0001 and p = 0.0002, respectively). Full professors had a higher wRCR than department chairs (p = 0.0184).

| DISCUSSION
There is interest in reliably quantifying research production and impact for academic surgeons, particularly for the purpose of considering candidates for grant awards, job offers, or promotion and tenure. 1 The NIH-supported RCR is a newer bibliometric that was designed to overcome key limitations of the widely adopted h-index, making it more suitable for comparison of different authors.
This study investigated the correlation between mRCR/wRCR scores and academic rank, gender, and PhD degree status for orthopedic surgeons. Here, we then compare our findings to previously reported data on the h-index and its variants. By examining patterns in the F I G U R E 2 (A and B) Median ± IQR (A) mRCR and (B) wRCR scores by gender. *p < 0.05; **p < 0.001; ***p < 0.0001. mRCR, mean RCR; RCR, relative citation ratio; wRCR, weighted RCR.

F I G U R E 4 (A and B) Median ± IQR (A) mRCR and (B) wRCR
scores by Ph.D. acquisition. *p < 0.05; **p < 0.001; ***p < 0.0001. mRCR, mean RCR; RCR, relative citation ratio; wRCR, weighted RCR. RCR, we hope to demonstrate its applicability in the field of orthopedic surgery and contribute to a growing body of evidence supporting its adoption by medical academia.
Scholarly output is generally considered an important (albeit not the only) prerequisite for advancement through faculty ranks in many specialties. Numerous studies have previously shown that bibliometrics including the h-index and RCR correlate positively with higher academic standing. 3,8,9,18 The data herein similarly demonstrate that both mRCR and wRCR increased significantly by successive academic rank from assistant through full professor, although the differences were more pronounced for wRCR. This is likely due to inherent differences in the underlying calculations for mRCR versus wRCR, as wRCR is sensitive to total number of publications and, by proxy, career duration. 9 As senior faculty positions are typically held by older and more experienced surgeons, larger jumps in wRCR with each consecutive academic rank highlight that these individuals have a greater total career research output than their junior faculty colleagues. Interestingly, department chairs did not have higher RCR metrics than professors, which contradicts previously reported findings with the h-index and its derivatives. 3 It is not clear why the RCR differs from the h-index in this respect, although this may be an effect of field normalization. Large subspecialties which contribute a greater fraction of chair positions may artificially inflate the h-index of the "chair" cohort, since those subspecialties also benefit from higher citation potential due to their audience size. The field-normalized RCR, by contrast, would not confer an advantage based on size alone, removing any undue influence of large subspecialties.
A prime example of the importance of time normalization lies with gender. wRCR was significantly higher for males than females, but mRCR was statistically similar. This gender pattern is consistent with previous studies of RCR trends in neurological surgery and radiation oncology. 8,9 However, when stratified by academic rank, the difference in wRCR persisted only among assistant professors.
Bastian et al. 3 reported the same finding in their h-index analysis, with no differences at higher ranks. Several prior studies investigating gender-based disparities in research productivity have similarly documented that while junior female faculty have lower h-indices than their male colleagues, both groups achieve equivalent productivity later in their careers. [22][23][24] Thus, although female surgeons may be publishing less frequently early in their careers, they experience a period of "catch-up productivity," matching their male colleagues later in their careers. 24 Interestingly, Bastian et al. 3 also found no difference in the time-adjusted m-index between male and female surgeons, which divides by the span of active publication years and is more appropriate for comparing authors with different career lengths, 5 much like the mRCR. These findings strongly suggest that observed disparities in scholarly "production" may also be due to women entering surgical fields relatively recently compared with men. Female surgeons, therefore, hold a much smaller number of senior faculty positions and have had less time to contribute to research. Statistically similar m-indices and mRCRs, however, indicate that male and female orthopedic surgeons have a comparable "impact" on the field; in other words, their work is cited with equivalent frequency. Additionally, it should be considered that, like most other bibliometric indicators, the RCR is not able to distinguish author seniority, so males and females who share authorship on a paper will benefit/suffer equally from its RCR score.
The data for PhD degree status further validate that mRCR and wRCR can be influenced by both research impact and production. As expected, surgeons with a PhD degree had higher mRCR and wRCR scores than those without, reflecting their production of highly impactful and more frequently cited literature. Notably, the difference in wRCR between these groups was more pronounced than the difference in mRCR, indicating that surgeons with a PhD publish more prolifically. The correlation between PhD acquisition and research productivity has been previously documented in various medical and surgical specialties, although there is some debate about the influence on scholarly impact. 8,9,18,25,26 Our results also suggest that academic orthopedic surgeons are remarkably productive compared with their colleagues in other specialties. In our study, orthopedic surgeons had a median mRCR

| Limitations
Although the RCR is specifically designed to overcome key limitations of the h-index, it is not without its own shortcomings. As with all currently available metrics, the RCR is unable to capture author contributions, for which authorship order is an important surrogate. This may be particularly problematic for large, multicenter, prospective studies with lengthy author lists, allowing some intermediate authors to benefit from tremendous citation potential despite having contributed considerably less than the primary investigators. The Matthew effect-wherein renowned authors benefit from greater citation attention and therefore higher citation metrics regardless of the objective impact of their work-has been previously described for the h-index and is likely applicable to the RCR as well. 27 Deliberate self-citation has also been discussed as a potential source of skewed bibliometric data, although one study with the h-index suggests that the practical effects of this are minimal. 28 Citation attention due to intense debate over highly controversial articles is also difficult to distinguish from citations earned through scholarly merit and genuine scientific impact, 3 although this is unlikely to have a substantial effect on the average scores of large cohorts as in our study. Furthermore, the accuracy of our data is highly dependent on whether individual departmental websites are well-maintained and up-to-date. When examined at the individual level, all bibliometrics (including the RCR, h-index, and so forth) should only be evaluated with appropriate contextual understanding of a physician's background and accomplishments to acquire a more holistic view of his/her academic career.
In other words, one should not use any bibliometric indicator as the sole measure of a person's career.
More specifically to the RCR, the "cocitation network" algorithm underlying field normalization has received some criticism. 16 Waltman 29 outlines a theoretical scenario in which a paper published in a very small field suddenly garners broader attention and is subsequently cited heavily by papers in other, larger fields-would the RCR of this paper go down since its cocitation network has been enlarged? Furthermore, in very small subspecialties, the distinction between high and low-impact papers may be less clear based on citation counts alone, such that high-impact papers in large subspecialties may earn a similar RCR to lower-impact papers in very small, less research-oriented subspecialties. Nonetheless, Bornmann et al. 17 have shown that the RCR maintains a high degree of correlation with other field-normalized bibliometric indicators including the mean normalized citation score, citation percentile, and source-normalized citation score 2.
There are also considerable limitations of the online NIH iCite database. The search mechanism is unable to differentiate between individuals with the same name, which can lead to errors.
We rigorously attempted to minimize this effect by including middle initials and excluding individuals for whom iCite reported an unjustifiable total publication count (>1000). Notably, we included only 29 surgeons (1% of the study sample) with total publication counts exceeding 500, so it is unlikely that this Similarly, the RCR is subject to a 2-year latency period to allow for field-normalization, which means that articles from the last 2 years are unavailable in the iCite database. 16

| CONCLUSION
In conclusion, the RCR is a relatively new field-and time-normalized metric that is designed to overcome certain key limitations of commonly used bibliometric indicators such as the h-index. Our data demonstrate that RCR is positively correlated with advanced academic rank and PhD acquisition among academic orthopedic surgeons, consistent with findings in other medical and surgical fields.
Male sex was associated with higher weighted-but not mean-RCR at the assistant professor level. This may be due to female surgeons publishing less frequently during their early careers, although the difference in research production disappears at higher ranks.

ETHICS STATEMENT
This research was performed using only publicly available information on public domains and does not contain any identifying information.
No health information, protected or otherwise, was accessed. This study was deemed "nonhuman research" and therefore was not subject to review by our institutional review board.