A critical assessment of the h-index



Editor's suggested further reading in BioEssays: Can we do better than existing author citation metrics? Abstract and Counting citations in texts rather than reference lists to improve the accuracy of assessing scientific contribution Abstract

Over the years various indices have been developed as a way to measure scientific performance and achievement 1, 2. These include Hirsch's “h-index” 3, Egghe's “g-index” 4, and Zhang's “e-index” 5, among others 1, 2. Metrics enterprises such as Thomson Reuters 6, Google (Google scholar) 7, and Elsevier (Scopus) 8, are routinely referred to in order to identify and quantify published work and citations, and to establish an order of ranking. Of the many performance indicators available, the “h-index” has become one of the most commonly used. This index has recently gained prominence since it is currently being used by major citation services such as the ISI Web of Knowledge (Thomson Reuters) 6 and Scopus (Elsevier) 8. It is now often used to make employment and promotion decisions, by grant evaluation committees, and even in comparisons between academic institutions.

The “h-index” was configured in 2005 by the physicist Jorge E. Hirsch in an effort to quantitate an individual researcher's publication record 3. The index reflects both the number of articles that a scientist has published and the number of citations that each publication has received. A scientist has index h if h of his or her Number of papers (Np) have received at least h citations each, and the remaining papers (Np − h) have ≤h citations each 3 (Fig. 1). The “h-index” improves as the number of cited publications and the number of citations per publication increases. This generally increases as the number of years the scientist has spent in academia increases. The h-index is by definition not dependent on the number of publications of a given scentist, but rather on how often his/her publications are cited. Indices based on citations can pose a problem for early career scientists since high citation numbers are dependent on either time or the field of research. Ideally, the h-index should be used to compare scientists with the same academic age and similar fields as citation patterns can vary considerably between different areas of research. Although the h-index is not defined by the number of publications a scientist has, the question remains as to whether the h-index is affected by the number of publications among scientists in the same research field.

Figure 1.

The h-index. Number of manuscripts and their citations for a single scientist.

In order to address this question, we analyzed the h-indices and the corresponding number of publications of 248 professors (associate professors and full-time professors) in the health sciences at the universities of Odense, Aarhus, and Copenhagen in Denmark. We determined the number of publications for each professor using Pubmed and corrected for persons with the same name but working in different scientific fields or institutions. Next we obtained the h-index for each professor via the Web of knowledge and cross-checked these with our Pubmed data to ensure that all papers contributing to the h-index were, in fact, authored or co-authored by the given professor (data covered until 1 July 2011). Figure 2 shows the relationship between the number of publications and the corresponding h-index for each of the 248 Danish professors in the health sciences. The h-index increased non-linearly with the number of publications for senior scientists. The association between number of publications and the corresponding h-index was best described using the equation h(n) = n0.694 (SPSS version 17). The correlation coefficient is r = 0.93 (p < 0.001, ANOVA) (Fig. 2), indicating that 87% of the variation in the h-index can be explained by the number of publications each scientist has. The number of publications increases faster than the h-index. This is expected according to the index definition, since increases in the h-index for a given scientist require that newer publications must accrue more citations in less time in order to have a similar impact on the h-index as older ones. Thus, it is inevitable that the h-index starts to plateau over time, regardless of how many new papers are published by the same scientist (Fig. 1).

Figure 2.

Association between the h-index of 248 professors and their corresponding total number of publications. We analyzed 248 professors (associate professors and full time professors) in the health sciences at the universities of Odense, Aarhus, and Copenhagen in Denmark. We identified the number of publications of each professor in Pubmed, and corrected for persons with the same name but working in different scientific areas or institutions. We also obtained the h-index of each professor through the Web of knowledge, and ensured that all papers contributing to the h-index were, in fact, authored or co-authored by the given professor through cross-checking the Pubmed publication list. The relationship between h-index and the number of publications is best described by a power function given by h(n) = n0.694 (SPSS version 17). The correlation coefficient r = 0.93 (p < 0.001, ANOVA).

These analyses showed that the h-index follows a specific pattern with a close association to the number of publications. Importantly, this study also took into account the fact that the scientists were at the same career level, from a limited subject area (Health Sciences), and from a small group of universities of similar quality, in the same country and thus subject to the same national citation patterns. This close association seems surprising as the h-index itself is not defined by the total number of publications a scientist has obtained. This raises the question as to what extent the h-index represents the quantity rather than the quality of a scientist's publications. Bornmann et al. 9 investigated nine different variants of h-indices by factor analysis and could divide these into two classes: one type of indices (e.g. h-index) describes the number of papers a scientist has published, which fits with our observations; the other type of indices indicates the impact of the publications (e.g. m-index).

Most scientists consider a high number of citations of their publications as a sign of recognition of their work by the scientific community, which would explain why the h-index seems reasonable. However, the citation rate of a given publication can be influenced by various factors (please refer to 10, 11) i.e. search database, discipline, institution, journal impact factor, length of article, number of authors, seniority of author, type of publication (review or original), gender and traditions for citation in a country, self-citations, and citations of colleagues. Moreover, a given citation rate gives no indication about which part of the publication has been cited, nor whether the scientific context is a negative finding, false or even a retracted publication, or to what extent an author contributed to a given publication. Given the variety of factors that influence and bias citation rates, it is evident that citations alone cannot be an evaluation marker of scientific quality.

The h-index is by definition based on citations but does not include the dynamics of citations. For example, if a highly cited paper is included in the index, it will not contribute further to the index regardless of how many additional citations the paper would obtain in the following years (Fig. 1). This means that it is difficult for a scientist with a limited number of excellent papers to obtain a high h-index, and moreover, the h-index cannot distinguish ground-breaking scientific papers from more conventional scientific studies. This can be further substantiated by the following example: two scientists (A) and (B) have eight publications each, where (A) has the following citation numbers 30, 20,15, 10, 8, 4, 2, 1, and scientist (B) has the following citation numbers 90, 80, 10, 8, 6, 4, 2, 1. Although scientist (B) has more than twice as many citations as scientist (A), both scientists will have the same h-index (h-index 5), indicating that the h-index evaluates quantity more than quality.

In order to overcome these limitations, Bornmann et al. 9 suggested to use pairs of indices as a meaningful indicator for comparing scientists. One index should relate to the number of papers and the other to the impact of the papers. However, in January 2011, the French Academy of Sciences published a report on current citation metric methods for evaluating individual researchers which has been commented on and summarized by Sahel 12; “Bibliometrics have enormous potential to assist the qualitative evaluation of individual researchers; however, none of the bibliometric indicators alone (or even considering a set of them) allow for an acceptable and well-balanced evaluation of the activity of a researcher”.

Evaluating scientific performance is a complex task that in the future may be more focused on the purpose of the evaluation, whether that be recruitment, promotion, or funding allocation. In order for this to happen, evaluations will need to be performed based on adequately validated bibliometric indices.

In summary, the h-index is a widely used performance indicator based on scientists' citations. The present study shows that the nature of the h-index is quantitative rather than qualitative since the h-index is closely associated to a scientist's number of publications. The h-index may be one of the most used performance indices at present, but may not, in that context, be the best.