ACADEMIC EMERGENCY MEDICINE 2011; 18:898–904 © 2011 by the Society for Academic Emergency Medicine
Objectives: Emergency medicine (EM) is a young but rapidly growing field. An evaluation of academic performance and the growing impact of EM journals would help to elucidate the increase in the number of EM scientific studies. The authors used the Journal Citation Reports (JCR) database to investigate the scientific achievements of EM journals in the past 10 years.
Methods: This was a literature review study. All data were collected from the JCR database. Journals listed in the EM category from 2000 to 2009 were included. Eleven categories that were considered most closely related to EM by a consensus of the authors were chosen for comparison, including cardiac and cardiovascular systems, clinical neurology, critical care medicine, gastroenterology and hepatology, infectious diseases, general and internal medicine, pediatrics, respiratory system, surgery, toxicology, and urology and nephrology. Data on journals in the EM category were collected, including journal title, language, journal country or territory, impact factor for each year, total number of EM journals for each year, and the EM category aggregate impact factor (available from 2003 to 2009). The variables in the comparison group included the number of journals in each of the 11 clinical medicine categories from 2000 to 2009 and the aggregate impact factors for 2003 to 2009. The category aggregate impact factor and journal impact factor were adopted as representative of category and journal academic performance. Linear regression was used to assess the trend of aggregate impact factor and journal impact factor. The slope (β) of the linear regression was used to represent the evolution of performance. The relationship between the 2000 EM journal impact factor and the impact factor trend of EM journals between 2000 and 2009 was measured by Pearson correlation coefficient to evaluate the evolution difference between journals with different initial impact factors.
Results: In 2000, all 12 EM journals were published in the United States or Europe, and the language of all was English. In 2009, 10% (2/19) of the journals originated from outside North America and Europe, and 16% (3/19) were non–English-language journals. The number of EM journals increased 58% from 2000 to 2009, twice the increase in the total number of JCR-listed journals, and rank first in the rate of journal number increase among categories of clinical medicine. The impact factor of all EM journals showed an increasingly positive trend since 2000. The impact factor increased faster for high impact factor EM journals than for low-impact-factor EM journals.
Conclusions: An increasing number of international EM journals have appeared over the past 10 years. Every EM journal exhibited a positive impact factor trend, but the gap between EM journals’ impact factors has widened in the past 10 years.
The idea of an impact factor was first suggested by Garfield in Science in 1955 as a means to help select journals and avoid selection bias that results from neglecting influential journals with smaller article counts.1 This idea led to the 1961 publication by the Institute for Scientific Information (ISI) of the Science Citation Index (SCI).2 Now, impact factors are recorded yearly in Journal Citation Reports (JCR) published by Thomson Reuters (New York, NY). JCR has been integrated with the ISI Web of Knowledge for Web-based searching. The impact factor of a journal is defined as the number of citations within a certain year to all articles published in that journal during the previous 2 years (the numerator), divided by the number of articles in that journal published during the previous 2 years (the denominator).3 The impact factor is now widely used to compare journal performance within particular scientific categories. Although many debates exist about impact factors, using a journal impact factor rather than article or author impact factor is likely a reasonable measure for evaluating journal impact.4
Emergency medicine (EM) is a relatively young specialty in clinical medicine.5 In the past 10 years, the clinical service provided and the importance of EM in the health care system have increased significantly in the United States.6 In contrast, Gallagher and Barnaby7 discovered the methodologic bias of the impact factor from no EM journal being included in SCI citation source journals in the 1990s, which led to an underestimate of academic development of EM. The progress in academic performance of EM journals has rarely been studied in the past decade. An objective and multiperspective evaluation of academic performance of EM journals might provide a more complete understanding of the recent evolution of EM scientific studies. The goal of this investigation was to compare academic performance in EM journals with that of other clinical medicine categories by using estimates for the trends in category aggregate impact factors and journal impact factors from the JCR database.
This was a literature review study and qualified for a waiver from our institutional review board because no human subjects were involved.
Study Setting and Population
All data were collected from the 2000 to 2009 JCR database. All journals included in the EM category of JCR from 2000 to 2009 were enrolled. In 2009 JCR, there were a total of 173 categories of journal.8 Within these categories, 47 categories were related to clinical medicine and 11 categories were considered most closely related to EM by a consensus of the authors and were chosen for comparison, including cardiac and cardiovascular systems, clinical neurology, critical care medicine, gastroenterology and hepatology, infectious diseases, general and internal medicine, pediatrics, respiratory system, surgery, toxicology, and urology and nephrology (a complete listing is available in Data Supplement S1, available as supporting information in the online version of this paper).
We collected data on journals in the EM category including journal title, language, journal country or territory, impact factor for each year, total number of EM journals for each year, and the EM category aggregate impact factor. The aggregate category impact factor has been calculated by JCR and has been available in the JCR category summary list since 2003. The calculation is based on the same concept as the journal impact factor, but represents the performance of a category rather than of an individual journal. The aggregate impact factor of a category is defined as the citations within a certain year to all articles in a category published in the previous 2 years (the numerator), divided by the number of articles published in that category in the previous 2 years (the denominator). The variables in the comparison group included the number of journals in each of the 11 clinical medicine categories in 2000 and 2009 and the aggregate impact factors for 2003 to 2009. The total number of journals in the JCR Science Edition in 2000 and 2009 was also recorded to compare the total number of journals according to category.
The category aggregate impact factor and journal impact factor were adopted as representative of category and journal academic performance. The aggregate impact factor trend and journal impact factor trend were used to represent the evolution of their performance. The primary outcome measure in the comparison of categories of clinical medicine was the trend for the aggregate impact factor. The primary outcome measure for the comparison among the EM journals was the trend in the journal impact factor.
Category journal numbers partially reflected the quantity and quality of studies in that field. Category rank for journal numbers and journal number increase rates were measured to evaluate category performance in addition to the aggregate impact factor trend. The relationship between the 2000 EM journal impact factor and the impact factor trend of EM journals between 2000 and 2009 was measured to evaluate the evolution difference between journals with different initial impact factors.
The EM journal numbers, countries, languages, and category journal numbers were analyzed using descriptive statistics. Spearman’s rank correlation coefficient (rs) was used to evaluate the correlation between category rank for journal numbers in 2000 and 2009 and increase rates in journal numbers. Bonferroni correction was adapted for multiple comparisons on correlations.
Linear regression was used to evaluate the trend in the category aggregate impact factors and the trend in impact factors for individual EM journals under the assumption of linear relationships between impact factors and time, and that normality was not strongly violated. The slope (β) of the linear regression was adopted as representative of trends in the aggregate impact factor and trends in the EM journal impact factor. The 95% confidence intervals (CIs) of β were calculated.
Pearson correlation coefficient (r) was used to evaluate the association between the 2000 impact factor and the impact factor trend from 2000 to 2009. A p-value of <0.05 was considered statistically significant. Data were analyzed with SAS version 9.2 (SAS Institute, Cary, NC).
EM Journal Numbers, Countries, and Languages
Twelve journals were included in the EM category in the 2000 JCR, and four journals were excluded between 2001 and 2006 because of title changes, category changes, or termination of publication. Thus, 11 new journals were listed in the EM category after 2000, and the category journal number increased to 19 in 2009 (Table 1). In 2000, 12 journals were published in four countries: eight in the United States, two in England, one in Germany, and one in Ireland. In 2009, 19 journals were published in seven countries: nine in the United States; four in Germany; two in England; and one each in Ireland, Australia, Croatia, and Turkey. In 2000, all 12 journals were English-language journals. In 2009, the 19 journals were published in three languages; these included 16 in English, two in German, and one in Turkish.
|Title of Journal*||Language||Country||Impact Factor in 2000||Impact Factor in 2009||Note†|
|Annals of Emergency Medicine||English||United States||2.183||4.232|
|Academic Emergency Medicine||English||United States||1.419||2.478|
|American Journal of Emergency Medicine||English||United States||1.054||1.542|
|Emergency Medicine Journal||English||England||—||1.477||2002|
|Prehospital Emergency Care||English||United States||—||1.297||2008|
|Journal of Emergency Medicine||English||United States||—||1.265||2001|
|Emergency Medicine Clinics of North America||English||United States||0.635||0.964|
|Pediatric Emergency Care||English||United States||0.428||0.916|
|Emergency Medicine Australasia||English||Australia||—||0.901||2009|
|European Journal of Emergency Medicine||English||United States||—||0.733||2009|
|Notfall and Rettungsmedizin||German||Germany||—||0.573||2009|
|Journal of Emergency Nursing||English||United States||—||0.359||2007|
|Ulusal Travma ve Acil Cerrahi Dergisi-Turkish (Journal of Trauma and Emergency Surgery)||Turkish||Turkey||—||0.274||2009|
|European Journal of Trauma and Emergency Surgery||English||Germany||—||0.207||2009|
|Critical Care Clinics||English||United States||1.143||—||Category|
|Journal of Burn Care and Rehabilitation||English||United States||0.810||—||Category and title|
|Journal of Accident and Emergency Medicine||English||England||0.668||—||Title|
|New Horizons: The Science and Practice of Acute Medicine||English||United States||0.619||—||End|
Journal Number Ranks and Aggregate Impact Factor Trend for Clinical Medicine Categories
The number of categories of clinical medicine journals in 2000 and 2009, changes in journal numbers, the rates of increase, and their ranks are summarized in Table 2. EM journal numbers ranked last in 2000 and 2009, but the 58% rate of increase ranked first and was twice the rate of increase seen for the total number of journals in the JCR Science Edition. Spearman’s rank correlation coefficient (rs) was calculated to analyze the correlation between journal number ranks in 2000 (ranking 1), journal number ranks in 2009 (ranking 2), journal number increase ranks (ranking 3), and ranks of increase rate in journal number (ranking 4). This analysis revealed that the rank for category journal numbers in 2009 was highly positively correlated with that rank in 2000 (ranking 1 × ranking 2, rs = 0.97, p < 0.001). The rank for journal number increase rate was negatively associated with journal number rank in 2000 (ranking 1 × ranking 4, rs = −0.84, p < 0.001)
|Category*||2000||Rank 1†||2009||Rank 2†||Increase‡||Rank 3†||Rate§||Rank 4†|
|Cardiac and cardiovascular system||63||6||98||4||35||1||56||3|
|Critical care medicine||15||11||22||11||7||10||47||5|
|Gastroenterology and hepatology||45||7||65||7||20||7||44||7|
|Medicine, general and internal||105||3||132||3||27||4||26||9|
|Urology and nephrology||43||8||63||8||20||7||46||6|
The aggregate EM impact factor in 2003 was 1.127, which placed EM last among 12 categories of clinical medicine. EM remained at the same rank in 2009, but showed a substantial increase in impact factor of 0.507, representing a 45% increase in citation rate, which was the highest rate of increase among the clinical medicine categories. A linear regression analysis of each category aggregate impact factor from 2003 to 2009 showed that every category had a positive trend (Table 3), indicating that the aggregate impact factor for every category increased significantly from 2003 to 2009, except the category “Medicine, general and internal” (β = 0.052, p = 0.356).
|Cardiac and cardiovascular system||0.075||0.005||0.034 to 0.116|
|Clinical neurology||0.107||<0.001||0.086 to 0.129|
|Critical care medicine||0.121||0.003||0.064 to 0.177|
|Emergency medicine||0.086||<0.001||0.056 to 0.116|
|Gastroenterology and hepatology||0.083||0.040||0.006 to 0.159|
|Infectious diseases||0.087||0.012||0.029 to 0.145|
|Medicine, general and internal||0.052||0.356||−0.080 to 0.185|
|Pediatrics||0.075||0.003||0.038 to 0.112|
|Respiratory system||0.085||0.002||0.047 to 0.123|
|Surgery||0.091||<0.001||0.063 to 0.118|
|Toxicology||0.101||<0.001||0.076 to 0.126|
|Urology and nephrology||0.071||<0.001||0.047 to 0.095|
EM Journal Impact Factor Trend
Eight EM journals included in JCR since 2000 were enrolled for the impact factor trend analysis. The impact factor for all EM journals increased between 2000 and 2009 (Figure 1). A linear regression analysis of the journal impact factor from 2000 to 2009 revealed that every journal exhibited a positive trend, indicating that each journal impact factor increased significantly over this 10-year period (Table 4). The relationship between the 2000 EM journal impact factor and the impact factor trend is plotted in Figure 2. Injury appeared to be a significant outlier. The Pearson correlation coefficient (r) before and after the outlier was excluded was 0.49 and 0.94, respectively. This result indicated that the impact factor increased trend of EM journals between 2000 and 2009, excluding Injury, has a strong linear relationship with the 2000 impact factor.
|Title of Journal*||Trend||p-value||95% CI|
|Annals of Emergency Medicine||0.223||<0.001||0.169 to 0.277|
|Resuscitation||0.139||0.013||0.039 to 0.240|
|Academic Emergency Medicine||0.125||<0.001||0.076 to 0.174|
|American Journal ofEmergency Medicine||0.029||0.435||−0.053 to 0.111|
|Emergency Medicine Clinics of North America||0.039||0.017||0.009 to 0.069|
|Unfallchirurg||0.021||0.023||0.004 to 0.038|
|Pediatric Emergency Care||0.040||0.002||0.019 to 0.062|
|Injury||0.221||<0.001||0.158 to 0.283|
The aggregate impact factor is a reference point indicating average category performance. The number of EM journals increased from 10 in 2003 to 19 in 2009, and the impact factors of four journals were consistently higher than the aggregate impact factor each year (Figure 1). As the total journal number increased 90%, this indicates that the rate of journals with impact factors higher than the aggregate impact factor decreased from 40% (4/10) to 21% (4/19). The impact factors for Academic Emergency Medicine, Annals of Emergency Medicine, and Resuscitation have been steadily higher than the aggregate impact factor since 2003. The impact factor for the American Journal of Emergency Medicine was higher than the aggregate impact factor from 2003 to 2006. The impact factor for Injury has been higher than the aggregate impact factor since 2007.
Comparing the trend in the impact factors for individual EM journals and the aggregate category impact factor discloses which journals have greater than average rate for increase. Aggregate EM impact factors and the impact factors for 10 EM journals included in JCR since 2003 were used in a trend analysis. A linear regression analysis of journal impact factors from 2003 to 2009 revealed that the trend for the Annals of Emergency Medicine and Injury increased significantly more than the category aggregate impact factor (Annals 95% CI = 0.198 to 0.356, Injury 95% CI = 0.233 to 0.382, and aggregate 95% CI = 0.056 to 0.116). No significant differences were observed between the remaining journal impact factor trends and the aggregate category impact factor trend (Table 5).
|Title of Journal*||Trend||p-value||95% CI|
|Annals of Emergency Medicine||0.277||<0.001||0.198 to 0.356|
|Resuscitation||0.215||0.044||0.008 to 0.423|
|Academic Emergency Medicine||0.115||0.031||0.016 to 0.215|
|Injury||0.308||<0.001||0.233 to 0.382|
|American Journal of Emergency Medicine||−0.069||0.261||−0.210 to 0.071|
|Emergency Medicine Journal||0.143||0.001||0.086 to 0.199|
|Journal of Emergency Medicine||0.081||0.031||0.011 to 0.152|
|Emergency Medicine Clinics of North America||0.028||0.304||−0.035 to 0.090|
|Pediatric Emergency Care||0.061||0.011||0.021 to 0.101|
|Unfallchirurg||0.021||0.190||−0.015 to 0.057|
|Aggregate impact factor||0.086||<0.001||0.056 to 0.116|
Impact factor is an imperfect method to evaluate the academic performance of journals, but no better measure of scientific performance is currently available.9 The journal impact factor is still the most widely accepted quantitative and objective method for evaluating journals. Our analysis of the SCI database from a variety of viewpoints revealed the evolution of academic performance in EM journals.
In EM journal countries and languages comparison, the extended geographic distribution and the increase in language diversity indicate qualitative and quantitative advances in EM journals around the world. This trend was also evident in the diversity of submissions by country to one of the leading peer-reviewed EM journals.10
Before 1990, EM journals were diffusely classified by JCR into the categories of anesthesiology, surgery, or miscellaneous. In 1991, the “critical care” category was first used, and EM journals were reclassified into that category. In 1996, the critical care category was renamed “emergency medicine and critical care.” In 2000, this category was separated into “critical care medicine” and “emergency medicine.” The method for categorizing journals in the JCR is heuristic,3 but a new category generally means that that the quantity and quality of studies in that field are sufficient to warrant an independent category. In our study period, after EM being an independent category in JCR, EM is the smallest, youngest category, but maintained the highest journal number increase rate within the 12 categories of clinical medicine.
We adopted the aggregate category impact factor as representative of average academic performance in a category. The range of clinical medicine aggregate impact factors is wide (medicine, general and internal vs. emergency medicine: 3.971 vs. 1.127 in 2003, 4.108 vs. 1.634 in 2009). Thus, directly comparing the aggregate impact factors may mask the aggregate impact factor change of smaller, newer categories. The aggregate impact factor trend more precisely reflects the growth rate of the category. The 95% CIs for all category trends overlapped, which may have resulted from the limited time period of available aggregate impact factors in JCR, but may also indicate that EM journals’ academic performance as measured by the SCI impact factor was comparable to that for other categories.
The possible confounding effect of natural increase of impact factor (aggregate impact factor simultaneously) should be discussed. In an era of expanding literature, more citations were produced as more articles were published. The net effect to impact factor depends on the increasing rate of JCR articles and the average amount of references per article.11 If these two factors show an increasing trend, the impact factor will increase naturally. Every category and journal was influenced by this effect. In our study, we compared the trend between different category aggregate impact factors and the trend between journal impact factors during the same period of time instead of the trend of a single category or journal. This can offset the effect and disclose the impact factor trend in addition to natural increase.
The gap between EM journals’ impact factors has been widening in the past 10 years. The impact factor increased faster for high-impact-factor journals than for low-impact-factor journals. The ratio of EM journals with impact factors higher than the aggregate impact factor to the total number of EM journals decreased. Two journals maintained greater impact factor trends than the aggregate impact factor trends. Taken together, these results suggest that the most influential studies are published in a few EM journals, leading to uneven impact factor increments among EM journals. This phenomenon is similar to that found in previous studies regarding citations of EM journal articles.12 The uneven distribution of citations to articles has been mentioned in previous literature.3 Some articles account for the majority of citations, but most articles are seldom cited. At the level of journals, the polarization of EM journal impact factor increase has never been mentioned before. Whether this polarization is also occurring in other disciplines is still unknown. This phenomenon may be explained by a positive-feedback hypothesis. A high impact factor represents a journal with high visibility, prestige, and influence. Researchers with well-designed studies and significant results tend to submit their articles to journals with these characteristics.13 The editor’s acceptance criteria will be higher while reviewing high-quality submissions. Publishing high-quality articles leads to more citations, which increases the journal impact factor. A positive-feedback loop is formed.
The impact factor of Injury has had a substantial increase in the past 10 years, and its impact factor has increased out of proportion to its 2000 impact factor compared with other EM journals. Impact factor is influenced by the peer review process, editorial evaluation, and research field variables. The ability of a field to be cited by other fields is considered the most important field variable.14 Aside from peer review process and editorial evaluation, one difference between Injury and other EM journals in JCR is that Injury is classified into four categories: critical care medicine, emergency medicine, orthopedics, and surgery. The remaining seven EM journals in our study are classified into one or two categories. Although the scope of a journal is defined by its editorial board, JCR establish journal categories partly based on citation relatedness in recent years.3 Multiple categories classification in JCR might represent citations from broad disciplines, and its influence on journal impact factor needs further investigation.
We used JCR impact factors as our main study data; therefore, limitations of the JCR database also apply to our study.11 The JCR database is an English-language resource and has an American publication bias; therefore, the contribution of non–English-language or non-American journals may be underestimated.3,11
Emergency medicine has been a JCR category since 2000, and the category aggregate impact factor has been available since 2003. Hence, relatively limited data were used to obtain the trends, so the 95% CIs overlapped. A longer study period in the future may reveal a more precise trend.
The quality and quantity of emergency medicine scientific studies have increased substantially in the past 10 years. An increasing number of international EM journals have appeared over the past 10 years. The trend in aggregate impact factor in EM journals is not dissimilar to that of other clinical medicine categories. Every EM journal exhibited a positive impact factor trend, with a widening gap between journals’ impact factors. Whether the polarization of journal impact factor is a common phenomenon in other disciplines remains unknown.