The Evolution of Academic Performance in Emergency Medicine Journals: Viewpoint from 2000 to 2009 Journal Citation Reports

Authors

  • Ching-Hsing Lee MD,

    1. From the Department of Emergency Medicine, Chang Gung Memorial Hospital and Chang Gung University College of Medicine (CHL, YCC, CHC), and the Graduate Institute of Business and Management, Chang Gung University (CPS), Taoyuan, Taiwan.
    Search for more papers by this author
  • Chia-Pang Shih MHA,

    1. From the Department of Emergency Medicine, Chang Gung Memorial Hospital and Chang Gung University College of Medicine (CHL, YCC, CHC), and the Graduate Institute of Business and Management, Chang Gung University (CPS), Taoyuan, Taiwan.
    Search for more papers by this author
  • Yu-Che Chang MD,

    1. From the Department of Emergency Medicine, Chang Gung Memorial Hospital and Chang Gung University College of Medicine (CHL, YCC, CHC), and the Graduate Institute of Business and Management, Chang Gung University (CPS), Taoyuan, Taiwan.
    Search for more papers by this author
  • Chung-Hsien Chaou MD

    1. From the Department of Emergency Medicine, Chang Gung Memorial Hospital and Chang Gung University College of Medicine (CHL, YCC, CHC), and the Graduate Institute of Business and Management, Chang Gung University (CPS), Taoyuan, Taiwan.
    Search for more papers by this author

  • The authors have no relevant financial information or potential conflicts of interest to disclose.

  • Supervising Editor: James Adams, MD.

Address for correspondence and reprints: Dr. Chung-Hsien Chaou, MD; e-mail: shien@url.com.tw.

Abstract

ACADEMIC EMERGENCY MEDICINE 2011; 18:898–904 © 2011 by the Society for Academic Emergency Medicine

Abstract

Objectives:  Emergency medicine (EM) is a young but rapidly growing field. An evaluation of academic performance and the growing impact of EM journals would help to elucidate the increase in the number of EM scientific studies. The authors used the Journal Citation Reports (JCR) database to investigate the scientific achievements of EM journals in the past 10 years.

Methods:  This was a literature review study. All data were collected from the JCR database. Journals listed in the EM category from 2000 to 2009 were included. Eleven categories that were considered most closely related to EM by a consensus of the authors were chosen for comparison, including cardiac and cardiovascular systems, clinical neurology, critical care medicine, gastroenterology and hepatology, infectious diseases, general and internal medicine, pediatrics, respiratory system, surgery, toxicology, and urology and nephrology. Data on journals in the EM category were collected, including journal title, language, journal country or territory, impact factor for each year, total number of EM journals for each year, and the EM category aggregate impact factor (available from 2003 to 2009). The variables in the comparison group included the number of journals in each of the 11 clinical medicine categories from 2000 to 2009 and the aggregate impact factors for 2003 to 2009. The category aggregate impact factor and journal impact factor were adopted as representative of category and journal academic performance. Linear regression was used to assess the trend of aggregate impact factor and journal impact factor. The slope (β) of the linear regression was used to represent the evolution of performance. The relationship between the 2000 EM journal impact factor and the impact factor trend of EM journals between 2000 and 2009 was measured by Pearson correlation coefficient to evaluate the evolution difference between journals with different initial impact factors.

Results:  In 2000, all 12 EM journals were published in the United States or Europe, and the language of all was English. In 2009, 10% (2/19) of the journals originated from outside North America and Europe, and 16% (3/19) were non–English-language journals. The number of EM journals increased 58% from 2000 to 2009, twice the increase in the total number of JCR-listed journals, and rank first in the rate of journal number increase among categories of clinical medicine. The impact factor of all EM journals showed an increasingly positive trend since 2000. The impact factor increased faster for high impact factor EM journals than for low-impact-factor EM journals.

Conclusions:  An increasing number of international EM journals have appeared over the past 10 years. Every EM journal exhibited a positive impact factor trend, but the gap between EM journals’ impact factors has widened in the past 10 years.

The idea of an impact factor was first suggested by Garfield in Science in 1955 as a means to help select journals and avoid selection bias that results from neglecting influential journals with smaller article counts.1 This idea led to the 1961 publication by the Institute for Scientific Information (ISI) of the Science Citation Index (SCI).2 Now, impact factors are recorded yearly in Journal Citation Reports (JCR) published by Thomson Reuters (New York, NY). JCR has been integrated with the ISI Web of Knowledge for Web-based searching. The impact factor of a journal is defined as the number of citations within a certain year to all articles published in that journal during the previous 2 years (the numerator), divided by the number of articles in that journal published during the previous 2 years (the denominator).3 The impact factor is now widely used to compare journal performance within particular scientific categories. Although many debates exist about impact factors, using a journal impact factor rather than article or author impact factor is likely a reasonable measure for evaluating journal impact.4

Emergency medicine (EM) is a relatively young specialty in clinical medicine.5 In the past 10 years, the clinical service provided and the importance of EM in the health care system have increased significantly in the United States.6 In contrast, Gallagher and Barnaby7 discovered the methodologic bias of the impact factor from no EM journal being included in SCI citation source journals in the 1990s, which led to an underestimate of academic development of EM. The progress in academic performance of EM journals has rarely been studied in the past decade. An objective and multiperspective evaluation of academic performance of EM journals might provide a more complete understanding of the recent evolution of EM scientific studies. The goal of this investigation was to compare academic performance in EM journals with that of other clinical medicine categories by using estimates for the trends in category aggregate impact factors and journal impact factors from the JCR database.

Methods

Study Design

This was a literature review study and qualified for a waiver from our institutional review board because no human subjects were involved.

Study Setting and Population

All data were collected from the 2000 to 2009 JCR database. All journals included in the EM category of JCR from 2000 to 2009 were enrolled. In 2009 JCR, there were a total of 173 categories of journal.8 Within these categories, 47 categories were related to clinical medicine and 11 categories were considered most closely related to EM by a consensus of the authors and were chosen for comparison, including cardiac and cardiovascular systems, clinical neurology, critical care medicine, gastroenterology and hepatology, infectious diseases, general and internal medicine, pediatrics, respiratory system, surgery, toxicology, and urology and nephrology (a complete listing is available in Data Supplement S1, available as supporting information in the online version of this paper).

Study Protocol

We collected data on journals in the EM category including journal title, language, journal country or territory, impact factor for each year, total number of EM journals for each year, and the EM category aggregate impact factor. The aggregate category impact factor has been calculated by JCR and has been available in the JCR category summary list since 2003. The calculation is based on the same concept as the journal impact factor, but represents the performance of a category rather than of an individual journal. The aggregate impact factor of a category is defined as the citations within a certain year to all articles in a category published in the previous 2 years (the numerator), divided by the number of articles published in that category in the previous 2 years (the denominator). The variables in the comparison group included the number of journals in each of the 11 clinical medicine categories in 2000 and 2009 and the aggregate impact factors for 2003 to 2009. The total number of journals in the JCR Science Edition in 2000 and 2009 was also recorded to compare the total number of journals according to category.

Measurements

The category aggregate impact factor and journal impact factor were adopted as representative of category and journal academic performance. The aggregate impact factor trend and journal impact factor trend were used to represent the evolution of their performance. The primary outcome measure in the comparison of categories of clinical medicine was the trend for the aggregate impact factor. The primary outcome measure for the comparison among the EM journals was the trend in the journal impact factor.

Category journal numbers partially reflected the quantity and quality of studies in that field. Category rank for journal numbers and journal number increase rates were measured to evaluate category performance in addition to the aggregate impact factor trend. The relationship between the 2000 EM journal impact factor and the impact factor trend of EM journals between 2000 and 2009 was measured to evaluate the evolution difference between journals with different initial impact factors.

Data Analysis

The EM journal numbers, countries, languages, and category journal numbers were analyzed using descriptive statistics. Spearman’s rank correlation coefficient (rs) was used to evaluate the correlation between category rank for journal numbers in 2000 and 2009 and increase rates in journal numbers. Bonferroni correction was adapted for multiple comparisons on correlations.

Linear regression was used to evaluate the trend in the category aggregate impact factors and the trend in impact factors for individual EM journals under the assumption of linear relationships between impact factors and time, and that normality was not strongly violated. The slope (β) of the linear regression was adopted as representative of trends in the aggregate impact factor and trends in the EM journal impact factor. The 95% confidence intervals (CIs) of β were calculated.

Pearson correlation coefficient (r) was used to evaluate the association between the 2000 impact factor and the impact factor trend from 2000 to 2009. A p-value of <0.05 was considered statistically significant. Data were analyzed with SAS version 9.2 (SAS Institute, Cary, NC).

Results

EM Journal Numbers, Countries, and Languages

Twelve journals were included in the EM category in the 2000 JCR, and four journals were excluded between 2001 and 2006 because of title changes, category changes, or termination of publication. Thus, 11 new journals were listed in the EM category after 2000, and the category journal number increased to 19 in 2009 (Table 1). In 2000, 12 journals were published in four countries: eight in the United States, two in England, one in Germany, and one in Ireland. In 2009, 19 journals were published in seven countries: nine in the United States; four in Germany; two in England; and one each in Ireland, Australia, Croatia, and Turkey. In 2000, all 12 journals were English-language journals. In 2009, the 19 journals were published in three languages; these included 16 in English, two in German, and one in Turkish.

Table 1. 
EM Journals Listed in 2000 and 2009 JCR
Title of Journal*LanguageCountryImpact Factor in 2000Impact Factor in 2009Note†
  1. JCR = Journal Citation Reports.

  2. *Titles are sorted according to 2009 impact factor. Four journals not included in the 2009 category of emergency medicine were listed at the bottom.

  3. †Year first included in JCR (since 2000). Reasons for not being included in the category: category = change in category; title = change in title; end = publication stopped.

Annals of Emergency MedicineEnglishUnited States2.1834.232 
ResuscitationEnglishIreland1.7602.712 
Academic Emergency MedicineEnglishUnited States1.4192.478 
InjuryEnglishEngland0.3632.383 
American Journal of Emergency MedicineEnglishUnited States1.0541.542 
Emergency Medicine JournalEnglishEngland1.4772002
Prehospital Emergency CareEnglishUnited States1.2972008
Journal of Emergency MedicineEnglishUnited States1.2652001
Emergency Medicine Clinics of North AmericaEnglishUnited States0.6350.964 
Pediatric Emergency CareEnglishUnited States0.4280.916 
Emergency Medicine AustralasiaEnglishAustralia0.9012009
European Journal of Emergency MedicineEnglishUnited States0.7332009
UnfallchirurgEnglishGermany0.4960.592 
Notfall and RettungsmedizinGermanGermany0.5732009
Journal of Emergency NursingEnglishUnited States0.3592007
Ulusal Travma ve Acil Cerrahi Dergisi-Turkish (Journal of Trauma and Emergency Surgery)TurkishTurkey0.2742009
European Journal of Trauma and Emergency SurgeryEnglishGermany0.2072009
Signa VitaeEnglishCroatia0.1752008
NotarztGermanGermany0.0752009
Critical Care ClinicsEnglishUnited States1.143Category
Journal of Burn Care and RehabilitationEnglishUnited States0.810Category and title
Journal of Accident and Emergency Medicine EnglishEngland0.668Title
New Horizons: The Science and Practice of Acute Medicine EnglishUnited States0.619End

Journal Number Ranks and Aggregate Impact Factor Trend for Clinical Medicine Categories

The number of categories of clinical medicine journals in 2000 and 2009, changes in journal numbers, the rates of increase, and their ranks are summarized in Table 2. EM journal numbers ranked last in 2000 and 2009, but the 58% rate of increase ranked first and was twice the rate of increase seen for the total number of journals in the JCR Science Edition. Spearman’s rank correlation coefficient (rs) was calculated to analyze the correlation between journal number ranks in 2000 (ranking 1), journal number ranks in 2009 (ranking 2), journal number increase ranks (ranking 3), and ranks of increase rate in journal number (ranking 4). This analysis revealed that the rank for category journal numbers in 2009 was highly positively correlated with that rank in 2000 (ranking 1 × ranking 2, rs = 0.97, p < 0.001). The rank for journal number increase rate was negatively associated with journal number rank in 2000 (ranking 1 × ranking 4, rs = −0.84, p < 0.001)

Table 2. 
Journal Numbers and Ranks of Categories of Clinical Medicine Listed in 2000 and 2009 JCR
Category*2000Rank 1†2009Rank 2†Increase‡Rank 3†Rate§Rank 4†
  1. JCR = Journal Citation Reports.

  2. *Categories are listed alphabetically.

  3. †Rank 1 = journal number ranks in 2000; Rank 2 = journal number ranks in 2009; Rank 3 = journal number increase in ranks; Rank 4 = ranks of increase rate in journal number

  4. ‡Journal number increase from 2000 to 2009.

  5. §Journal number increase rate (%) from 2000 to 2009.

Cardiac and cardiovascular system636984351563
Clinical neurology137116713022211
Critical care medicine15112211710475
Emergency medicine12121912710581
Gastroenterology and hepatology457657207447
Infectious diseases369579216581
Medicine, general and internal10531323274269
Pediatrics715945235328
Respiratory system29104310149484
Surgery136216623022210
Toxicology774776012012
Urology and nephrology438638207466
Total JCR5,6867,3471,66129

The aggregate EM impact factor in 2003 was 1.127, which placed EM last among 12 categories of clinical medicine. EM remained at the same rank in 2009, but showed a substantial increase in impact factor of 0.507, representing a 45% increase in citation rate, which was the highest rate of increase among the clinical medicine categories. A linear regression analysis of each category aggregate impact factor from 2003 to 2009 showed that every category had a positive trend (Table 3), indicating that the aggregate impact factor for every category increased significantly from 2003 to 2009, except the category “Medicine, general and internal” (β = 0.052, p = 0.356).

Table 3. 
Trend in the JCR Aggregate Category Impact Factors From 2003 to 2009
Category*Trendp-value95% CI
  1. JCR = Journal Citation Reports.

  2. *Categories are listed alphabetically.

Cardiac and cardiovascular system0.0750.0050.034 to 0.116
Clinical neurology0.107<0.0010.086 to 0.129
Critical care medicine0.1210.0030.064 to 0.177
Emergency medicine0.086<0.0010.056 to 0.116
Gastroenterology and hepatology0.0830.0400.006 to 0.159
Infectious diseases0.0870.0120.029 to 0.145
Medicine, general and internal0.0520.356−0.080 to 0.185
Pediatrics0.0750.0030.038 to 0.112
Respiratory system0.0850.0020.047 to 0.123
Surgery0.091<0.0010.063 to 0.118
Toxicology0.101<0.0010.076 to 0.126
Urology and nephrology0.071<0.0010.047 to 0.095

EM Journal Impact Factor Trend

Eight EM journals included in JCR since 2000 were enrolled for the impact factor trend analysis. The impact factor for all EM journals increased between 2000 and 2009 (Figure 1). A linear regression analysis of the journal impact factor from 2000 to 2009 revealed that every journal exhibited a positive trend, indicating that each journal impact factor increased significantly over this 10-year period (Table 4). The relationship between the 2000 EM journal impact factor and the impact factor trend is plotted in Figure 2. Injury appeared to be a significant outlier. The Pearson correlation coefficient (r) before and after the outlier was excluded was 0.49 and 0.94, respectively. This result indicated that the impact factor increased trend of EM journals between 2000 and 2009, excluding Injury, has a strong linear relationship with the 2000 impact factor.

Figure 1.

 Impact factors for emergency medicine journals from 2000 to 2009 and aggregate impact factors from 2003 to 2009. Note: Eight journals included in JCR since 2000 were used for the analysis. JCR = Journal Citation Reports.

Table 4. 
Trends in the EM Journal Impact Factors From 2000 to 2009
Title of Journal*Trendp-value95% CI
  1. JCR = Journal Citation Reports.

  2. *Journals included in JCR since 2000 were used in this analysis. Titles are sorted according to the 2000 impact factor.

Annals of Emergency Medicine0.223<0.0010.169 to 0.277
Resuscitation0.1390.0130.039 to 0.240
Academic Emergency Medicine0.125<0.0010.076 to 0.174
American Journal ofEmergency Medicine0.0290.435−0.053 to 0.111
Emergency Medicine Clinics of North America0.0390.0170.009 to 0.069
Unfallchirurg0.0210.0230.004 to 0.038
Pediatric Emergency Care0.0400.0020.019 to 0.062
Injury0.221<0.0010.158 to 0.283
Figure 2.

 Emergency medicine journals 2000 impact factor and impact factor trends from 2000 to 2009.

The aggregate impact factor is a reference point indicating average category performance. The number of EM journals increased from 10 in 2003 to 19 in 2009, and the impact factors of four journals were consistently higher than the aggregate impact factor each year (Figure 1). As the total journal number increased 90%, this indicates that the rate of journals with impact factors higher than the aggregate impact factor decreased from 40% (4/10) to 21% (4/19). The impact factors for Academic Emergency Medicine, Annals of Emergency Medicine, and Resuscitation have been steadily higher than the aggregate impact factor since 2003. The impact factor for the American Journal of Emergency Medicine was higher than the aggregate impact factor from 2003 to 2006. The impact factor for Injury has been higher than the aggregate impact factor since 2007.

Comparing the trend in the impact factors for individual EM journals and the aggregate category impact factor discloses which journals have greater than average rate for increase. Aggregate EM impact factors and the impact factors for 10 EM journals included in JCR since 2003 were used in a trend analysis. A linear regression analysis of journal impact factors from 2003 to 2009 revealed that the trend for the Annals of Emergency Medicine and Injury increased significantly more than the category aggregate impact factor (Annals 95% CI = 0.198 to 0.356, Injury 95% CI = 0.233 to 0.382, and aggregate 95% CI = 0.056 to 0.116). No significant differences were observed between the remaining journal impact factor trends and the aggregate category impact factor trend (Table 5).

Table 5. 
Trends in EM Journals and Aggregate Impact Factors From 2003 to 2009
Title of Journal*Trendp-value95% CI
  1. JCR = Journal Citation Reports.

  2. *Journals included in JCR since 2003 were used for the analysis. Titles are sorted according to the 2009 impact factor.

Annals of Emergency Medicine0.277<0.0010.198 to 0.356
Resuscitation0.2150.0440.008 to 0.423
Academic Emergency Medicine0.1150.0310.016 to 0.215
Injury0.308<0.0010.233 to 0.382
American Journal of Emergency Medicine−0.0690.261−0.210 to 0.071
Emergency Medicine Journal0.1430.0010.086 to 0.199
Journal of Emergency Medicine0.0810.0310.011 to 0.152
Emergency Medicine Clinics of North America0.0280.304−0.035 to 0.090
Pediatric Emergency Care0.0610.0110.021 to 0.101
Unfallchirurg0.0210.190−0.015 to 0.057
Aggregate impact factor0.086<0.0010.056 to 0.116

Discussion

Impact factor is an imperfect method to evaluate the academic performance of journals, but no better measure of scientific performance is currently available.9 The journal impact factor is still the most widely accepted quantitative and objective method for evaluating journals. Our analysis of the SCI database from a variety of viewpoints revealed the evolution of academic performance in EM journals.

In EM journal countries and languages comparison, the extended geographic distribution and the increase in language diversity indicate qualitative and quantitative advances in EM journals around the world. This trend was also evident in the diversity of submissions by country to one of the leading peer-reviewed EM journals.10

Before 1990, EM journals were diffusely classified by JCR into the categories of anesthesiology, surgery, or miscellaneous. In 1991, the “critical care” category was first used, and EM journals were reclassified into that category. In 1996, the critical care category was renamed “emergency medicine and critical care.” In 2000, this category was separated into “critical care medicine” and “emergency medicine.” The method for categorizing journals in the JCR is heuristic,3 but a new category generally means that that the quantity and quality of studies in that field are sufficient to warrant an independent category. In our study period, after EM being an independent category in JCR, EM is the smallest, youngest category, but maintained the highest journal number increase rate within the 12 categories of clinical medicine.

We adopted the aggregate category impact factor as representative of average academic performance in a category. The range of clinical medicine aggregate impact factors is wide (medicine, general and internal vs. emergency medicine: 3.971 vs. 1.127 in 2003, 4.108 vs. 1.634 in 2009). Thus, directly comparing the aggregate impact factors may mask the aggregate impact factor change of smaller, newer categories. The aggregate impact factor trend more precisely reflects the growth rate of the category. The 95% CIs for all category trends overlapped, which may have resulted from the limited time period of available aggregate impact factors in JCR, but may also indicate that EM journals’ academic performance as measured by the SCI impact factor was comparable to that for other categories.

The possible confounding effect of natural increase of impact factor (aggregate impact factor simultaneously) should be discussed. In an era of expanding literature, more citations were produced as more articles were published. The net effect to impact factor depends on the increasing rate of JCR articles and the average amount of references per article.11 If these two factors show an increasing trend, the impact factor will increase naturally. Every category and journal was influenced by this effect. In our study, we compared the trend between different category aggregate impact factors and the trend between journal impact factors during the same period of time instead of the trend of a single category or journal. This can offset the effect and disclose the impact factor trend in addition to natural increase.

The gap between EM journals’ impact factors has been widening in the past 10 years. The impact factor increased faster for high-impact-factor journals than for low-impact-factor journals. The ratio of EM journals with impact factors higher than the aggregate impact factor to the total number of EM journals decreased. Two journals maintained greater impact factor trends than the aggregate impact factor trends. Taken together, these results suggest that the most influential studies are published in a few EM journals, leading to uneven impact factor increments among EM journals. This phenomenon is similar to that found in previous studies regarding citations of EM journal articles.12 The uneven distribution of citations to articles has been mentioned in previous literature.3 Some articles account for the majority of citations, but most articles are seldom cited. At the level of journals, the polarization of EM journal impact factor increase has never been mentioned before. Whether this polarization is also occurring in other disciplines is still unknown. This phenomenon may be explained by a positive-feedback hypothesis. A high impact factor represents a journal with high visibility, prestige, and influence. Researchers with well-designed studies and significant results tend to submit their articles to journals with these characteristics.13 The editor’s acceptance criteria will be higher while reviewing high-quality submissions. Publishing high-quality articles leads to more citations, which increases the journal impact factor. A positive-feedback loop is formed.

The impact factor of Injury has had a substantial increase in the past 10 years, and its impact factor has increased out of proportion to its 2000 impact factor compared with other EM journals. Impact factor is influenced by the peer review process, editorial evaluation, and research field variables. The ability of a field to be cited by other fields is considered the most important field variable.14 Aside from peer review process and editorial evaluation, one difference between Injury and other EM journals in JCR is that Injury is classified into four categories: critical care medicine, emergency medicine, orthopedics, and surgery. The remaining seven EM journals in our study are classified into one or two categories. Although the scope of a journal is defined by its editorial board, JCR establish journal categories partly based on citation relatedness in recent years.3 Multiple categories classification in JCR might represent citations from broad disciplines, and its influence on journal impact factor needs further investigation.

Limitations

We used JCR impact factors as our main study data; therefore, limitations of the JCR database also apply to our study.11 The JCR database is an English-language resource and has an American publication bias; therefore, the contribution of non–English-language or non-American journals may be underestimated.3,11

Emergency medicine has been a JCR category since 2000, and the category aggregate impact factor has been available since 2003. Hence, relatively limited data were used to obtain the trends, so the 95% CIs overlapped. A longer study period in the future may reveal a more precise trend.

Conclusions

The quality and quantity of emergency medicine scientific studies have increased substantially in the past 10 years. An increasing number of international EM journals have appeared over the past 10 years. The trend in aggregate impact factor in EM journals is not dissimilar to that of other clinical medicine categories. Every EM journal exhibited a positive impact factor trend, with a widening gap between journals’ impact factors. Whether the polarization of journal impact factor is a common phenomenon in other disciplines remains unknown.

Ancillary