• Open Access

The Quantitative Evaluation of the Clinical and Translational Science Awards (CTSA) Program Based on Science Mapping and Scientometric Analysis

Authors


Abstract

The Clinical and Translational Science Awards (CTSA) program is one of the most important initiatives in translational medical funding. The quantitative evaluation of the efficiency and performance of the CTSA program has a significant referential meaning for the decision making of global translational medical funding. Using science mapping and scientometric analytic tools, this study quantitatively analyzed the scientific articles funded by the CTSA program. The results of the study showed that the quantitative productivities of the CTSA program had a stable increase since 2008. In addition, the emerging trends of the research funded by the CTSA program covered clinical and basic medical research fields. The academic benefits from the CTSA program were assisting its members to build a robust academic home for the Clinical and Translational Science and to attract other financial support. This study provided a quantitative evaluation of the CTSA program based on science mapping and scientometric analysis. Further research is required to compare and optimize other quantitative methods and to integrate various research results.

Introduction

Translational medicine is an important area of the biomedicine, which facilitates the development of the medical research greatly.[1, 2] In 2006, the National Institutes of Health (NIH) launched the Clinical and Translational Science Awards (CTSA) program, and since then, the program has become one of the most important initiatives in translational medical funding.[3-5] The goal of the CTSA program is to speed two-way translational research between bench and bedside, and to remove the hurdles that block the discovery and development of the new treatments.[6-8] In recent years, the funding scope of the CTSA program has grown, and the members of the CTSA consortium have increased from 12 institutes in 2006 to approximately 60 institutes in 2012.[9] Nowadays, the evaluation of the efficiency and performance of the CTSA program has been considered as a valuable indicator in the decision making of US and global translational medical funding.[10]

Several studies regarding the evaluation of the CTSA program have been done in recent years, including developing a multidisciplinary model to evaluate the roles of the CTSA program in supporting its members,[11] assessing the program implementation efficiency of the CTSA centers,[12] and surveying the capacity of the CTSA program in overcoming major translational barriers.[13] However, most of these studies employed qualitative methods, lacking of the quantitative evaluation. Utilizing science mapping and scientometric analytic tools, this study tried to answer the following questions:

  1. What are the quantitative productivities of the CTSA program?
  2. What are the emerging trends of the research funded by the CTSA program?
  3. What are the academic benefits from the CTSA program?

Science mapping and scientometric analysis are important methods in the health science informatics research. They can be used for quantitatively analyzing the intellectual connections and relationships of the scientific articles, displaying visual landscapes and dynamic trends of the knowledge structures in a specific research field.[14, 15] In this study, two open-source software applications, CiteSpace[16] and BibExcel,[17] were used for science mapping and scientometric analysis in order to evaluate the efficiency and performance of the CTSA program. CiteSpace and BibExcel are freely accessible and have been used widely in science mapping and scientometric analytic research.[18-20] CiteSpace was developed by Chaomei Chen at Drexel University (the United States), and BibExcel was developed by OllePersson at Umeå University (Sweden).

Methods

Retrieval strategy

The data source used for this study was the Web of Science database from Thomson Reuters, formerly known as ISI, one of the most authoritative and multidisciplinary platforms. The retrieval strategy was set based on the funding agency search for “Clinical and Translational Science Awards” OR “Clinical and Translational Science Award” OR “CTSA.” The retrieval time span began in 2006 when the CTSA program was launched and extended through 2012, since the data entry for 2013 had not been completed. The document types were limited to research articles and review articles.

Research flow

The scientific articles funded by the CTSA program were collected according to the above retrieval strategy. BibExcel was used for analyzing the publication efforts, as well as ranking the active institutes and coinvesting agencies of the CTSA program based on information extraction. In this study, publication efforts consisted of the number of scientific articles, the total citations, and the average citation rate for each article, which represented the quantitative productivities of the CTSA program. The active institutes and coinvesting agencies represented the academic benefits from the CTSA program. Furthermore, with CiteSpace, the hot topics and new themes were mapped based on coword cluster and burst detection, which revealed the emerging trends of the research funded by the CTSA program (Figure 1).

Figure 1.

The flow chart of the quantitative evaluation of the CTSA program.

Information extraction

Information extraction is a component of natural language processing (NLP) that focuses on finding the relationships between particular entities of scientific articles.[21, 22] These entities include authors, affiliations, citations, keywords, and so forth. In this study, BibExcel was used for extracting two specific entities, “reprint author address,” and “funding agency.”

Coword cluster and burst detection

Coword cluster is based on the hypothesis that a specific field consists of series of scientific articles with marked keywords. The frequency of the co-occurrence of these keywords reflects the similarities between different articles.[23, 24] Scientific articles with similarities could be clustered into a plot, and different plots could form a coword clustering graph, representing the hot topics relevant to the specific research field.[25]

A burst in scientific articles usually refers to a sudden increase in keyword frequency. Burst detection aims to reveal the unexpected change of the data streams in a limited time period,[26, 27] which may be used as a valid indicator, suggesting new themes of the specific research fields.[28]

Coword cluster and burst detection have been used in scientific evaluation and technology foresight,[29, 30] facilitating the decision making of the specific research funding. In this study, the coword cluster and burst detection of keywords were implemented by using CiteSpace.

Results

Publication efforts

Based on the retrieval strategy above, 1,428 scientific articles funded by the CTSA program were identified (research articles: 1,345, 94.2%; review articles: 83, 5.8%). While the number of scientific articles had a stable increase since 2008 (Figure 2A), there was no published article in 2006 and 2007. The probable reason for the absence of publication during this period would be that the researchers needed a period of time to obtain experimental data and publish scientific articles.

Figure 2.

The annual evolution of the publication efforts of the CTSA program: (A) The number of scientific articles. (B) The total citations. (C) The average citation rate for each article.

Regarding the citations and citation rate, the total citations were 7,658, and the average citation rate for each article was 5.36. While the total citations increased sharply from 2009 to 2012 (Figure 2B), the average citation rate for each article showed a slight descending trend from 2011 to 2012 (Figure 2C). However, this phenomenon should not be considered as a loss of scientific interest or influence of the CTSA program. It was likely due to 2012's temporal proximity to this study.

Hot topics and new themes

An overview of the hot topics and new themes of the research funded by the CTSA program was formed based on coword cluster and burst detection by using CiteSpace (Figure 3, Table 1). In this visualization, there were six major clusters labeled in blue text representing the hot topics. These hot topics were mainly clustered by log-likelihood ratio test method (LLR). Moreover, the purple rings and red nodes were mapped in line with high betweenness centrality and sudden increase of the keywords respectively, suggesting new themes.

Table 1. The six major clusters of the research funded by the CTSA program derived by using CiteSpace
Cluster IDSizeLabel (TFIDF)Label (LLR)Label (MI)
1105Insulin; atherosclerosis; insulin resistanceEpidemiology; population; cardiovascular diseaseChronic illness
791Pharmacokinetics; chronic renal failure; hemodialysis patientsPharmacokinetics; in vitro; glomerulonephritis;Lupus nephritis
676Acute renal failure; proximal tubule; injury; albuminuriaAcute renal failure; proximal tubule; albuminuria;Frequency
057Carcinoma; mutation; C-reactive protein; tumorigenesisBreast cancer; tumor necrosis factor alphaCell death
553Virulence; outbreak; ampicillin resistantStaphylococcus aureus; virulence; ampicillin resistant
339Alzheimer's disease; dementiaAlzheimer's disease; beta-structure; microtubule bindingPredictors
Figure 3.

The visualization of hot topics and new themes of the research funded by the CTSA program based on coword cluster and burst detection.

The hot topics and new themes of the research funded by the CTSA program focused on the following areas: (1) epidemiological research, such as the virulence of Staphylococcus aureus and ampicillin-resistant; (2) chronic illness therapies, including insulin-resistance and cardiovascular disease therapies; (3) disease biomarker research, including Alzheimer's disease (beta-structure and microtubule binding), breast cancer (tumor-necrosis-factor-alpha), and acute renal failure (albuminuria and proximal tubule); (4) basic medical research, especially in vitro pharmacokinetic of glomerulonephritis.

Active institutes and coinvesting agencies

By using BibExcel, two specific entities, “reprint author address” and “funding agency,” were extracted and ranked statistically from 1,428 scientific articles.

When submitting articles, a research institute or funding agency may use different descriptions, such as abbreviating an institute's name in different ways, writing a full name with or without his/her middle name or initial. In this study, these entities were manually reviewed, screened, and represented by a single entity. For example, “National Heart Lung and Blood Institute;” “National Heart, Lung, and Blood Institute, National Institutes of Health;” and “NHLBI” were all represented by “NHLBI.” Then, the top 10 active institutes and the top 20 coinvesting agencies were sorted based on the number of scientific articles (Table 2, Figure 4).

Table 2. The top 10 active institutes funded by the CTSA program based on the reprint author address of scientific articles
RankInstitutesThe number of scientific articlesOverall contributionFunding year
1Mayo Clinic1299.03%2006
2Vanderbilt University926.44%2007
3Yale University694.83%2006
4University of Washington604.20%2007
5University of Pittsburgh443.08%2006
6University of Chicago433.01%2007
6University of Texas Health Science Center at Houston433.01%2006
8Washington University372.59%2007
9Johns Hopkins University362.52%2007
10Duke University352.45%2006
Figure 4.

The top 20 coinvesting agencies of the CTSA program based on the funding agency of scientific articles.

According to the study, Mayo Clinic was the most active institute funded by the CTSA program. Mayo Clinic became a member of the CTSA consortium in 2006, and by the year 2012, the institute had published 129 papers (9.03%). Next in the rankings were Vanderbilt University (92, 6.44%) and Yale University (69, 4.83%). It is worth noting that all of the top 10 institutes were awarded the CTSA funding at the beginning years, 2006 or 2007, of the CTSA program. This could be one of the reasons why these institutions were more productive.

The NIH institutes, such as NCI, NHLBI, and NIDDK, showed positive impacts on the research funded by the CTSA program, publishing 105, 83, and 73 papers (7.35%, 5.81%, and 5.11%), respectively. Moreover, the members of the CTSA consortium, such as Vanderbilt University and Mayo Clinic, provided supplementary matching grants, publishing 161 and 105 papers (11.3% and 7.35%), respectively. Finally, the data showed that the CEOs of pharmaceutical companies and the leaders of academic and military organizations put substantial investments into the research funded by the CTSA program (Figure 4).

Discussion

In this study, science mapping and scientometric analysis were integrated into the evaluation of the efficiency and performance of the CTSA program. According to the evaluation research, the overall quantitative productivities of the CTSA program had a stable increase since 2008, which was consistent with the expansion of the scope of this funding. The emerging trends of the research funded by the CTSA program covered clinical and basic medical research, especially in the fields of cardiovascular disease, nervous system diseases, cancer, and kidney disease. One of the academic benefits from the CTSA program was assisting its members to build a transformative, novel, and integrative academic home for the Clinical and Translational Science.[31] Vanderbilt University and Mayo Clinic were the exemplary cases. The other benefit was helping the institutes funded by the CTSA program attract other financial support from the NIH institutes, pharmaceutical companies, academic societies, and Department of Defense, which sped the translational efficiency from basic science discoveries to clinical practice.

Other researchers evaluated the importance of the CTSA program in the creation of a resource, program, or service, and found that the CTSA program improve the activities of diverse research funded by the NIH institutes.[32] This study, quantitatively analyzing and evaluating the CTSA program from the point of view of scientific articles, brought new and interesting results. For example, the pharmaceutical companies, academic, and military organizations were interested in the related research of the CTSA program besides the NIH institutes. This indicated that the CTSA program had significant and practical influence in facilitating two-way translational progress between bench and bedside.

The main innovations of this study were to evaluate the efficiency and performance of the CTSA program by using science mapping and scientometric analytic methods and to reveal the landscapes of the quantitative productivities, the emerging trends, and the academic benefits in this field. The quantitative research results could be used as valuable supplements to the qualitative research, such as online survey and expert consultation. In addition, this study also proposed a new insight with respect to the decision making of global translational medical funding that was using science mapping and scientometric analytic tools to quantitatively evaluate the related funding based on scientific articles analysis.

While this study explored, evaluated, and presented the efficiency and performance of the CTSA program from a different angle, one limitation in this study could be improved in the future: although there were many science mapping and scientometric analytic approaches available, only two research tools, CiteSpace and BibExcel, were used to carry out the evaluation research in this study. Different quantitative methods or tools might yield different results due to variance in computer algorithms and mathematical calculations. Therefore, further research is required to compare and optimize other quantitative approaches and to integrate various research results.

Conclusion

In this study, the efficiency and performance of the CTSA program were analyzed and evaluated by using quantitative methods. The research showed that the CTSA program had positive impacts on translational medical research, including the stable growth of scientific article productivities, the accelerated integration of clinical and basic medical research, and the robust support to its members' developments. In addition, this study provided new insight into the decision making of global translational medical funding that was integrating the science mapping and scientometric analytic methods into the decision making.

Acknowledgments

Special thanks to Dr. Harold Lehmann, Director of Research and Training, Division of Health Sciences Informatics, and Dongming Zhang, Director of Advanced Technologies & Information Services, School of Medicine, Johns Hopkins University, for their constructive suggestions and comments.

Sources of Funding

This research is supported by grant 2010SF006 from the Ministry of Science and Technology of the People's Republic of China.

Ancillary