• Open Access

Performance Metrics for an Academic Society—The Annual Meeting


AM Feldman (arthur.feldman@jefferson.edu)

inline image

It is challenging to develop metrics that can be used to accurately define the success or direction of a new society. An organization’s annual meeting can sometimes provide a lens through which the sponsoring society or the discipline represented by the society can be viewed. Although using the somewhat subjective and anecdotal experiences at a meeting as a surrogate measure for evaluating the goals and objectives of the societies that participate has never been scientifically validated as a useful instrument, I think that a critical review of a meeting can provide relevant and useful information. To that end, I took a critical look at this past April’s combined meeting of the Society for Clinical and Translational Science, the Association for Clinical Research Training and the American Federation for Medical Research. This meeting was the third annual meeting for the Society for Clinical and Translational Science and the Association for Clinical Research Training and was the first time that they were joined by the American Federation for Medical Research.

The didactic sessions were well attended and drew a large percentage of the over 800 attendees and pointed out the great diversity of the audience. The sessions spanned the gamut of the clinical and translational sciences and in many aspects were focused on the Clinical and Translational Science Award (CTSA) scholars and their mentors. For example, sessions directed toward the scholars dealt with an array of topics including: writing a successful career development award, professionalism in translational science training, moving from K to R—competing successfully for your next grant, trajectories for success in clinical research—no one path fits all; foundation opportunities in basic, translational and clinical research training, and opportunities for research career development through the Veterans Administration. Sessions relevant for mentors were also diverse and relevant. They included: mentor training trial—a national CTSA study; creating networks for training in health and healthcare disparities research; what core competencies should every translational scientist have?; and efforts to promote translational science training for Ph.D.s. Finally, the didactic program also provided information for the established investigator, albeit less than that provided for the scholar or mentor: longevity’s impact on translational science, developmental origins of adult health and disease; obesity—what are the critical issues; databases for clinical translational research; best practices in community-engaged research; translational development of specimen and imaging biomarkers; new models for drug development, and the promise, the pitfalls and the policy implications of whole genome sequencing.

The abstract sessions provided an important window into the composition of the CTSA scholars. The presentations spanned the entire spectrum of the translational sciences. A total of 185 abstracts were submitted in time for publication. I graded each abstract as whether they represented T1, T2, T3, or T4 research. Abstracts were defined as T1 if they were carried out in preclinical animal models of disease or if they were performed largely at the laboratory bench. T2 studies were traditional “clinical research” that evaluated the ability of an intervention to change the course of disease or a biomarker to predict the outcome of a disease. In the past, I have steadfastly avoided using the T4 nomenclature; however, I learned an important lesson from reviewing the abstracts. T3 encompasses such a large compendium of topics that relegating all of the areas encompassed within the traditional T3 to a single designation obfuscates the ability to understand the complexity and breath of the clinical and translational sciences. Therefore, I assigned clinical epidemiology, outcomes research, and bioethics to the designation T3 while health policy research, evaluation of new healthcare delivery systems such as electronic health record, comparative effectiveness research, and studies of healthcare disparities were relegated to T4.

The largest number of abstracts, 40%, represented clinical research. Sixteen percent of the abstracts described T3 research, while 32% of the abstracts represented T4 research. Only 11% of the abstracts could be defined as T1 or basic research. The 185 abstracts were submitted by scholars from 25 different states, the District of Columbia and Puerto Rico. Almost half of the abstracts represented collaborative efforts of partnering institutions—and many of these partnerships crossed state lines. Analysis of the abstract content and the presenters raised interesting points. First, there was a broad representation of both topics and geographic regions. Nonetheless, with over 50 CTSA centers and thousands of scholars, it is somewhat disappointing that there were not more presentations. Were trainees reluctant to submit abstracts because of competing meetings, limited finances to support travel, the costs of preparing a poster, or other constraints on their time as scholars? Were mentors encouraging trainees to participate in the Society for Clinical and Translational Science meeting? Do CTSA trainees prefer to submit their work to subspecialty societies rather than to national meetings having a broader audience? Were the trainees who presented abstracts predominantly M.D.s, Ph.D.s, or a mixture of both. The societies should consider a simple survey to try to shed light on some of these important questions.

I was also surprised by the relative paucity of basic science abstracts when compared with abstracts relating the results of clinical trials, epidemiology, or health delivery system initiatives. Some health policy analysts have raised concerns that a focus on translational research could direct research dollars away from “basic research” and toward “applied research.” A recent article in the Wall Street Journal paraphrased a concern raised by Roy Vagelos, the former head of Merck, that “shifting resources away from NIH’s key function of supporting [basic science] research shrinks the funding possibilities for PhDs who want to move along the academic job pipeline. Letting them languish would reduce the discoveries that form the basis for applied research.” The proposed creation of the National Center for Advancing Translational Sciences (NCATS) and its focus on creating new drugs through government investment and partnerships with the pharmaceutical industry has been a lightening rod for critics of translational research. Therefore, it would be important for the societies to create a registry across all of the CTSAs to help understand whether there is an equal distribution of trainees and mentors across all four areas of translational research (T1, T2, T3, and T4) or whether the critics are correct and the CTSAs have created an imbalance that favors health policy, epidemiology, clinical research, and outcomes research over the T1 translational research. If the representation at this year’s abstract session accurately reflects the overall balance of clinical research versus basic research across the CTSAs, it would be important to try and rebalance both the Society and the CTSAs to insure that all disciplines are equally represented and in particular that basic science research that forms the underpinning of clinical research continues to be nurtured and rewarded.

A critically important part of the Clinical and Translational Research and Education Meeting was that it provided both trainees and investigators the opportunity to hear first hand from some of the national authorities who are guiding the future of scientific investigation and patient care. For example, the first Plenary Session gave attendees the opportunity to hear how translational research fit into the key elements of our government’s plans for the future of healthcare delivery. They heard from Dr. Margaret Hamburg, Commissioner of the Food and Drug Administration; Dr. Francis Collins, Director of the National Institutes of Health; and Dr. Donald Berwick, Administrator of the Centers for Medicare and Medicaid Services—an all-star lineup of national healthcare leaders. In a later session, they had the opportunity to hear from administrative leaders of the National Institutes of Health and to learn the future plans for translational research, and in individual didactic sessions, attendees had the opportunity to learn from a large cadre of highly experienced translational investigators. This provided invaluable insight for both trainees as well as mentors.

For me, the most exciting part of the meeting was the opportunity to speak with the poster presenters. They were enthusiastic, articulate, focused, determined, and stimulating. They knew how their research fit into the bigger picture of science, technology, and healthcare, and they seemed fully capable of taking the next steps toward an independent research career. Their research spanned the full gamut of the translational sciences and it was clear that they understood the challenges before them. They seemed highly prepared to answer the tough questions about their research and they were quick to credit the outstanding mentoring they had received at their CTSA center. My time at the poster sessions reinforced for me the notion that the CTSAs are fulfilling their mission of training the next generation of scientists. Yet, my conversation with many of the trainees focused on their fears that the budgetary constraints placed on the NIH may impede their career development.

Taken together, my observations from the meeting and from my review of the abstracts strongly suggested that the Societies that were represented at the meeting are fulfilling their objectives and that the meeting is an outstanding forum for communication and education. Further evaluation is certainly warranted to understand whether there is a need to rebalance the societies’ efforts to insure that basic research continues to be an important and critical part of the clinical and translational sciences. I also came away from the meeting recognizing the overwhelming need for those of us who are mentors to continue to push our institutions, our federal and state leaders, and the public about the critical need to continue to support the creation of the next generation of investigators—both basic and clinical—through continued funding for Clinical and Translational Science Centers.