Dr. Furst has received consultant fees, speaking fees, and/or honoraria (less than $10,000 each) from Abbott, Actelion, Amgen, Bristol-Myers Squibb, Janssen, Gilead, GlaxoSmithKline, Novartis, Pfizer, UCB, Roche, and Genentech.
The American College of Rheumatology (ACR) and the Association of Rheumatology Health Professionals (ARHP) Annual Scientific Meeting is an important forum for early dissemination of novel ideas. However, unlike published studies in peer-reviewed journals, reviewers select abstracts based solely on a general summary of the research. Analyses of the scientific impact and the publication record of the ACR/ARHP Annual Meeting have not been previously described. This study characterizes publication trends and outcomes associated with abstracts presented at the ACR/ARHP Annual Scientific Meeting.
We identified all abstracts accepted for oral or poster presentation at the 2006 ACR/ARHP Annual Scientific Meeting. Using a defined search algorithm, we conducted a manual PubMed search for each accepted abstract, which was repeated by a custom computerized search, and analyzed the resulting journal title, impact factor, and time to publication.
A total of 2,149 abstracts were analyzed. The overall publication ratio was 59.1%. The mean ± SD time from abstract presentation to publication was 18.2 ± 15.2 months with a mean ± SD impact factor of 5.61 ± 4.20. Overall, studies presented in oral format were significantly more likely to be published than poster presentations (P < 0.0001). The average time to publication was significantly shorter for basic science studies than clinical research studies (P < 0.0001). The average journal impact factor of published studies presented in oral format was significantly higher than those presented as posters (P < 0.0001).
These results reflect high research productivity with a publication ratio of approximately 60% for abstracts presented at the 2006 Annual Scientific Meeting.
Scientific presentations at professional organization meetings are recognized as a primary method to provide up-to-date and novel research findings to both the medical and scientific communities. The advantage of utilizing these forums to deliver information is the opportunity for medical personnel and scientists to be exposed to original research methods, novel clinical trials, and innovative basic science research products prior to publication in peer-reviewed journals. The mean time to publication following presentation at a scientific meeting has been estimated to be approximately 18–20 months (1), suggesting that presentation of a study at a scientific conference may reflect a more rapid distribution of current research information. However, this forum utilizes gross classifications based on subject matter in order to sort abstracts and select those for presentation; much of the presented data may be preliminary, and the processes that determine the format in which the study is presented may be arbitrary. The extent to which these factors properly reflect the ultimate publication of these studies is unknown. Furthermore, there is little knowledge about differences in publication ratios or times to publication among different medical specialties (2–4).
The American College of Rheumatology (ACR) and the Association of Rheumatology Health Professionals (ARHP) Annual Scientific Meeting serves as an important forum for introducing new clinical and basic science research in the fields of rheumatology and immunology and includes a reasonable representation of professionals from around the world. Given the expected dissemination of quality information presented at the Annual Scientific Meeting by meeting attendees, we conducted a study to explore the outcome of abstracts presented at the ACR/ARHP Annual Scientific Meeting in 2006 and compared these results to annual conferences of other medical societies.
Significance & Innovations
This study utilizes a unique methodology that provides novel and reliable data describing the productivity and the quality of abstracts presented at the American College of Rheumatology/Association of Rheumatology Health Professionals (ACR/ARHP) Annual Scientific Meeting and compares these results with the caliber of presentations from scientific meetings of other biomedical societies.
The information presented at the ACR/ARHP Annual Scientific Meeting can have a significant impact on the development of future research and the implementation of novel medical practices by its attendees; therefore, this study characterizes and critically assesses the quality and the dissemination of this novel information.
MATERIALS AND METHODS
Abstract eligibility criteria.
All abstracts (n = 2,156) that were accepted for presentation at the 2006 ACR/ARHP Annual Scientific Meeting were identified from the corresponding conference proceedings (5) and the ACR Annual Scientific Meeting web site (6). Only abstracts that presented scientific research studies were included in the assessment; as such, abstracts formatted as individual case reports were excluded from analysis. Abstracts were categorized by presentation format (poster or oral presentation), research scope (basic science or clinical research), and field of interest (common or rare disease).
Data extraction strategies.
To achieve maximum accuracy, a unique search method comprised of 3 distinct phases was developed in collaboration with a data mining expert (DSP). In the first phase, 3 investigators (OLH, RE, and CP) conducted a manual search of the online PubMed database for each accepted abstract using a defined search algorithm (Figure 1). In order to maximize the sensitivity of the search, the second phase consisted of a systematic electronic search using a custom computer program that analyzed abstracts that initially were not found to be published. The program automatically generated a list of PubMed publications for each of these abstracts by searching for the last name of the last author and a general topic keyword (e.g., Smith [au] AND lupus). Each list of potential publications was reviewed by 2 independent authors (GA and JMPW) in order to identify subsequent matches between the ACR/ARHP Annual Scientific Meeting abstract and corresponding publications that were previously overlooked.
Finally, in order to assure accurate matching between the Annual Scientific Meeting abstracts and the publications that were identified through PubMed, an additional custom computer program was developed that calculated a Jaccard similarity coefficient (JSC; a statistical measurement used for comparing the similarity and diversity of sample sets) for each Annual Scientific Meeting/PubMed pair. Pairs receiving low JSC scores (i.e., JSC <0.05) were identified and the corresponding matches were reassessed. In all data extraction phases, abstracts were deemed to be published if the resulting publication shared a common hypothesis and study design, and the first or senior author was included.
Publication classification criteria.
Publication was defined as those manuscripts published as full-length articles or brief reports in peer-reviewed journals; reviews or case studies were not categorized as published. Journal title, journal impact factor (IF), author's affiliation, and time to publication (in months) were analyzed for each published study.
Identification of subsequent publication was limited to the period between November 2005 and November 2011, establishing a window of approximately 6 years to allow for adequate time for publication. The lower date limit was set to 12 months prior to the conference, due to embargo policies that prohibit abstract contents to be released until they are made available by the ACR approximately 2 months before the Annual Scientific Meeting. Exceptions to this policy include those abstracts published in a peer-reviewed journal after the ACR abstract submission deadline in June. November 2006 was set as month 0, with studies published prior to this month receiving negative time to publication values and studies published after this month receiving positive values.
Chi-square test or Fisher's exact test was used to compare appropriate subsets based on explanatory variables, including format of presentation, research scope, and field of interest. Student's t-test or Wilcoxon's rank sum test was used to assess the number of months to publication and IF of the journal between appropriate subsets. P values less than 0.05 were considered statistically significant. All values are expressed as the mean ± SD unless noted otherwise.
Overall, 2,149 abstracts were analyzed (this included all abstracts presented at the 2006 ACR/ARHP Annual Scientific Meeting, excluding 7 abstracts that described individual case reports). During the initial manual search of the PubMed online database for the subsequent publication of presented abstracts, 893 abstracts were classified as reaching publication as a full-length manuscript or brief report. In phase 2, an electronic search using a custom computer program identified 4,802 potential publication matches for the remaining 1,264 abstracts that were not previously noted as being published; these results were based on matching authors, word overlap, and word pair overlap. Following assessment of the program output, 376 additional abstracts were identified as reaching publication. In total, 1,269 abstracts were classified as reaching publication as a full-length manuscript or brief report. In order to ensure accuracy of the data, abstract/publication pairs with JSC scores <0.05 for word pair overlap were assessed (n = 240; range 0.018–0.050); all of the identified pairs were acknowledged as correct and were included in the analysis.
Abstracts were subdivided into 2 main groups based on the scope of research: clinical or basic science (Figure 2A). Clinical abstracts, comprising approximately 63% of the total presentations, were defined as abstracts describing prospective, retrospective, or case–control studies. Basic science abstracts included all the abstracts that involved primarily laboratory-based experiments (including translational experiments). Each category was further subdivided into abstracts that were presented at the meeting as poster sessions versus oral presentations.
The overall publication ratio for all abstracts was 59.1%. There was no significant difference in publication ratios between clinical and basic science studies (Figure 2B). Oral presentations had a significantly higher publication ratio than poster sessions (68.5% versus 57.1%; P < 0.0001). Specifically, the percentages of abstracts resulting in publication were not significantly different when comparing basic science poster presentations and basic science oral presentations (P = 0.26); however, clinical poster presentations were significantly less likely to be published than clinical oral presentations (56.2% versus 71.2%; P < 0.0001) (Figure 2B). The mean journal IF was slightly, albeit significantly, higher for basic science publications than clinical publications (5.8 versus 5.5; P = 0.026) (Figure 2C). Furthermore, clinical oral presentations had significantly higher mean IF than clinical poster presentations (7.11 versus 5.08; P < 0.0001), with similar results observed when comparing basic science oral presentations and poster presentations (7.46 versus 5.43; P < 0.0001) (Figure 2C). In both poster and oral presentation formats, clinical abstracts had significantly longer mean time to publication than basic science abstracts (19.5 versus 16.3 months, median 17.0 months [interquartile range (IQR) 9–28 months] versus 13.0 months [IQR 5–26 months]; P < 0.0001) (Figure 2D). As seen in Figure 2D, the time to publication of abstracts identified as published between November 2005 and November 2011 ranged from 10 months prior to the meeting to 61 months (more than 5 years) after the meeting. Approximately 7.1% of subsequent manuscripts reached publication prior to the 2006 ACR/ARHP Annual Scientific Meeting, with approximately 50% of the manuscripts published within 15 months following the Annual Scientific Meeting, and 87% of all manuscripts published within 3 years postpresentation at the Annual Scientific Meeting.
Fourteen abstracts were presented at the plenary session in the 2006 ACR/ARHP Annual Scientific Meeting, all of which resulted in subsequent publication. The mean ± SD time to publication was 15.8 ± 18.28 months (median 16.0 months, IQR 1.75–29.5 months) and ranged from 9 months prior to the meeting up to 51 months after the meeting. The mean ± SD IF (14.75 ± 16.24) of plenary session publications was significantly higher than other presentation categories (P < 0.001) and ranged from 3.4 to 50.02.
Abstracts presented at the 2006 ACR/ARHP Annual Scientific Meeting that resulted in publication were published in 199 unique journals. Table 1 shows the top 10 journals that were responsible for 68.4% of all published Annual Scientific Meeting abstracts. All of these journals were categorized as rheumatologic journals, and the top 4 journals are credited with publishing more than half of the resultant manuscripts. Arthritis & Rheumatism was the most common journal, which published about 25% of the published Annual Scientific Meeting abstracts and also had the highest IF (7.469) among the top 10 journals. Annals of the Rheumatic Diseases, Journal of Rheumatology, and Rheumatology (Oxford) followed with 12.1%, 10.3%, and 4.9% of published abstracts, respectively.
Table 1. Top 10 journals containing published abstracts from the 2006 ACR/ARHP Annual Scientific Meeting*
Average IF (2006)
Abstracts published, no.
Total published abstracts, %
ACR/ARHP = American College of Rheumatology/Association of Rheumatology Health Professionals; IF = impact factor.
Arthritis & Rheumatism
Annals of the Rheumatic Diseases
Journal of Rheumatology
Journal of Immunology
Arthritis Research and Therapy
Clinical and Experimental Rheumatology
Osteoarthritis and Cartilage
Following classification of clinical abstracts presented at the 2006 ACR/ARHP Annual Scientific Meeting as investigations into either common or rare diseases (defined according to the Rare Disease Act of 2002 as any disease or condition that affects less than 200,000 persons in the US), approximately 80% of the clinical abstracts were found to have presented research on common diseases with a 58.5% publication ratio (Table 2). The remaining abstracts presented research focused on rare diseases. No significant differences were observed between the groups in terms of publication ratio, time to publication, and IF (P = 0.78, 0.83, and 0.34, respectively).
Table 2. Comparison of publication ratios for common versus rare diseases
Publication ratio, %
Rare disease was defined according to the Rare Disease Act of 2002 as any disease or condition that affects less than 200,000 persons in the US.
Publication ratios for each country that submitted abstracts to the 2006 ACR/ARHP Annual Scientific Meeting were assessed; the US had the highest number of abstracts that reached publication (474 abstracts, representing 37.3% of total published abstracts). Japan, The Netherlands, Canada, and France rounded out the top 5 countries with the highest number of abstracts resulting in publication, contributing an additional 26.3% of all the subsequent publications (Table 3). Productivity rates (defined as publications per 1 million citizens) were highest among European countries, especially The Netherlands and Sweden, which had productivity rates of 5.02 and 4.19, respectively (Table 3).
Table 3. Top 15 countries based on presentation and publication productivity
No. of abstracts resulting in publication (% of total published abstracts)
Productivity rate, publications per 1 million citizens
Mean impact factor
A subanalysis of 547 randomly selected abstracts (25.4% of all abstracts) was conducted to assess statistical analysis included in presented data and study sponsorship. Approximately 60% (327 of 547) of the abstracts in the subcohort included results from their statistical analysis. Of these, 261 (80%) presented statistically significant results as demonstrated with P values or confidence intervals, 22 (7%) presented results that were not statistically significant, and 44 (14%) presented statistical analysis without indication of significance level. The publication ratios were similar for abstracts with or without documented statistical analysis and whether or not the results were statistically significant (range 55–63%). The subanalysis revealed 113 abstracts (21%) whose authors disclosed either receiving grant support or being a full-time employee of a commercial institution. The disclosures for 48 (42.5%) of these abstracts did not appear to be directly related to the content of the presented study, with 26 abstracts (54.2%) reaching publication. Conversely, 65 abstracts (57.5%) presented studies that were directly related to the reported disclosures of the first or last author and were considered “sponsored studies.” Compared to nonsponsored studies, the publication ratio of sponsored studies was slightly higher, albeit not significantly higher, at 61.5%. Of note, only 5 abstracts had sponsorship disclosures directly noted in the text of the abstract.
The publication ratio from the 2006 ACR/ARHP Annual Scientific Meeting was comparable to other scientific meetings from recent years (2001–2007) (7–17) (Table 4). Publication ratios from other meetings varied remarkably between 20.5% and 68.9%. The publication ratio of the 2006 ACR/ARHP Annual Scientific Meeting was lower than only 1 conference (Society of University Surgeons [SUS], 2002–2004) of the scientific conferences with available publication data.
Table 4. Comparison of publication ratio with data from other scientific societies*
Publication outcomes (including overall publication ratio, length of time to publication, and IF of the journal in which the study reached publication) for abstracts presented at the 2006 ACR/ARHP Annual Scientific Meeting were variable, based on format of presentation and scope of research. The scope of research presented in the meeting abstract (either clinical or basic science) appeared to have the greatest impact on length of time to publication, with clinical studies requiring more time before reaching publication. However, the format of presentation (i.e., poster format or oral presentation) correlated more with overall publication ratios and journal IF; oral presentations were more likely to have better publication ratios and resulted in publication in journals with higher IFs. Furthermore, JSC scores for word pair overlap between presented abstracts and the abstracts of resultant manuscripts were all greater than 0.01 (range 0.018–0.934), indicating that a cutoff of JSC scores for word pair overlap at 0.01 may be useful in identifying potential matches when implementing a computerized abstract search such as the one conducted in this study.
To the best of our knowledge, this is the first time that an extensive analysis has been performed on the outcome of abstracts that are presented in the ACR/ARHP Annual Scientific Meeting. We believe it is useful to estimate the quality and effectiveness of this method of disseminating information because 1) thousands of physician and researchers in the field of rheumatology are annually exposed to the abstracts of the ACR/ARHP Annual Scientific Meeting, which represent in part the most innovative and novel advances in the field, and 2) these studies may directly influence the manner in which rheumatologists treat their patients and may inspire researchers to find new directions for their studies. Thus, it is important to know if these data can be considered credible, as defined by peer review and subsequent publication.
We selected 1 year before and 5 years after the 2006 ACR/ARHP Annual Scientific Meeting as an appropriate timeframe to identify subsequent publications. Conferences of other medical societies reported that few abstracts were published beyond 4 years after their initial scientific meeting presentations (1, 8–10, 13, 16, 17). Therefore, a period spanning 5 years following the meeting would make it unlikely that published works from the presentations at the 2006 ACR/ARHP Annual Scientific Meeting were not included.
This study reports a publication ratio of almost 60% from the 2006 Annual Scientific Meeting. Although this is an impressive finding compared to other societies (Table 4), approximately 40% of the abstracts were not published within 5 years following their presentation. In other words, these unpublished abstracts never underwent a peer review process to validate the quality of their findings. This fact is somewhat concerning and suggests that conference attendees need to interpret the data presented at the Annual Scientific Meeting somewhat cautiously.
This study found no difference in the publication ratio between clinical and basic science studies. However, clinical studies were published on average more than 3 months later than basic science studies. The difference in time to publication may be attributed, at least in part, to either a delay in the publication of prospective studies that are presented as interim analyses, requests for clarification of results, or for secondary data following initial submission for publication. These delays may not be as characteristic of basic science studies, reflecting a shorter time to publication. Similarly, as IF is determined by the frequency of citations, differences in IF between basic science and clinical research journals can partially be attributed to the fact that clinical studies tend to cite both basic science and clinical studies, whereas basic science studies tend to cite only other basic science studies, resulting in higher IFs for basic science journals (18).
Abstracts that are thought to present more important or generalizable findings are usually selected by the conference reviewer to be presented as oral presentations, suggesting that abstracts presented in an oral format would have a higher publication ratio. Although clinical oral presentations had a significantly higher publication ratio than clinical poster presentations, no significant difference was observed between basic science oral presentations and poster sessions. The method for sorting abstracts used at the ACR/ARHP Annual Scientific Meeting by the conference reviewers seemed appropriate as oral presentations in both research scopes yielded significantly higher IFs than poster sessions. As expected, all of the abstracts that were accepted for plenary presentations were eventually published and had impressive IFs with a median of 7.7 and a mean of 14.7. In part, this may be a self-fulfilling prophecy, as the reviewers who participated in the decision-making process for the ACR/ARHP Annual Scientific Meeting may also be included in the panel of reviewers for the journals to which the research is submitted.
A quarter of the abstracts that resulted in publication were published in Arthritis & Rheumatism (ACR/ARHP official journal). Annals of the Rheumatic Diseases (European League Against Rheumatism [EULAR] official journal), with the second highest number of resultant publications, published less than half of the number of studies published by Arthritis & Rheumatism. This discrepancy may be attributed to the fact that the 2006 Annual Scientific Meeting is organized by the ACR. Similar data from the annual EULAR scientific conference were not evaluated.
Various national and international organizations have highlighted the need to support research and drug development for the diagnosis, treatment, and management of rare diseases (19). Rheumatology is a unique subspecialty in that it deals with few common diseases and instead encompasses many uncommon or rare diseases, which includes all the pediatric rheumatology diseases. Although research on rare diseases only comprised approximately one-fifth of the abstracts that were presented at the 2006 ACR/ARHP Annual Scientific Meeting, publication ratios for these abstracts were comparable to the publication ratios of common diseases.
The 2006 ACR/ARHP Annual Scientific Meeting employed an embargo policy that stated that discussion of the abstract included as part of a scientific presentation or the presentation of additional new information available at the time of the meeting was under embargo until the conference date. Abstracts published in a peer-reviewed journal after the submission deadline were not considered in violation of this policy.
Approximately 7% (90 of 1,269) of the published abstracts, including abstracts that were presented as a plenary session, were published prior to the 2006 ACR/ARHP Annual Scientific Meeting. Forty-three percent (39 of 90) of these abstracts were published before the submission deadline at the end of June 2006 and were in violation of the embargo policy. However, these abstracts were not noted as withdrawn at the time of presentation at the ACR/ARHP Annual Scientific Meeting.
Approximately 13% of the articles were published more than 3 years after their corresponding 2006 ACR/ARHP Annual Scientific Meeting abstract. This finding is comparable with other medical conferences (1, 8–10, 13, 16, 17). A Cochrane systematic review of 29,729 abstracts that were presented in biomedical meetings and were summarized in 79 studies reported a mean publication ratio of 44.5% (1). However, since the review was based on published studies from the years 1983–2003, it may not reflect recent advances in the publication process, such as the development of online systems for article submission and publication and the great increase in the number of online and offline journals in recent years (20). In order to acknowledge this evolution, we have collected data from recent years on publication trends resulting from biomedical conferences from other societies. We found that the 2006 ACR/ARHP Annual Scientific Meeting had the highest publication ratio among biomedical society scientific conferences in recent years, except for the SUS conference, which had a remarkable publication ratio (68.9%) despite its mean IF (3.4) that was lower than publications from the 2006 ACR/ARHP Annual Scientific Meeting.
Information on the study sponsorship and the statistical significance of the findings in the submitted abstract may lead to selective bias for acceptance and may support future publication. A subanalysis indicated that abstracts were more likely to be accepted for presentation if they contained some statistical analysis or if they presented results from sponsored studies. However, there was no significant difference in publication ratios based on the presence of statistical analysis, the level of significance of presented results, or study sponsorship. Despite these findings, future evaluation of the role of sponsorship disclosure and statistical significance of presented findings in the publication ratios within all presented abstracts may further elucidate underlying trends in publication at the ACR/ARHP Annual Scientific Meeting that are not currently apparent in the conducted subanalysis.
One potential limitation of this study involves the singular investigation of the 2006 meeting; this is due to the large number of abstracts that were included in the analysis and the extensive effort required to ensure the accuracy of the data. Furthermore, based on other studies that assessed annual conferences (7, 8, 12, 14, 16) over a period of 2–3 years, similar publication outcomes can be expected for subsequent ACR/ARHP Annual Scientific Meetings.
In summary, for the first time to our knowledge, a study has provided an extensive analysis of the research products resulting from research presented as abstracts in an ACR/ARHP Annual Scientific Meeting. These results show higher research productivity than other medical societies and reflect appropriate selection of abstracts for presentation by the ACR/ARHP. Nonetheless, approximately 40% of the abstracts were never published, warranting cautious interpretation of the studies that are presented at the ACR/ARHP Annual Scientific Meeting.
All authors were involved in drafting the article or revising it critically for important intellectual content, and all authors approved the final version to be submitted for publication. Ms Woo had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study conception and design. Amarilyo, Woo, Furst, Parker, McCurdy.
Acquisition of data. Amarilyo, Woo, Hoffman, Eyal, Piao.
Analysis and interpretation of data. Amarilyo, Woo, Furst, Parker.
The authors would like to thank Paul Maranian, MS, for his guidance with the statistical analyses included in this study.