Outcome measures reported in abstracts of randomized controlled trials in leading clinical journals: A bibliometric study

Abstract Backgrounds The CONSORT for Abstracts checklist published in 2008 recommends that authors report effect size for their studies. Meanwhile, the FDA strongly recommends reporting both ratio and difference measures. However, the measures of effect used in recent clinical trial reports remain unknown. This study is aimed to reveal trends regarding the measures of effect of interventions described in abstracts of recent randomized controlled trials (RCTs) in leading journals. Methods A bibliometric analysis of data was obtained by electronic searches. Human RCTs published in 2016 in the following five journals were searched using PubMed: Annals of Internal Medicine, British Medical Journal, Journal of American Medical Association, The Lancet, and New England Journal of Medicine. Main outcome is numbers of studies reporting each measure in their abstracts. Results Among abstracts of 334 articles, measures most frequently used were relative risk alone (n = 169), followed by absolute risk alone (n = 92), and raw data alone (n = 58). Reporting of the following measures was relatively limited: both ratio and difference measures (n = 8), raw data with ratio measures (n = 5), and raw data with difference measures (n = 2). None of the studies reported raw data with both ratio and difference measures. Only 15 articles described multiple measures of effect in their abstracts. Conclusions More than half of the RCT abstracts published in the five leading journals in 2016 reported risk ratio alone to indicate effect size. Even abstracts in the five leading journals did not adhere fully to the CONSORT for Abstracts or FDA recommendations.

compared with the absolute value or difference index. 2,3 The absolute value is inevitable for evaluating the size of the problem. Despite repeated recommendation to report both the RR and the RD, 4-8 RD has been relatively underreported. [9][10][11] Abstracts of medical articles must contain accurate descriptions of contents with sufficient transparency, as many readers use abstracts to obtain efficiently the necessary information. The CONSORT statement for abstracts published in 2008 lists items that authors should consider when writing an abstract to report results of their randomized controlled trials (RCTs) in medical journals. 12,13 In this statement, the use of estimated effect sizes (ie, relative risk [RR], relative risk reduction [RRR], odds ratio [OR], hazard ratio [HR], and risk difference [RD]) is recommended to describe study outcomes. For example, PLOS Medicine suggests reporting raw data and RR along with 95% confidence intervals. 13 The FDA has repeatedly noted that ratio measures tend to emphasize effects more than difference do, even when the same data are used, and that interpreting results requires caution. 2,9,10 It remains unknown as to how effect size is reported in abstracts of recent RCTs conducted in various fields. In this study, we reviewed the submission guidelines of representative clinical medicine journals regarding the abstract for RCTs and aimed to clarify the measures of effect reported in the abstracts published in these journals.

| Inclusion and exclusion criteria
Among articles retrieved electronically, abstracts that did not clarify the type of study (ie, RCT or not) and those reporting RCTs that were not related to treatment or prevention (eg, diagnosis) were excluded.
If clinical effects were assessed using multiple measures within one study, only the primary endpoint was subjected to evaluation. If the primary endpoint was assessed using multiple effect measures, only the first effect measure mentioned was subjected to evaluation.

| Classification of intervention purpose
Randomized controlled trials ultimately included in this study were classified according to the purpose of interventions (treatment or prevention) based on the primary outcome described in the methods section of each article.

| Primary outcome
We read through the abstracts of all included studies and counted the numbers of articles that described the primary endpoint in terms of ratio measures (eg, RR, HR, OR), difference measures (RD), raw data alone, both ratio and difference measures, raw data with ratio measures, raw data with difference measures, and raw data with both ratio and difference measures. In addition, the number of characters in the abstract of the final inclusion RCTs published on PubMed was counted. When these approaches were unsuccessful, email inquiries were sent to the editorial office of each journal via the "Contact us" link on the web site. If no responses were received, a reminder was sent two weeks after the first inquiry; no further contact was made after this, regardless of the absence of any response.

| Author roles
The first author (TS) designed the study and prepared the study protocol through discussions with YT. TS extracted data necessary for the analysis from the target journals (this included determining whether the included RCTs were treatment-or prevention-focused studies).
Following this, TN checked all the processes performed by TS and asked questions, if any; differences in opinion were resolved through discussion. YY and MH read and proofread the finished manuscript. All authors approved the final version of the submitted manuscript.

| Statistical analysis
We evaluated how many times each measure of effect was used to describe the primary endpoint in the abstracts of the included RCTs and assessed differences among journals using the chi-squared test.
The Bonferroni correction for multiple comparisons was used to test differences among the five journals regarding 10 items. A P < .005 was considered statistically significant. All statistical analyses were performed using Stata ver. 14.

| Electronic searches
Among 363 RCT articles extracted from five leading journals in 2016, we read through the abstracts of 334 articles identified to meet the inclusion criteria, excluding five non-RCT reports (which could not be determined from the abstracts) and 24 RCT articles with objectives other than prevention/treatment (determined from the abstracts; Figure 1). In addition to RR, HR, and OR, attack rate ratio, incidence rate ratio, recurrence ratio, and rate ratio were used as ratio measures; these were all accepted as ratio measures in this study. We identified 169 abstracts reporting relative risk alone, 92 reporting absolute risk alone, 58 reporting raw data alone, eight reporting both ratio and difference measures, five reporting raw data with ratio measures, and two reporting raw data with difference measures. None of the journals had abstracts reporting raw data with both ratio and difference measures. Table 1

| Differences between treatment studies and prevention studies
As shown in Table 2 The proportion of prevention studies reporting absolute risk was significantly lower than that of treatment studies (prevention studies: 18/127, 14･2% vs treatment studies: 74/207; 35･7%; P < .01).

| Statistical test for rates of use of multiple effect measures among journals
The results of the chi-squared test revealed no significant differences in the rates of use of multiple effect measures among journals (Table 1).

| Number of characters in the abstract of the final inclusion RCTs
Abstracts of all 334 RCTs used in this study were obtained from PubMed, and the number of characters in the abstract was counted. As a result, not only BMJ but also other four journals exceeded the upper limit of the number of characters in abstracts (Table 1).

TA B L E 1
Details of studies presenting ratio measures, difference measures, and raw data to describe effect size (Overall) The guidelines note the following: "Abstracts should be 250-300 words long: you may need up to 400 words, however, for a CONSORT or PRISMA style abstract."

| Evaluation of submission guidelines
Only JAMA included submission guidelines in the journal publication itself (eg, the December 2016 issue; Table 1 Why is the "ratio measurement" preferred? First, the reason the "ratio" is more popular than the "difference" is the large amount of information indicated by the "ratio" in the comparison of the results of the two groups. For example, the "difference between the two groups" is 5%. The difference between "1% and 6%" is 5%, and "90% and 95%" is 5%. If the former is indicated by a ratio, it is 6 times, and the latter about 1.05. Comparing the difference and the ratio, it is estimated that the ratio is more likely to give the reader an image of the research results. In the comparison of two groups, it seems that the ratio is preferred when selecting one of the difference or the ratio.
Second, this is not just a matter of awareness of individual researchers: Among the five journals examined in the present study, NEJM and Lancet tended to prefer ratio, whereas AIM, BMJ, and JAMA tended to prefer difference. Furthermore, if the word limit for abstracts is low, contents that can be described also become limited.
As for BMJ, the guidelines specifically state that CONSORT-style abstracts may be up to 400 words in length. 14 Thus, the results ob-  It is desirable that each journal provide submission instructions for authors to report their results "along with multiple effect measures," as this would promote the dissemination of and adherence to the recommendations of the CONSORT for Abstracts and FDA.