Corresponding author: Dr Michelle Giuffrida, Department of Small Animal Clinical Sciences, University of Florida College of Veterinary Medicine, 2015 SW 16th Avenue, Gainesville, FL 32608; e-mail: email@example.com.
Level of evidence (LOE) hierarchies rank scientific articles according to the use of study design features intended to limit bias. Citation analysis of medical articles has shown that studies with high LOE ranking are preferentially cited.
To determine whether clinical companion animal articles reporting study designs classified as high LOE are more frequently cited than those with designs classified as low LOE and to characterize other factors associated with 5-year citation rate.
Literature survey of all original clinical articles published in 2004 in 5 peer-reviewed clinical veterinary journals. For each eligible article, details of scientific and nonscientific characteristics were collected, an LOE classification was assigned, and the 5-year citation rate following publication was determined. Linear regression was used to identify factors associated with citation rate.
Overall LOE was low with 188 of 209 eligible articles describing a study design classified as low LOE. An association was not identified between 5-year citation frequency and LOE classification or any specific feature of study methodology. Articles pertaining to infectious disease or published in the Journal of Veterinary Internal Medicine were associated with significantly greater subsequent citations.
Conclusions and Clinical Importance
Reports of veterinary studies designed to limit the influence of bias are not more widely referenced than articles reporting data obtained through less stringent methodologies. Medical subspecialty and publishing journal prestige can influence an article's subsequent citation rate.
The purpose of medical research is to discover information that will advance the understanding of disease processes and delivery of patient care. New information is relayed to the scientific community through publication of research findings in peer-reviewed journals. The impact of a published article on scientists and clinician-investigators can be estimated by evaluating how frequently the study is cited in subsequent peer-reviewed publications. The number of times an article is cited over a given time period might indicates the level of importance attributed to its findings by research-active members of the medical community.[2, 3] Although commercial research tools provide numerical data regarding which articles are disseminated throughout the literature, they have limited ability to explain why specific articles are cited more than others. Reference or nonreference of research findings based on characteristics of individual studies and their results is termed citation bias. Awareness of existing citation bias is important because this type of bias can influence dissemination of information in the primary and secondary literature and shape systems of belief within the medical community.
Higher level of evidence (LOE) classification has been associated with positive citation bias in a number of studies evaluating citation rate of health science articles.[2, 7-9] LOE hierarchies are organizational tools that rank methods of research according to their freedom from various biases; the less susceptible a study's methodology is to the influence of bias, the higher it ranks on the LOE scale.[10-12] As the concept of evidence-based medicine (EBM) has gained momentum in the veterinary literature, so too has discussion and use of LOE schemes as a means to expedite the core EBM principles of identifying and comparing various types of scientific evidence.[10, 13-19] Methodology-based evidence classification has been acknowledged and utilized in systematic reviews and other evidence-based veterinary review articles pertaining to cruciate ligament disease, oncology, reproduction, osteoarthritis, behavior, and dermatology.[23-28]
A recent survey reported that readers of medical journals attach importance to article LOE classifications and are more likely to cite high-level studies than other articles. If those who read and write articles published in veterinary journals have similar confidence in the results of high LOE studies, it stands to reason that these studies might be widely referenced in the literature. However, citation analysis of veterinary articles has rarely been reported, and existing reports are limited to listing highly cited articles[3, 29] and describing features of the citing articles (as opposed to features of the work being cited).[30, 31] There are no reports designed to assess whether veterinary articles with certain baseline or methodologic features are more likely to be cited. Therefore, the primary objective of this study was to determine whether companion animal articles with high LOE methodologies were associated with higher subsequent 5-year citation rate than those reporting low LOE designs. The secondary objective was to characterize other factors associated with the 5-year citation rate of clinical companion animal articles. We hypothesized that citation evaluation of a broad group of articles would reveal bias in favor of articles reporting high-level evidence study designs.
Materials and Methods
A group of original clinical articles pertaining to dogs and cats was identified by retrospective review of all articles published from January 2004 to December 2004 in the following journals: Journal of the American Animal Hospital Association, Journal of the American Veterinary Medical Association, Journal of Small Animal Practice, Journal of Veterinary Internal Medicine (JVIM), and Veterinary Surgery. Original clinical articles were defined as those in which client-owned companion animals were the primary study subjects, and at least 5 animals were included in the study population. This included but was not limited to studies evaluating treatments, risk factors and prognostic factors, diagnostic testing, and descriptive and comparative case series. Review articles, case reports of fewer than 5 animals, in vitro and cadaveric experiments, studies utilizing purpose-bred or random-source animals, editorials, and other types of articles that did not satisfy the inclusion criteria were excluded from the study. Journals were searched by hand, and each published item was evaluated by the primary investigator to determine eligibility. Evaluation entailed reading title, abstract (if present), and body of each published item as needed to determine whether eligibility or ineligibility criteria were met. The number of items published in each journal in 2004 was queried separately via a search of the Institute of Scientific Information (ISI) Journal Citation Reports (JCR). These figures were compared with the number of articles identified by hand-searching to ensure that no published items were inadvertently overlooked.
Information was extracted from the journal articles that might reasonably be associated with citation rate. Species, number of study subjects, type and number of study centers, and primary geographic location of study were recorded. The number of authors was tabulated and any statements of group authorship or funding disclosure were noted. The number of previous publications attributed to the first and last author on each paper was identified through a search of the ISI Web of Science's Science Citation Index Expanded. Two separate queries were performed for each author to reduce the chance of missing citations. The first search identified the article within the database and initiated a distinct author citation set search based on the name exactly as cited in the article. The second search separately queried the author's last name and first initial followed by an asterisk (so that articles indexed with and without a middle initial would be identified). Each publication abstract was reviewed by the primary investigator to determine whether it could be reasonably attributed to the given article's author (as opposed to a different author of the same name) based on journal of publication, institution, topic, and date of publication. Materials published up to the date of the article being queried were included in the author's tally; subsequent citations were not.
Each entire article was read and then categorized according to subspecialty, primary study aim, and LOE by the first author under the guidance of the senior author who is a senior academic veterinary specialist with 20 years’ experience and advanced training in clinical epidemiology and study design. Each article was assigned to one of the following subspecialty categories: Anesthesia, Behavior, Cardiology, Critical Care, Dermatology, Dentistry, Infectious Disease, Internal Medicine, Neurology, Nutrition, Oncology, Ophthalmology, Orthopedics, General Surgery, or Theriogenology. The Journal of the American Animal Hospital Association assigned a subspecialty to each original article, so for articles published in this journal the designated subspecialty was recorded rather than assigned by the investigator. For statistical purposes, subspecialty categories that accrued more than 10 articles were considered separately, whereas subspecialties that accrued fewer than 10 articles were consolidated into the category “Other.” Each article was categorized according to whether the investigation pertained primarily to diagnostics, prognostics, risk factors, therapeutics, or descriptions of disease. Investigations categorized as descriptions of disease included prevalence studies and descriptive case series that did not seek to test a hypothesis. Each article was assessed for the presence of specific methodologic features and assigned an LOE according to guidelines adapted from those published by the American Academy of Orthopedic Surgeons (Table 1). This set of guidelines provides criteria for assigning LOE to diagnostic and prognostic studies in addition to therapeutic investigations, and its use in this context has been previously described.
Table 1. Level of evidence classification scheme, adapted from The American Academy of Orthopedic Surgeons.
Randomized controlled trial.
Study started before enrollment of first patient.
Patients identified for the study based on their outcome, characteristic, or exposure (“cases”) are compared with those who did not have the outcome, characteristic, or exposure (“controls”).
Study started after enrollment of first patient.
Patients treated one way are compared with a group of patients treated in another way.
Study of patients with a given disease state, treatment, characteristic, or exposure with no comparison group of patients without the disease state, treatment, characteristic, or exposure.
Observation of a defined population at single time point.
Therapeutic studies: investigating the results of a treatment
High-quality RCTa with statistically significant difference or no difference but narrow confidence intervals
Lesser quality RCTa (<80% follow-up, no blinding, improper randomization) or Prospectiveb comparative study
Case-control studyc or Retrospectivec comparative studyd
Prospectiveness, randomization, comparison with a control population, and use of blinding were also recorded separately. When applicable, it was recorded whether stated outcome measures were objective or subjective, whether a statistically significant difference between study group outcomes was reported, and whether authors made clear clinical recommendations based on study findings.
The 5-year citation rate of each article was determined via query of the ISI Science Citation Index Expanded and Social Science Citation Index during the month of January 2011. Articles were identified within the database and a cited reference search was conducted for each article. This search tool tabulates the number of times an article appears in any reference list of the over 8,600 journals indexed by the ISI. The citations returned by each search were evaluated individually to ensure that only citations published within the 60 months after the original publication of each eligible article were tabulated. If a search failed to return any citations for an article, a separate search by study title was conducted to reasonably ensure that no citations were missed.
The 5-year impact factor (IF) for each journal was also determined via query of the ISI JCR during the month of January 2011. The reported IFs were calculated by dividing the number of cites in 2009 to items published during the previous 5 years (2004–2008) by the total number of items published during those 5 years. The publication year 2004 was chosen for study because at the time of data collection, 2004 was the latest year for which complete 5-year citation data were available through the ISI.
Descriptive statistics were calculated. Noncontinuous data were expressed as median values and ranges. Categorical data were expressed as frequencies. Kruskal-Wallis test was used to evaluate the association of citation frequency over 5 years with LOE ranking. LOE was the grouping variable, and citation frequency was the outcome variable. Four levels of LOE were evaluated. Multivariable linear regression analysis was used to evaluate the association between LOE and citation rate after adjustment for other factors that might have an association with citation frequency including specialty, type of study, journal of publication, journal IF, geographic location of study, primary center where the study was performed, species studied, single versus multicenter studies, number of study subjects, disclosure of study funding, number of authors, number of previous 1st author publications, number of previous last author publications, prospective versus retrospective data collection, presence of randomization, presence of a control group, and the presence of blinding. Interactions among the main effects were evaluated. Univariate analysis was then performed evaluating those variables that were not involved in effect modification. In addition to effect modifiers variables with P < .20 on univariate analysis were evaluated in the multivariable model. Postestimation, model specification was evaluated with a link test and the Ramsey regression specification-error test for omitted variables. The overall fit of the model was evaluated with the F statistic. All analyses, including plots evaluating model assumptions, such as the histogram for residuals to evaluate the error term for a normal distribution, were performed in STATA 11. P < .05 was considered statistically significant.
From January through December 2004, 654 items were published in the 5 journals. Of these articles, 445 did not meet the inclusion criteria and were excluded from further analysis. Two hundred and nine articles were included in the study group (Table 2). Three quarters of the studies had dogs as study subjects (156/209), with the minority of reports using only cats (40/209) or cats and dogs (13/209). Studies were most often conducted exclusively at academic institutions (169/209) and most studies were conducted at a single center (182/209). Most studies were performed entirely or partly in the United States (143/209). Seventy-seven primary study centers across 6 continents were represented. The number of study subjects ranged from 5 to 1,842 (median 37, interquartile range 17–90). Funding sources were disclosed in the 71 of 209 studies (34.0%), including 23 studies funded by medical supply or pharmaceutical companies. Of these 23 studies, 17 reported results favoring the funding source.
Table 2. Impact factor and article eligibility according to journal surveyed.
5-Year IF (2004–2009)
# Items Published in 2004
# Eligible Articles
% Eligible Articles
JAAHA, Journal of the American Animal Hospital Association; JAVMA, Journal of the American Veterinary Medical Association; JSAP, Journal of Small Animal Practice; JVIM, Journal of Veterinary Internal Medicine; Vet Surg, Veterinary Surgery.
Number of authors ranged from 1 to 13 (median 4, interquartile range 3–6) and group authorship was rare (4/209). First author prior publications ranged from 0 to 316 (median 3, interquartile range 1–12); last author prior publications ranged from 0 to 223 (median 30, interquartile range 7–61). Specialty categories containing 10 or more articles were as follows: Internal Medicine (44), Oncology (32), General Surgery (25), Cardiology (22), Infectious Disease (21), Orthopedics (16), Neurology (14), and Dermatology (12). All other articles were grouped into the category Other, which contained 23 articles: Critical Care (9), Theriogenology (5), Behavior (3), Nutrition(3), Anesthesia (2), and Ophthalmology (1). There were no articles pertaining to Dentistry.
Overall study design methodology was categorized low with 188 of 209 articles assigned LOE 3 or 4. Article objectives pertained most frequently to therapeutics or description of disease (Table 3). Data were collected prospectively in 114 (54.5%) studies and retrospectively in the remainder. Randomization of animals into study groups was reported in 16 studies (7.6%), comparison of study subjects with a control population was reported in 70 studies (33.5%), and use of blinding was reported in 20 studies (9.6%). Authors made direct recommendations for clinical application of results in 105 studies (50.2%).
Table 3. Level of evidence classification according to primary objective of investigation.
Risk factor studies
Description of disease studies
In the 5-year period after publication, nearly all articles were cited at least once (205/209) (Fig 1). The distribution of citations by LOE category is described in Figure 2. Based on linear regression analysis, there was no association between citation frequency and LOE (r = 0.09, P = .21). This was unchanged when LOE was condensed into high (1 and 2) or low (3 and 4) classifications. However, there was a significant association between citation frequency and the specialty of study, as well as the journal of publication. While controlling for the journal of publication, articles about infectious disease had 8 more citations over 5 years compared to other specialties (95% CI 5 to 11; P < .001). Controlling for specialty studied, articles published in JVIM had 3 more citations over 5 years compared to articles in the other journals evaluated (95% CI 1–5; P = .004). No other factors were found to be significantly associated with greater subsequent citations. The link test revealed correct specification of the model (P = .04 on the prediction and P = .23 on the prediction squared). The Ramsey regression specification-error test suggested no omitted variables (P = .12). Overall the model was significant (F = 18.86, P < .001).
In this survey, LOE classification of clinical companion animal articles was overwhelmingly low and articles reporting high LOE designs were not cited more frequently than articles with low LOE designs. Among the baseline and study design features analyzed, only medical subspecialty and journal of publication were associated with preferential article citation.
Although increased citation of higher LOE articles is a common finding in surveys of health science citation data, not all reports agree.[1, 33, 34] We analyzed articles for many previously identified characteristics associated with preferential citation, but few were found to relate to citation rate in this group of articles. Thus, it is likely that other unidentified variables influence 5-year citation of veterinary clinical articles. For example, study outcome could influence what importance authors attribute to an article and whether it is subsequently cited. In fact, reporting of statistically significant positive results outweighed LOE as a predictor of subsequent citation rate in one study of human hepatobiliary disease. However, because most studies evaluated in this survey were descriptive, we were unable to meaningfully evaluate whether the direction or magnitude of reported results influenced citation rate.
The low LOE of articles in this study supports previous works identifying serious deficiencies in the quality of veterinary research reporting and methodology.[16, 35-38] Because of financial, ethical, and practical obstacles,[11, 17, 39] high LOE articles are limited or nonexistent in many fields of veterinary medicine[12, 16, 20, 21] and so are simply not available for citation, appropriately necessitating the citation of lower level evidence. Furthermore, even studies that appear to be of high LOE design often lack the appropriate power or conduct for their results to truly be considered high-level evidence. Because a thorough investigation into the internal validity of the methods reported in each publication was beyond the scope of this study, it is possible that some or all of the supposed high LOE studies were not considered by authors as such. Therefore, the lack of identifiable association between LOE and citation rate might represent an insufficient number of well-designed and conducted high LOE articles available for citation, rather than a disregard for or lack of understanding among authors of the potential value of results derived from randomized controlled trials and other studies with high LOE designs.
Given the small number of high LOE articles identified, it may be that a significant relationship does exist between citation rate and LOE, but the sample size of this study group was insufficient to detect it. It is also possible that despite our systematic approach to data collection, citations were missed and important data are unaccounted for by our results. We evaluated 5 journals over a 1-year time period, but different results might be returned if data were collected over multiple years or from a larger number of journals. These journals were chosen because they contain articles pertaining to many specialty fields, they are likely to publish high LOE studies, and complete journal and IF data were available through the ISI at the time of query (whereas information was unavailable for many smaller or more specialized veterinary publications). This choice resulted in a group of studies conducted primarily in North America, which might not be representative of the larger body of veterinary clinical research. Arguably, a stronger approach would have been to include any published paper meeting the inclusion criteria, regardless of journal of publication. This could be possible in the future if the JCR expands to tabulate data for all veterinary periodicals.
We found that articles pertaining to infectious disease were more likely to be cited those concerning other specialties. It is not immediately apparent why this should be the case. Articles in this group reported on many infectious agents, both zoonotic and species specific. Analysis of a larger number of infectious disease studies might reveal that group citation rate is driven by a subset of highly cited papers sharing certain characteristics (eg, relating to zoonoses or a specific etiologic agent). The high citation rate could alternatively reflect the popularity of infectious disease as a research topic, as articles reporting on commonly studied subjects will have greater opportunity for citation compared with reports of rare subject matter. By analyzing the total number of articles published in any journal pertaining to each specialty over the 5-year time period, it might be possible to determine the influence of field of research on this study results.
Journal of Veterinary Internal Medicine had the highest IF among the periodicals surveyed in this study, and articles published in JVIM were more highly cited than those published in other journals. However, in all 5 journals, most published items were excluded from analysis, and IF of the publishing journal was itself not associated with article citation rate. This might indicate that JVIM's IF is influenced by the citation rate of clinical small animal articles to a greater degree than that of the other journals evaluated. Future studies evaluating the relative contribution of different types of articles to a journal's IF could be useful in explaining this finding. Journals with high IFs do not necessarily publish articles of high methodologic quality and journal prestige has been reported to outweigh traditional measures of study quality in predicting citation of health science articles. Given that LOE was not associated with higher citation rate, it could be that JVIM articles were preferentially cited not because of article quality but because authors regard the journal itself more highly than other publications.
In this study, citation was regarded as a quantitative rather than qualitative property and as such the context, type of citing article, and publishing journal of subsequent citations were not considered in this analysis. Because citations were only numerically tabulated, we cannot comment on how they were used (ie, if a citation was used in a positive or negative light), although most citations in the health sciences do not contest the findings of the cited article.[1, 42] Medical citations are typically used to persuade readers that the author's methods and conclusions are correct, but they may alternatively be used to show currency by citing the latest research, or to show awareness of social consensus by citing papers considered important in the field, among other reasons. A better understanding of why veterinary authors choose their references could help contextualize the results of numerical citation analysis.
Author self-citation was also not evaluated in this study. An author might cite his or her own work for self-promotional reasons beyond the simple use of a cite to support or refute a claim.[43, 44] In a small field such as companion animal clinical research, an author might be one of only a handful to have published on a topic, which could lead to high rates of self-citation. Self-citation can enhance the apparent impact of an article, particularly if there are multiple authors and therefore multiple opportunities for self-citation, as was the case for many articles in this survey. Journal self-citation can occur for similar reasons. The JCR tabulates journal self-citations and calculates self-citation-adjusted IFs. When ranking the journals in this study according to 5-year IF, the results were identical whether original or adjusted IF was used. JVIM did not have the highest self-citation rate among the 5, and so it is unlikely that journal self-citation alone accounts for the higher citation rate of JVIM articles reported in this study. Evaluating citation data over a time period of 5 years or longer, as we have done, has been suggested as a means to limit self-citation bias, but it is possible that by controlling for subsequent self-citation different results would have been obtained.
This study attempts to determine whether veterinary articles with certain attributes, particularly methodologic features, are more widely disseminated than others. In this survey of clinical companion animal articles, LOE was generally low and a difference in the 5-year citation rate between articles reporting high and low LOE designs was not identified. Medical subspecialty and journal of publication were the only article factors associated with preferential citation. Given the presence of EBM discussion in veterinary publications, citation bias in favor of high LOE studies may become apparent if overall LOE improves in the veterinary literature. However, at the present time reports of veterinary studies designed to limit the influence of bias are not more widely referenced than articles reporting data obtained through less stringent methodologies.