• Open Access

Effects of Diagnostic Test Accuracy and Treatment Efficacy on the Occurrence of Suspected Failure of Heartworm Prophylaxis in Dogs

Authors

Errata

This article is corrected by:

  1. Errata: Erratum Volume 27, Issue 5, 1289, Article first published online: 13 September 2013

Corresponding author: Dr B.W. Rohrbach, VMD, MPH, Dipl. ACVPM (Epidemiology), Department of Biomedical and Diagnostic Sciences, College of Veterinary Medicine, University of Tennessee, Knoxville, TN 37996-4543; e-mail: brohrbac@utk.edu.

Abstract

Background

Food and Drug Administration Center for Veterinary Medicine (FDA/CVM) cited concern regarding failure of heartworm prophylaxis. The positive and negative predictive value of the heartworm antigen test is an estimate of the probability of adult heartworm infection.

Hypothesis/Objectives

Assess the sensitivity, specificity, and predictive value of heartworm antigen tests. Explore the role of heartworm test accuracy and treatment with immiticide to generate reports of suspected failure of heartworm prophylaxis.

Methods

Literature searches for published information on the accuracy of heartworm antigen tests and efficacy of immiticide for treatment of the adult heartworm.

Results

Weighted averages for heartworm antigen test sensitivity and specificity were 78.2 and 97.3%, respectively. Efficacy of immiticide by 2-injection or alternate dose protocols were 88.3 and 89.1%, respectively. Depending on prevalence, the positive predictive value of the heartworm antigen test ranged from 15 to 54% and negative predictive value from 99 to 99.9%. For a hospital testing 1,000 dogs per year, false-positive test results may vary from 24 to 27 dogs. If these dogs were on heartworm preventive, they may undergo treatment and be classified as prophylaxis failures. Ten percent of dogs who were treated and continued or placed on prophylaxis at the time of treatment may have adult heartworms when tested 1 year later and be presumed to represent failure of prophylaxis.

Conclusions

When interpreting heartworm antigen test results, clinical signs, history, and regional prevalence of heartworm disease should be considered in estimating the predictive value of the test result. Limitations of test and treatment should be discussed with clients as part of the decision-making process.

Abbreviations
AHS

American Heartworm Society

APPA

American Pet Products Association, Inc

CAPC

Companion Animal Parasite Council

FDA/CVM

Food and Drug Administration Center for Veterinary Medicine

LOE

lack of efficacy

The prevalence of heartworm infection among dogs presented to veterinary hospitals varies depending on the regional prevalence of infection among the reservoir species and mosquito vector, environmental conditions, and the use of heartworm control interventions. The infection rate among dogs presented to a national veterinary hospital chain, when analyzed by state from 2002 to 2005, varied from 7.6% in Mississippi, 5.4% in Louisiana, 3.4% in Alabama, 3.4% in Arkansas, 0.07% in Washington State, and 0.08% in Oregon.[1] In this study, the overall percentage of dogs with a positive test for heartworm infection was 1.03%. In another survey of data from 2001 to 2006 from a commercial diagnostic laboratory, the overall proportion of dogs testing positive for heartworm infection was 1.37%.[2] The proportion of dogs with a positive test from the northeastern United States was 0.6%, midwestern states 0.8%, western states 1.2%, and southeastern states 3.9%.

Heartworm prophylaxis is considered to be highly effective. However, several recent publications cite evidence of resistance of heartworm larvae to the macrocyclic lactone class of drugs. In 1 study, Dirofilaria immitis was found to be genetically heterogeneous. This genetic heterogeneity has implications for the development of genetically selected strains of the parasite.[3] Authors of 2 additional experimental studies reported that a single dose of a macrocyclic lactone may not prevent development of adult heartworms in experimentally infected dogs.[4, 5] However, the latter 2 studies were designed to mimic prior standard efficacy studies as dictated by FDA/CVM rather than the American Heartworm Society (AHS) recommendations for repeated monthly administration of heartworm prophylaxis.

Surveys indicate that >1% percent of dogs tested at veterinary hospitals are positive for infection with adult heartworms.[1, 2] The Companion Animal Parasite Council (CAPC) estimates that up to 300,000 dogs per year are infected with adult heartworms.[6] A survey by the AHS of 25,000 veterinary clinics in the United States (50% response rate) reported 250,000 cases of heartworm infection in 2006.[7] Based on the CAPC estimate and the AHS survey, the total number of cases presented to veterinary clinics may be between 300,000 and 500,000 cases annually.

The true incidence of suspected failure of heartworm prophylaxis is unknown, and those reported to the FDA/CVM are likely only a small proportion. One author has placed the rate of suspected failure at <0.1–0.0001% of dogs on prophylaxis.[8, 9] Instances of suspected failure of heartworm prophylaxis are reported to the FDA/CVM.[10] A report in 2005 cited 5,794 cases of suspected failure of heartworm prophylaxis, 1,301 of which, when analyzed, were assumed to be related to product failure.[11] In that report, the FDA/CVM stated that, at this time, it is unclear whether reports of failure of prophylaxis represent rare occurrences of failure that have existed for a long time, but not reported regularly or promptly, or whether there is a true increase in complaints of ineffectiveness and real variability among products.[11]

The accuracy of a diagnostic test is described as test sensitivity and specificity. Sensitivity is calculated as proportion of patients known to have a disease that are accurately classified by the test. Specificity is the proportion of patients known to be free of disease that are accurately classified by the test. Accuracy of a diagnostic test is determined by applying the test to a group of animals with disease status that has been established by an accepted gold standard for diagnosis. In the case of heartworm disease, the gold standard of diagnosis is necropsy. The ability of a test to predict the disease status of a patient with unknown disease status is described as the test predictive value, and depends on the sensitivity and specificity of the test and the pretest probability of disease in the patient. Pretest probability is estimated from patient history, clinical signs and regional prevalence of heartworm infection. The positive predictive value of a test is the probability, given a positive test result, that the patient has the disease in question. The negative predictive value is the probability, given a negative test, that the patient is free of disease.

Causes of suspected failure of heartworm prophylaxis may result from resistance of L3–L4 stage D. immitis to macrocyclic lactones, lack of compliance with administration of heartworm prophylaxis, failure of some dogs to absorb or metabolize the drug, false-positive results of the heartworm test, failure of treatment to eliminate adult heartworm, and poor timing for administration of heartworm tests. This study was designed to examine the role of the heartworm antigen test and treatment for adult heartworm infection with immiticide to contribute to the incidence of unnecessary treatment and suspected failure of heartworm prophylaxis.

Materials and Methods

Literature searches were performed with the PubMed and CAB abstract databases. Search terms heartworm, test, and (sensitivity or specificity) were combined and used to identify published studies citing the accuracy of heartworm antigen tests. Only studies with the results of tests confirmed by necropsy, that specified the number of dogs studied, were included. Immiticide,1 RM340, and melarsomine were used as search terms to identify articles relating to the efficacy of immiticide to eliminate adult heartworm infection. Field and laboratory studies with treatment by the 2-injection protocol (2 doses of immiticide 2.5 mg/kg IM 24 hours apart) or the 3-dose alternate protocol (1 dose of immiticide 2.5 mg/kg followed in 1 month by 2 doses of 2.5 mg/kg IM 24 hours apart) were included. Efficacy of the adulticide was confirmed either by a change from a positive heartworm test result pretreatment to negative posttreatment heartworm test result or by necropsy. Abstracts retrieved were reviewed to identify a list of relevant articles that met the inclusion criteria and duplicates were removed from the list. The full text of all relevant articles then was obtained. Additional information on the prevalence of reported failures of prophylaxis, population estimates for dogs in the United States, number of dogs seen by a veterinarian annually, percentage of dogs on heartworm preventive, percentage tested annually, and percentage positive on heartworm tests each year was obtained from a variety of published and online sources. Where data were lacking, expert opinion was obtained.

Results

The 2008 census reported there are 72.9 million pet-owning households of which 39% own an average of 1.69 dogs for a total of 48,048,390 dogs.[12] An estimated 91% of dog-owning households reported ≥1 veterinarian visits during 2010. Assuming 1 visit per dog, approximately 39,788,872 dogs were presented to veterinary hospitals of which 58% received heartworm preventive medication for a total of 23,077,546.[12] According to an earlier June 2001 Gallup poll of 18,000 veterinary clinics nationwide, 55% of dogs in US dog-owning households are on heartworm preventative, leaving 27 million dogs at risk of acquiring heartworm disease. One author has stated that there are approximately 500,000 dogs, 1% of the US dog population, that are treated for heartworm disease each year in the United States.[13] In a recent study of 84 naturally infected dogs, only 1 dog had a unisex male infection. The authors suggest that sex disequilibrium occurs at low worm burdens but may disappear with higher worm numbers. If even 1% of adult heartworm infections are male unisex, they would not be identified by the heartworm antigen test or treated.[14] The number of dogs treated with melarsomine,1 by either the 2-injection or 3-dose alternate protocols in which adult heartworms were eliminated based on a negative posttreatment antigen test or necropsy results, is found in Table 1. The weighted average for melarsomine efficacy among dogs treated with a 2-injection protocol of melarsomine was 88.3% and when treated with the 3-dose alternate protocol was 89.1%.

Table 1. Efficacy of melarsomine in dogs with adult heartworm infection by the 2-injection protocol (2 doses of 2.5 mg/kg IM in 24 hours) or 3-dose alternate protocol (single dose of 2.5 mg/kg followed in 1 month by 2 doses of 2.5 mg/kg IM in 24 hours) treatment
AuthorYearInfectionTypeDoseNClass[20]PIOutcome MeasureResult (%)a
  1. PI, posttreatment interval before outcome measurement; –, not given.

  2. a

    Percent that seroconverted to negative or no worms at necropsy.

  3. b

    2-dose treatment followed by a 2nd 2-dose treatment after 4 months; data combined with alternate dose treatment.

Atwell[21]1989NaturalField2251–30 daysNecropsy60.0
Miller[22]1995NaturalField24224 monthsAntigen76.2
    2b39 8 monthsAntigen89.7
Case[23]1995NaturalFieldAlternate3734 monthsAntigen91.9
Keister[24]1992Experimental 262 monthsNecropsy50
  Laboratory Alternate62 monthsNecropsy83.3
Atwell[25]1992NaturalField2533 monthsAntigen92
Vezzoni[26]1992NaturalField212113 monthsAntigen99.2
    214423 monthsAntigen96.6
    Alternate5533 monthsAntigen98.2
Morini[27]1998NaturalFieldAlternate1735 monthsAntigen100
Polizopoulou[28]2000Natural 2351–36 monthsAntigen100
Merial[29]NaturalField21691,24 monthsAntigen76.2–81
    Alternate1691,29 monthsAntigen89.7–98.2
  NaturalField21021,24 monthsAntigen84
    2b1021,29 monthsAntigen94
  NaturalFieldAlternate4434 monthsAntigen89.2
  NaturalFieldAlternate1034 monthsAntigen100
Weighted average   2    88.3
    Alternate    89.1

Published data citing accuracy for various heartworm antigen tests are reported in Table 2. The weighted average for sensitivity and specificity among all tests was 78.2 and 97.3%, respectively. The positive and negative predictive value and total number of false-positive and negative results, at a prevalence of heartworm infection at intervals from 1/1,000 to 1/10, when testing 1,000 dogs are presented in Table 3.

Table 2. Sensitivity and specificity of heartworm antigen tests
TestYearHW+ DogsHW− DogsGold StandardBlindingSensitivitySpecificityReferences
  1. a

    Not stated.

  2. b

    Weighted average for all studies.

Filarochek1986251123Necropsy-a9098Courtney[30]
Dirochek1986251123Necropsy-8895Courtney
Filarochek198793109Necropsy-9698Ely, Courtney[31]
Dirochek19881713NecropsyYes76.584.6Brunner et al[32]
Filrochek19881713NecropsyYes94.1100Brunner et al
ClinEase CH19881713NecropsyYes64.792.3Brunner et al
CITE19881713NecropsyYes88.2100Brunner et al
Dirochek1988225202Necropsy-9096Courtney et al[33]
Cite1990341213Necropsy-83100Courtney et al[34]
Dirochek1990341213Necropsy-8697Courtney et al
Dirochek1995115109NecropsyYes77100Courtney, Zeng[35]
Petchek PF1995115109NecropsyYes76100Courtney, Zeng
Dirochek19963328NecropsyYes85100Hoover et al[36]
Petchek19963328NecropsyYes73100Hoover et al
Assure/CH19963328NecropsyYes8289Hoover et al
Snap19963328NecropsyYes48100Hoover et al
Uni-Tec CHW19963328NecropsyYes67100Hoover et al
CITE19963328NecropsyYes58100Hoover et al
Petchek1996175110Necropsy-7398Martini et al[37]
Diasystems1996175110Necropsy-8195Martini et al
Uni Tec1996175110Necropsy-7997Martini et al
Dirochek200114097NecropsyYes7194Courtney, Zeng[38]
Petchek200114097NecropsyYes7697Courtney, Zeng
Snap PF200114097NecropsyYes6798Courtney, Zeng
Solo Step200114097NecropsyYes6098Courtney, Zeng
Abboscreen200114097NecropsyYes6496Courtney, Zeng
Witness200114097NecropsyYes7194Courtney, Zeng
Snap RT200320832NecropsyYes8497Atkins[39]
Solo Step CH200320832NecropsyYes7997Atkins
Chat200320832NecropsyYes7897Atkins
VetScan VS2201111830NecropsyYes33100Lee et al[40]
SnapHW RT20114030NecropsyYes98100Lee et al
Totals 34622088  78.2b97.3b 
Table 3. Positive and negative predictive values, false-negative and false-positive test results of the heartworm testa when applied to 1,000 dogs with varying pretest probability (prevalence) of infection with adult heartworm from 0.10 to 10%
Prevalence (%)Proportion PositivePPV%NPV%False NegativeFalse Positive
  1. PPV, positive predictive value; NPV, negative predictive value.

  2. a

    Estimated sensitivity = 78.2%, specificity = 97.3%.

0.11/10002.8~100~027
0.55/100012.799.9127
11/10022.699.8227
22/10037.299.5426
33/10046.399.3726
44/10054.799.1926
55/10060.498.81126
66/10064.998.61325
77/10068.698.31525
88/10071.698.11725
99/10074.197.82025
1010/10076.397.62224

Discussion

The weighted average for studies on the efficacy of melarsomine for treatment of adult heartworm infection is approximately 90%. Assuming conservatively that 300,000 dogs are diagnosed and treated for heartworm infection annually, approximately 10% of 300,000 dogs (30,000) would harbor ≥1 adult heartworms after treatment with either the 2-injection or 3-dose alternate protocols. If half (personal communication: Drs Tom Nelson Anniston, AL, Craig Prior, Nashville, TN, and Gary Block, Coventry, RI) (15,000) of these dogs were treated and placed on, or continued on, heartworm preventive and were retested 1 year later, they may be classified as suspected failure of prophylaxis.

The heartworm test may be applied to an animal suspected to have heartworm disease or as a screening test before or after a course of heartworm prophylaxis or treatment. Most heartworm tests conducted in the clinical setting are used annually to screen asymptomatic dogs for the presence of adult heartworms. Many of these dogs have been on heartworm preventive and if infected may have a low heartworm burden, thus decreasing the sensitivity of the heartworm test. In the private practice setting, heartworm tests are conducted sporadically by personnel with a wide variety of skills, training, and supervision. Therefore, the published estimates of sensitivity and specificity for the heartworm test, obtained under experimental conditions, may overestimate the accuracy of the test when applied in the private practice setting. However, based on the published sensitivity of the heartworm test, approximately 22% of dogs with existing adult heartworm infection would have a false-negative test result and would be missed before starting a program of heartworm prophylaxis. Additionally, approximately 2.7% of all dogs tested that did not harbor adult heartworms would have a false-positive test result. In the latter case, if these dogs were on heartworm preventive the previous year, they may be classified as suspected failure of heartworm prophylaxis and may be subjected to unnecessary treatment.

A separate analysis of data published since 2000, including only tests that are currently available, yielded a sensitivity of 70.7 and specificity 96.7%. These estimates, for both sensitivity and specificity, are lower than our overall estimate of 78.2 and 97.3%, respectively. Therefore, the overall estimate provides a more conservative estimate of the effect of the antigen test on suspected lack of efficacy (LOE).

Using the weighted average for sensitivity and specificity from published data on the accuracy of the heartworm test and the estimated prevalence of heartworm infection found in a published survey for various parts of the United States, the positive and negative predictive values for the test among dogs in the northeastern states are 14.9 and 99.9%, midwestern states 18.9 and 99.8%, western states 26.0 and 99.7%, and southeastern states 54.0 and 99.1%, respectively.[2] Given the reported average prevalence of adult heartworm infection among the various regions in the United States, as many as 46–85% of dogs with a positive heartworm test may not harbor adult heartworms. Among the same regions, between 0.1 and 1% of dogs with a negative heartworm test result may be infected with adult heartworm.

For the following scenario, we use data from the 2010 American Pet Products Association (APPA) survey. An estimated 38,788,872 dogs were presented to veterinarians during the year. Assuming a stable population with an average life expectancy of 13 years, the estimated number of new puppies born to this population each year would be 3,060,682. If 58% (1,775,196) were placed on heartworm preventive at 8 weeks of age, and not given an initial heartworm test until approximately 1 year later, an estimated 44,286 of these dogs that were free of heartworm infection would test positive at the first test.

In the survey of Banfield hospitals, the estimated prevalence of heartworm infection was 7.6% in Mississippi. This would result in a positive predictive value of 70.4% and negative predictive value of 98.2%. The implication is that almost 30% of dogs that undergo annual screening for heartworm infection, and have a positive heartworm test, are uninfected, and 2% of dogs with a negative test would harbor adult heartworms. Therefore, an estimated 30% of puppies, started on heartworm prophylaxis and tested after 1 year, having a positive heartworm test and residing in Mississippi, may be free of adult heartworms, classified as suspected prophylaxis failures, and may be subjected to unnecessary treatment for adult heartworm infection.

Based on the APPA data, we estimate that 39,788,872 dogs in the United States are presented to a veterinarian at least once annually of which 58% (23,077,546) are placed on heartworm prophylaxis. Conservatively estimating that 50% (11,538,773) of these dogs will be tested annually and by substituting an average prevalence of adult heartworm infection of 1% among the population of dogs in the United States, the positive predictive value of the heartworm test would be 22.6% and the negative predictive value 99.8%. This would result in a total of 308,431 dogs negative for adult heartworm that would have positive heartworm test results. If we substitute a sensitivity of 70.7% and a specificity of 96.7% for antigen tests that are currently available and evaluated since 2000, the estimated number of dogs free of adult heartworms that would have a positive antigen test result would increase to 376,972. When these test results are accepted as truth, it is reasonable that there would be widespread doubt, among both veterinarians and dog owners, regarding the efficacy of current drugs used for heartworm treatment and prophylaxis in the United States.

For a hospital that screens approximately 1,000 dogs annually for heartworm infection, using an average prevalence of 1%, one would expect to incorrectly classify 2 dogs as negative for adult heartworms and misclassify 27 dogs as positive for heartworms when they are in fact heartworm-free. At a prevalence of 0.1–10%, a hospital that tests 1,000 dogs annually may have between 0 and 22 dogs that have false-negative test results and between 24 and 27 dogs with false-positive test results. Assuming that many dogs with false-positive test results were on heartworm preventive during the previous season, they may be considered prophylaxis failures. Alternatively, between 0 and 22 dogs would have false-negative test results. If placed on heartworm prophylaxis and retested positive the next year, they also would be considered prophylaxis failures.

When estimating the overall effect on sensitivity and specificity by combining ≥2 tests in serial or parallel, the assumption that each test measures a characteristic the presence or value of which is independent of the presence or value of the other applies.[15, 16] Currently available heartworm antigen tests detect protein secreted mainly by adult female D. immitis and therefore do not meet the assumption of independence.[17-19] Therefore, when repeating the antigen test, whether the result of the 1st test is true or false, with a second test from a different manufacturer, the result of the 2nd test has a greater, but unknown, probability to mimic the original test result. Therefore, when ≥2 tests are combined, either serially or in parallel, the overall sensitivity and specificity of the combined tests cannot be estimated. The value of performing a 2nd antigen test to confirm the result of a previous test would be primarily to compensate for any procedural errors that may have occurred during the performance of the initial test. The use of a test for microfilariae, thoracic radiographs, and echocardiography may be useful to confirm a positive antigen test. Although a positive test for microfilariae is highly specific, it lacks sensitivity. Information on the specificity of echocardiography was not found. Radiography lacks both sensitivity and specificity. These ancillary tests may be useful to confirm the presence of adult heartworms among dogs with heavy burdens, but they have limited use to confirm the antigen test result in dogs with a low heartworm burden and subclinical disease.

The authors recognize that the results and conclusions of this study are based on imprecise estimates of test accuracy, efficacy of treatment, prevalence of heartworm infection, and the number of dogs tested annually. In addition, there are numerous sources of error that can play a role in determining the accuracy of an individual heartworm test performed in the clinical setting. Differences in sensitivity among tests exist, but are minor, and specificity is consistently very high with all tests, which is an important attribute. Potentially erroneous results may occur because of variable technician capabilities, critical timing for reading results, and clarity of end result. To obtain reliable and reproducible results, antigen tests must be performed in strict compliance with the manufacturer's instructions. False-positive results can occur, but usually are because of technical error, such as inadequate washing steps or delay in reading the test. False-negative test results occur most commonly when infections are light, female worms are still immature, only male worms are present, the test kit instructions have not been followed or some combination of these factors. Assessment of the accuracy of heartworm tests in the published literature was done under controlled conditions by personnel trained to conduct the tests. In the clinical setting, tests are conducted by personnel with a wide range of skills, varying degrees of supervision, and performed on a sporadic basis. Therefore, published estimates of sensitivity and specificity would be expected to overestimate the accuracy of test results. However, published data is the best information currently available and when possible we have attempted to use conservative estimates.

Although the commercially available antigen tests for heartworm infection are considered to be accurate, when applied to a large population of dogs with a low prevalence of adult heartworm infection, even a test with a high specificity will produce a substantial number of false-positive results. When these test results are accepted as accurate and used to assess the effectiveness of heartworm treatment or prophylaxis, the efficacy of these procedures may be brought into question. Veterinarians should be aware of the limitations of the heartworm antigen test and communicate these limitations to their clients in order to increase client acceptance and confidence in the utility of heartworm treatment and prophylaxis.

The current antigen tests for heartworm infection emphasize high specificity at the expense of sensitivity. This is an attempt to decrease the number of false-positive diagnoses. An ideal screening test for heartworm infection will have high sensitivity, at the expense of specificity, in order to identify as many infected individuals as possible. Once patients with initial positive test results are identified, these patients would be subjected to a second confirmatory test that measures an independent characteristic (eg, circulating genetic material from the heartworm), and has both high sensitivity and specificity. Although the 2nd test may be more expensive and may have to be performed in a commercial laboratory setting, it would prevent numerous dogs from undergoing unnecessary treatment. In addition, dogs positive on the heartworm antigen test, but free of adult heartworms, may be properly classified and their status would not be automatically attributed to failure of treatment or prophylaxis. Currently there is no such test available.

Acknowledgment

Conflict of Interest Declaration: Authors disclose no conflict of interest.

Footnote

  1. 1

    Immiticide; Merial, Duluth, GA

Ancillary