How do we establish a biological marker for a behaviorally defined disorder? Autism as a test case


  • Benjamin E. Yerys,

  • Bruce F. Pennington


We discuss the recent push to develop support vector machines and other cluster analyses as a means for biological signatures as early screens for autism. These methods not only hold great promise but also require careful consideration prior to implementation. We outline three validity tests and potential methods for addressing them. Autism Res2011,4:239–241. © 2011 International Society for Autism Research, Wiley Periodicals, Inc.

Using autism spectrum disorders (hereafter, autism) as a test case, we can ask how successful neuroimaging measures are at predicting a behavioral diagnosis [See Table 1 in Lange et al., 2010]. This endeavor has yielded apparently encouraging findings with a recent diffusion tensor imaging (DTI) study showing over 90% sensitivity, specificity, and accuracy [Lange et al., 2010] based on white matter microstructure (WMM) differences in the superior temporal gyrus and temporal stem. These WMM differences could be a promising biological marker (“biomarker”) for autism.

There are three potential uses for such a biomarker: (1) a reliable phenotype for identifying homogeneous subtypes of a disorder for genetic studies; (2) improving the reliability of clinical diagnosis, which can support diagnosis in difficult cases, indicating that the problems are real and confirm the need for a specific treatment; (3) a reliable screening tool for infants and children to determine risk for developing a disorder. Autism or “Autisms” are disorder(s) characterized by genetic and behavioral heterogeneity [Bishop, 2009], and therefore it is likely that more than one biomarker will be identified for all three of the applied uses listed above. The remainder of this editorial will focus on the use of biomarkers as screening tools.

We contend that in order for WMM differences with DTI or any other biological finding to be developed as a biomarker for autism screening, it must pass three validity tests. These validity tests include:

  • 1.Is the biomarker present prior to the onset of symptoms?
  • 2.Is the biomarker specific to the disorder?
  • 3.Does the biomarker define the disorder independently from the behavioral symptoms used to make the diagnosis?

Currently, identifying behavior symptoms through standardized caretaker or semi-structured, play-based interviews, or questionnaires [Lord, Cook, Leventhal, & Amaral, 2000] combined with clinical judgment are the “gold standard” for making an autism diagnosis. An inherent limitation of this diagnostic method is that a child must be of a certain chronological age in order to assess the absence or presence of critical diagnostic behaviors (e.g. problems in joint attention, gestures, social reciprocity, pointing, directing facial expressions to others). In order for a biomarker to supplant a behavioral measure, the marker should be observable as a precursor to the behavioral symptoms. White and gray matter tissue development have been the potential biomarkers submitted to classification analyses to date [Lange et al., 2010]; those tissues undergo dramatic change from infancy into early adulthood [Giedd et al., 1999], and therefore it is unclear if a WMM biomarker like the one suggested in Lange et al. [2010] is causative of autism or merely a correlated outcome from the interaction of genetic predisposition, environmental factors, and experiences shaping brain structure. Furthermore, a biomarker that predates behavioral symptoms would allow clinicians to identify children at high risk for developing the disorder and initiate treatment immediately. Studies to date have attempted to classify children and adults on the autism spectrum, but have not yet examined the viability of these techniques in longitudinal studies following infants at-risk for autism [Ecker et al., 2010a, b; Lange et al., 2010; Oller et al., 2010]. With two large-scale multi-site infant sibling studies under way in the United States (Early Autism Risk Longitudinal Investigation and Infant Brain Imaging Study), we may have answers to these questions in the coming years.

A second key feature of a biomarker is that it not only be sensitive to detecting autism, it should also be very specific to autism. That is, detection of white matter changes in brain regions relating to language delay, cognitive impairments, or executive control impairments are likely to be present in multiple disorders and may not assist in differentiating between children with autism and other neurodevelopmental disorders. Therefore, in addition to conducting this analysis in children prior to the onset (or observation) of behavioral symptoms, multiple “risk” populations need to be included such as children at-risk for ADHD, anxiety, dyslexia, and phonological disorder to establish the specificity of the biomarker [Ecker et al., 2010a]. Ecker et al. [2010a] should be commended for including an additional clinical group (ADHD); however, future delineations between clinical groups will require other disorders that are often involved in diagnostic differentials (e.g. speech/language impairment or social phobia). If unique biomarker signatures could be developed for closely related behavioral disorders that are often involved in demanding differential diagnostic decisions (currently based upon behavioral symptoms), that would be a significant tool for future clinicians.

A related point is that the interpretation of a biomarker test is constrained by the base rate of the neurodevelopmental disorder in the population, and the likelihood ratio of the biomarker test. This point has been discussed in detail elsewhere [Bishop, 2010], but in brief, the studies in search of a biomarker often include a rate of the disorder that is significantly higher than the population base rate (e.g. 50% if the autism and control sample have equal n's). When the sensitivity and specificity for the biomarker derived from such a selected sample are applied to a sample distribution approximating the population base rate, we find that the screen identifies a large portion of nonautistic children as autistic (high false-positive rate, meaning low specificity), and thus is significantly less useful than established questionnaires [e.g. Berument, Rutter, Lord, Pickles, & Bailey, 1999]. One statistical method for determining the accuracy of a biomarker is to calculate the likelihood ratio, which indexes the likelihood that a positive biomarker test is affiliated with a true case of autism [Heneghan, 2010]. For example, Heneghan [2010] points out that the Ecker et al. [2010a] study has a likelihood ratio of 4.5 based on the sensitivity and specificity calculations, indicating that an individual with a positive biomarker test has only a 4.5% chance of having autism! Clearly, we do not want to tell parents that their child will have autism, and be wrong 95% of the time. This illustration highlights the need for including more realistic statistical analyses that include base rates in future biomarker investigations.

The third, and perhaps the most critical feature, of a biomarker is that a behavioral diagnosis cannot be replaced by a neuroimaging test until the neuropathology of the disorder is well understood (at least in a well-defined subset of cases). Otherwise, the accuracy of the biomarker is dependent on its correlation with the behavioral measures used to define the disorder. This conundrum of using the behavioral measure to confirm the accuracy of a biomarker is the current state of our research [Stevenson & Kellett, 2010]. Using Alzheimer's disease as an example, Dr. Alois Alzheimer presented his data on the disease's neuropathology (amyloid plaques and neurofibrillatory tangles) over 100 years ago, but only in recent years biomarkers have replaced behavioral tests. Furthermore, there are examples of individuals presenting with the biomarkers at autopsy without the memory loss and disorientation associated with Alzheimer's disease [Riley, Snowdon, & Markesbery, 2002]. Does this mean the individual had the disease? Were the behavioral measures insensitive to those individuals' symptoms? Or, does some third factor lead to both the observable neuropathology and the behaviors characterizing Alzheimer's disease, and we have yet to identify this third factor? In the case of Phenylketonuria, white matter tissue alterations could have been considered a biomarker for the metabolic disorder because of their presence [Anderson & Leuzzi, 2010]; however, many disorders, including autism, have white matter alterations, and thus identifying elevated levels of phenylaneline as a cause of white tissue alterations is a superior biomarker. Our understanding of neuropathology in autism is still within its early stages, and while data suggest atypical neural connectivity as a cause [Levitt & Campbell, 2009], additional research is needed to conduct careful characterization of the neuropathology. Understanding the fundamental neuropathology that causes phenotypic behaviors, and is not merely correlated with them, will provide the new gold standard to shift the diagnosis of neurodevelopmental (and any behavior-based) disorders from behavior symptoms to biology.

In summary, there is an increasing need for developing a biomarker for a variety of behaviorally defined disorders, including autism. One potential use for a successful biomarker of autism is the development of a screening tool to identify all or a well-defined subset of infants and children at-risk for developing autism. As discussed above, any successful biomarker test for early screening must predate the presence of autism symptoms, be specific to autism and limit false positives, as well as be independent of the behavioral measures currently used to diagnose the disorder. Finally, we must also address the issue that the biomarkers are likely to identify high risk for the presence or development of a disorder, but may not equate to definitive presence of a disorder as in Alzheimer's disease. This issue becomes of supreme importance for neurodevelopmental disorders where parents may have a child identified as atypical prior to birth. This could result in widespread changes for family planning, and significant medical or behavioral interventions during infancy even though we are not aware of the protective factors that may prevent behavioral symptoms from emerging. Ethical issues regarding the implementation and clinical recommendations from the outcomes of biomarker tests require greater examination than can be discussed here; however, future discussion and research into the use of biomarkers for screening behaviorally defined disorders are of high importance for researchers, clinicians, and families.