should be sent to R. A. Poldrack, Departments of Psychology and Neurobiology & Imaging Research Center, University of Texas at Austin, 3925-B W. Braker Ln., Austin, TX 78759. E-mail: email@example.com
The goal of cognitive neuroscience is to map mental functions onto their neural substrates. We argue here that this goal requires a formal approach to the characterization of mental processes, and we present one such approach by using ontologies to describe cognitive processes and their relations. Using a classifier analysis of data from the BrainMap database, we examine the concept of “cognitive control” to determine whether the proposed component processes in this domain are mapped to independent neural systems. These results show that some subcomponents can be uniquely classified, whereas others cannot, suggesting that these different components may vary in their ontological reality. We relate these concepts to the broader emerging field of phenomics, which aims to characterize cognitive phenotypes on a global scale.
Whereas the philosophical field of ontology is concerned with the nature of existence, ontologies in informatics are meant to formally specify the entities that exist in a domain and the relations between them (Bard, 2003; Bard & Rhee, 2004; Gruber, 1993). Such ontologies have become central within many areas of bioscience. For example, the Gene Ontology (http://www.geneontology.org) (Ashburner et al., 2000) (Fig. 1) was established originally to describe gene product roles/attributes (e.g., cellular components) for the fly (Drosophila), mouse, and yeast (Saccharomyces) organisms. The ontology includes definitions of each entity along with a specification of ontological relations between entities, which can include hierarchical relations such as “is-a” or “part-of” or spatiotemporal relations such as “preceded-by” or “contained-within.” This knowledge structure allows for a consistent representation of the cellular components of any cell, across organism models, which can facilitate communication between domains (i.e., by providing an objective, concise, common, and controlled vocabulary to compare gene product roles across organisms), as well as improving interoperability (i.e., providing links between levels of analysis). In the following sections we first demonstrate how an ontological approach can inform our understanding of executive control functions, and then provide an example of such an application.
2.1. Ontological principles and control functions
One goal of cognitive neuroscience is to “map” cognitive processes onto brain systems given the assumption that these processes are localizable to some pattern of brain activity. Put differently, this mapping can only be successful if the cognitive constructs being mapped to the brain are actually implemented in the brain as separate constructs. Thus, determining whether the cognitive ontology matches with the brain’s functional organization is a fundamental problem. Unfortunately, the elements of the mental ontology are not directly accessible but rather must be accessed through experimental manipulations and measurements (i.e., tasks). Thus, in order to be useful, an ontology of mental processes must also include mappings of the latent mental constructs to observable variables collected under specific manipulations (which we refer to generically as indicators). Fig. 2 provides an example of such an ontology for the domain of cognitive control.
Cognitive control is presented as consisting of four components: working memory, response selection, response inhibition, and task-set switching. These units form the vocabulary of cognitive control (Sabb et al., 2008). The ontology has been simplified such that no interactions exist between these units (cf. Fig. 1), and thus emphasizes their implied independence. For demonstration, beneath two of these units are listed the behavioral measures that may be associated with each, along with theoretical findings that may be expected for different instantiations (i.e., domains) of cognitive control. For instance, considering the “working memory” unit, we may quantify it by measuring accuracy on a delayed matching to sample task, the number of items remembered during a working memory span task (e.g., Sternberg task) or resistance in performance to interference during an n-back task (Conway et al., 2005; Kane, Conway, Miura, & Colflesh, 2007). Each ontological construct should be associated with a distinct (as of yet unknown) pattern of neural activity. As such, distinct indicators and brain activation patterns will exist for each of the entities.
Given such an ontology, how should we determine whether it accurately reflects the organization of mental processes in the brain? Foremost, we should expect that patterns of brain activity should be predictably different for task manipulations that are associated with putatively different processes, and they should overlap to the degree that the task manipulations are thought to engage the same processes. This highlights the fact that we cannot test hypotheses about mental processes without relying upon assumptions that link these processes to specific observable indicators. A correct ontological mapping from mental to neural processes should provide selectivity, in that the likelihood of activation of the region or network in question should be higher for the mental process in question than for other mental processes (cf. Poldrack, 2006). In sum, to the degree that the theoretical ontology presented in Fig. 2 is correct, then its entities (working memory, response inhibition, response selection, and task-set switching), relations (e.g., is-a-part-of-cognitive-control), and indicators (reaction time, span, SSRT, etc.) will be associated with patterns in brain activity that are selective and reproducible.
2.2. Informatics approaches to characterizing cognitive constructs of control
The first step to defining an ontology is specifying candidate entities. One fruitful approach to understanding the nature of mental constructs is to characterize their relations within existing text corpora, such as journal abstracts. While there are a number of obvious limitations to text mining, it is now well established that literature co-occurrence statistics can provide substantial insights into the semantics of a domain (Griffiths & Steyvers, 2004; Landauer & Dumais, 1977). In the present context text mining may serve the purpose of systematically and objectively identifying candidate constructs of control functions.
This approach was used recently by Sabb et al. (2008) to evaluate the relationship between estimates of heritability, behavioral measures, and component constructs of executive function. They examined over 478 articles in the PubMed database and applied a phrase search algorithm to isolate key terms most commonly used in cognitive neuroscience literature. Within this set, they then isolated a set of five terms that summarized the literature on executive function: “working memory” (WM), “response selection” (RS), “response inhibition” (RI), “task switching” (TS), and “cognitive control” (CC) (Fig. 3). The former four terms were selected based on their frequency of co-occurrence with the term “cognitive control.” These terms also showed high internal consistency for their indicators and associated heritability measures, suggesting that they may effectively capture distinct components of executive functions. However, Sabb et al. (2008) also found that the same indicators that are currently associated with the term “cognitive control” were historically associated with the other constructs, suggesting that they may overlap in other ways. In particular they may share neural systems, which we argue are critical in establishing distinct cognitive constructs.
To test for evidence of selective mapping of the proposed components of cognitive control onto neural systems, we performed a meta-analysis comparing patterns of reported brain activity across a range of cognitive tasks that were labeled as engaging one of these components. We reasoned that these constructs should show a good degree of selectivity in their neural representations if they capture distinct components of cognitive control (cf. Fig. 2). The analysis was conducted by using the labels identified by Sabb et al. (2008) as the basis for a set of queries against the BrainMap database (Laird, Lancaster, & Fox, 2005). As a comparison and sanity check to verify that this method can yield significant results, we also ran a query for papers labeled as “bilingual” (BI) within BrainMap. The motivation for this approach was that while these tasks may engage overlapping networks with experiments probing aspects of cognitive control, they should also engage those regions more uniquely associated with language processing or production (e.g., left-lateralized temporal and inferior frontal cortices). If the classification method is sound, then the ability to classify between the cognitive control constructs and the bilingual language construct should provide an upper bound for the quality of the classifier given these summary BrainMap data sets.
First, a search was run for each of the labels (Table 1), and the peak coordinate voxels stored in BrainMap describing each contrast were then converted from Talairach to MNI coordinates using GingerALE (Laird, Fox et al., 2005). These points were then projected onto a lateralized version of the Harvard–Oxford atlas (distributed as part of the FSL MRI image analysis toolset, Smith et al., 2004), a probabilistic labeled atlas with 114 unique regions of interest (ROI). Using this atlas provides a means of describing reported brain activity for a given contrast as anatomic regions rather than stereotactic coordinates. Using this atlas, each voxel was converted from a three-dimensional coordinate (x, y, z) to a 114-dimensional vector: each of the 114 values represented a probabilistically weighted count of the voxel falling into the corresponding anatomical region. Of course, any given voxel will only probabilistically occupy a small number of anatomic regions; so each of these vectors will necessarily be sparse in nature. Within each contrast, the set of vectors for all reported voxels was summed to generate one vector representing that contrast. The summation thus resulted in something akin to a weighted count “score” reflecting the degree to which a given region was active in a contrast.
Table 1. Contrast and paper frequencies for the meta-analysis (BrainMap database ontological label search results)
Note. The number of contrasts returned by the BrainMap database for a given label. The number of unique papers contributing to those search results is given in parentheses.
The result of this process was a set of vectors in “ROI” space, one for each reported contrast and labeled as belonging to a specific ontological construct. To assess discriminability between constructs, we then performed a classification analysis on these vectors using a k-nearest neighbor classifier with “leave one out” cross-validation (neighborhood parameter, k = 3). The idea behind this technique is to, for a pair of constructs: (a) identify a representative or “centroid” ROI vector for each construct using all but one exemplars, and (b) determine to which of the construct vectors the left out exemplar is closest. A match is interpreted as a successful classification. This was done for each possible pairing of constructs and classification accuracy was assessed. However, as there were an unequal number of vectors representing each construct, the raw accuracy measures can be misleading; classifiers are often sensitive to differences in base rates even when there are no true differences to classify. To assess classification sensitivity independent of bias, A′ (a-prime) statistics were calculated. This index provides a measure of detection sensitivity, as a function of successful classifications versus false alarms, that is, free from any assumptions regarding underlying distributions. An A′ value of .5 indicates chance performance. To assess the significance of any given A′ value, we ran a permutation test randomly reassigning labels to the contrast values being classified and observing the resulting A′ statistic.
Fig. 4 (lower diagonal) shows the classification accuracy for this analysis. All classifiers comparing to the BI data set were highly significant (A′ > .74, p <.05), suggesting that this method is capable of discriminating patterns of activity. Within the set of conceptual labels describing aspects of cognitive control, classifiers discriminating between RS and CC, RI or WM performed remarkably well (A′ > .75, p <.05). Classifiers discriminating the latter three were more ambiguous (A′ = .63, p <.15), suggesting the presence of neural commonalities. Finally, TS lacked any real discriminability with RS (A′ = .49, p >.15) and showed only trend classifications with RI or CC. Examples of regions that were most active in discriminating between pairs of constructs are shown in Fig. 4 (upper diagonal). These images were created by taking the difference between mean activation patterns for each construct within the pair, normalized by their standard deviations.
To aid interpretation, in Fig. 5, we present the discriminating regions for each construct taken from the average volume of all its pairwise classifications (e.g., Fig. 4). Thus, these images represent regions that were systematically involved in discriminating a construct from all other constructs. From this image we see that classifier performance is roughly proportional to similarity of classification maps. Classification maps that were distinct were also well classified and the converse. As such, regions associated with BI, on average, were left lateralized and included temporal pole and inferior frontal cortex (Fig. 5, column 1, A′ > .74, p <.05), two regions that were not involved in discriminating other constructs. Similarly, RS (Fig. 5 column 4) was discriminable from CC, RI, and WM (A′ > .75, p <.05). Unlike the latter, it showed involvement of bilateral precentral cortex and middle frontal gyrus. In parallel to this finding, CC, RI, and WM all showed involvement of right frontal pole, right pallidum, and right caudate nucleus (A′ = .63, p <.15) (Fig. 5, columns 2, 3, and 6). Interestingly, we note that TS, which showed a classification map that appeared similar to RS (Fig. 5, columns 4 and 5), lacked significant classification regarding the latter (A′ = .49, p > .15).
3. Learning from ontologies
The results of our analysis show that for several of the constructs identified by Sabb et al. (2008), including BI versus others, and RS versus CC, RI, and WM, it was possible to accurately classify studies based on the pattern of brain activity. For others (CC, RI, and WM) classification was more ambiguous. And for others still (TS) classification was inconsistent. Before interpreting these results, we acknowledge that our method is noisy by its very nature. The search results are a sample of complete literature that happens to be indexed within BrainMap. Different individual curators have applied the ontological labels categorizing these data. There is also high variability in the subject populations, scanning equipment, statistical methods, and thresholds that went into producing these data. Finally, we note that the present approach does not explicitly evaluate relations between constructs (e.g., is-part-of) that are naturally part of ontologies (cf. Section 2). Nonetheless, the analysis provides substantial predictive power. The fact that several of the components of cognitive control are nearly as discriminable from one another as from a completely different construct (bilingual language processing) provides initial evidence for their ontological reality.
3.1. Interpreting classification
When constructs appear to have met selectivity criteria, we have arrived at an educated guess of which constructs comprise the core entities of control. In our demonstration we found that brain activation reported in conjunction with the label RS was distinguishable from that reported in conjunction with CC, RI, or WM. Based on this result we may conclude that RS represents one distinct control function, and thus a distinct entity in the ontology of control functions (Fig. 2A vs. B). Based on its average discrimination map (Fig. 5, column 4), we may also conclude that precentral gyrus and middle frontal gyrus are on average associated with discrimination of this construct across all pairwise classifications. Similarly, CC, RI, and WM may in their similarities represent another distinct control function, with a classification map that includes a right-lateralized network involving frontal and subcortical regions. The finding of overlap with CC is not surprising given that the other constructs were identified based on their co-occurrence with these terms. However, the similarity between RI and WM may suggest that these constructs share neural systems and may thus be part of the same control function.
These conclusions are of course limited by the fact that construct labels are only as good as the mappings between constructs and tasks. For instance, it may be that RS and CC, RI and WM are mostly overlapping constructs, but the tasks used to study them vary slightly in their processing demands. These differences may lead to subtle differences that, nevertheless, are classified successfully. In the case of a classification failure, inconsistent use of terminology across literature may be the culprit. For instance “working memory” may be used interchangeably with “central executive” which may also pop up in the discussion of RI. A related problem is that these terms may not be consistently or distinctly operationalized across tasks. For instance, many tasks that include task switching will also include response inhibition (or response selection), although the latter may not be discussed or uniquely measured. This fact may have contributed to the lack of discrimination between TS and RS or RI.
An advantage of our approach, however, is that, in the long term, it is capable of discovering such inconsistencies provided that new paradigms are eventually introduced into the database. A superficial distinction between RS and CC, RI and WM would be expected to disappear as new versions of tasks, or unique contrasts between conditions, are introduced into the database. In principle, the neural patterns associated with the discrimination of this construct should be reliable across contrasts. The effect of task-specific bias may be further attenuated by the fact that novel, multivariate analysis methods are entering the mainstream of neuroimaging research. Another benefit of our method is that, given a large enough database, one may analytically evaluate how much different tasks contribute to the classification of a given construct (in analog to how brain regions contributed to pairwise classifications). Therefore, the effect of outliers may be evaluated in both successful and failed classifications.
3.2. Discovering new cognitive constructs
Because of the problems that task validity imposes onto interpretation, it is important to recognize that the initial constructs used to define an ontology may not be the right ones. They may not have systematic representations in brain activity and so classification may be inconsistent (e.g., TS in Fig. 4). This may be because multiple control functions are being elicited, some of which may have systematic patterns of neural activations that do not have a well-defined cognitive label. In this way our selection of cognitive constructs subject to classification may very well be incomplete. This is an important problem and one that has been addressed previously. Price and Friston (2005) have argued that we may discover new cognitive constructs by clustering patterns of neural activation first and inferring a cognitive label second. In their example they re-examined the cognitive labels that were associated with left posterior lateral fusiform gyrus activity to conclude that this region is most likely involved in sensorimotor integration. This new label succinctly replaced a number of previously hypothesized functions (e.g., word form recognition, visuo-tactile integration, and visual attribute recognition) and thus should improve the classification success for this brain–behavior mapping.
Similarly, there may exist core constructs of executive function that have not yet been defined in behavioral research but which may be clear at the neural level. For instance, recent reports of neural specialization within subregions of prefrontal cortex include tracking of reward-object associations in orbital frontal cortex (Rudebeck et al., 2008; Walton, Rudebeck, Bannerman, & Rushworth, 2007) and temporal sequencing of motor commands in superior frontal cortex (Mushiake, Inase, & Tanji, 1990; Saito, Mushiake, Sakamoto, Itoyama, & Tanji, 2005) (see Fig. 2B). A step forward toward validating these as core functions from a neuroinformatics perspective would be to first identify all studies that report either activation in orbital frontal cortex or that in superior frontal cortex, and then apply a text-mining algorithm to identify the common cognitive labels described in these studies. If the conclusions based on neuronal recordings are correct, then the most systematically reported cognitive labels in these studies will include reward processing and motor sequencing. These constructs can then serve as new candidate labels for component control functions ready for classification.
4. Interoperability and phenomics
The additional utility of ontologies is that they facilitate interoperability, simultaneously safeguarding against the pitfalls of single-level analysis. First, ontologies afford more direct communication across domains of research. For example, using the Gene Ontology for cellular components a researcher who works with Drosophila can directly compare their findings on “ribosome” or “nuclear membrane” or “Golgi apparatus” properties with a mouse researcher because of the well-defined shared vocabulary. The utility of an ontology of control functions would be to similarly facilitate communication among researchers in various domains of control function, such as in neuropsychiatric disorders, neural development, and cross-population comparisons. Second, a shared vocabulary increases communication across levels of analysis. Again returning to the gene ontology, multiple ontologies have been created to accommodate various aspects of protein function. There are distinct ontologies available to describe molecular function (e.g., nucleic acid binding) as well as DNA metabolism (e.g., DNA packaging and replication) in addition to cellular components (cf. Fig. 1). As a result, changes that are observed across domains can be simultaneously observed and compared with those observed across levels of analysis. The utility of an ontology of control functions would be to facilitate communication between researchers across scales of analysis, such that the construct of response inhibition, for example, could be approached similarly from the behavioral through neural level. More importantly, some aspects of cognition may be identifiable at one level of analysis but not at another. For instance, some aspects of working memory may be better captured by the firing patterns of neurons or cAMP activation, and not in contrasts of neuroimaging measures as presented in the BrainMap database, the subject of our analysis. The ontological approach allows discovery of such discrepancies.
The proposed method for the study of control functions is directly compatible with the phenomics approach, the systematic study of phenotypes on a genome-wide scale. It is increasingly recognized that the rate-limiting step in identifying genetic influences of complex neurobehavioral phenotypes is not our ability to process and characterize genomic data but rather our ability to define and reliably measure phenotypes on a large scale with high throughputs. As such, the essential goal of phenomics is to determine what constitutes a phenotype and define the complete set of phenotypes of a given species (Freimer & Sabatti, 2003). One of the key goals of phenomics research is to overcome not only limitations of diagnostic classification systems for psychiatric disorders but also the lack of consistent and reliable neurocognitive measures. This rests on the ability to define and measure quantitative neuropsychiatric phenotypes that can be examined across disciplines and across species, and these goals are critically dependent on communication across levels of analysis. It has been suggested that the key step is to reframe current conceptualizations of cognitive constructs, particularly in terms of their neural bases, in order to effectively bridge disciplines and research across species (Bilder et al., 2009; Freimer & Sabatti, 2003). A successful example of this multilevel approach has been recently presented by Sabb et al. (2009) for the domains of “intelligence” and “declarative memory.” Undoubtedly, a similar multilevel model would be greatly beneficial in clarifying the core function of control.
5. Concluding remarks
One aim of this issue was to evaluate the plausibility of the existence of distinct control functions against the possibility that control functions are emergent. From the perspective of cognitive neuroscience we argued that one potential answer to this question lies in the degree to which such control functions are systematically associated with brain activity. Our demonstration showed evidence that some of these constructs may be ontologically distinct, as reflected in the ability to accurately classify them on the basis of brain imaging data. It also highlighted that the specification of control functions may require systematic analysis of neural activation in particular regions or across networks; novel conceptualizations of control may emerge from the properties of neural activation.
In sum, control functions that are typically associated with control (e.g., inhibition, task setting, and updating) are defined largely based on behavioral observations (Miyake, Friedman, Emerson, Witzki, & Howerter, 2000; Miyake, Friedman, Emerson, Witzki, Howerter, & Wager, 2000). How these functions correspond to distinct indicators of brain activity will determine whether they are truly distinct componential entities or whether they emerge from the interactions of various systems in the brain and are therefore manifest only in the minds of cognitive scientists.