The author and his work are supported by the Deutsche Forschungsgemeinschaft (DFG) and the National Institutes of Health (NIH). The editor, Bob Simons, made invaluable suggestions and corrections at all stages of the manuscript's preparation. The comments of two anonymous reviewers, of Many Donchin, Andrea Kübler, Theresa Vaughan, and Jon Wolpaw are greatly appreciated. The data from my laboratory presented here could not have been realized without a functioning interdisciplinary research team: the names of the team members appear in the references cited. They deserve all the credit for this work. The manuscript was prepared during my stay as a research fellow at the NIH, NINDS, Bethesda, MD: My friend Leonardo Cohen, M.D., Chief of the Human Cortical Physiology Section at NINDS, and Cornelia Weber made the time in Washington, DC, a unique and productive experience.
Address reprint requests to: Niels Birbaumer, Ph.D., Institute of Medical Psychology and Behavioral Neurobiology, MEG-Center, University of Tübingen, Gartenstrasse 29, D-72074 Tübingen, Germany. E-mail: niels.birbaumer@uni-Tuebingen.de.
Brain–computer interfaces (BCI) allow control of computers or external devices with regulation of brain activity alone. Invasive BCIs, almost exclusively investigated in animal models using implanted electrodes in brain tissue, and noninvasive BCIs using electrophysiological recordings in humans are described. Clinical applications were reserved with few exceptions for the noninvasive approach: communication with the completely paralyzed and locked-in syndrome with slow cortical potentials, sensorimotor rhythm and P300, and restoration of movement and cortical reorganization in high spinal cord lesions and chronic stroke. It was demonstrated that noninvasive EEG-based BCIs allow brain-derived communication in paralyzed and locked-in patients but not in completely locked-in patients. At present no firm conclusion about the clinical utility of BCI for the control of voluntary movement can be made. Invasive multielectrode BCIs in otherwise healthy animals allowed execution of reaching, grasping, and force variations based on spike patterns and extracellular field potentials. The newly developed fMRI-BCIs and NIRS-BCIs, like EEG BCIs, offer promise for the learned regulation of emotional disorders and also disorders of young children.
A brain–computer interface (BCI) or brain–machine interface (BMI) activates electronic or mechanical devices with brain activity alone. BCIs and BMIs allow direct brain communication in completely paralyzed patients and restoration of movement in paralyzed limbs through the transmission of brain signals to the muscles or to external prosthetic devices. We differentiate invasive from noninvasive BCIs: Invasive BCIs use activity recorded by brain implanted micro- or macroelectrodes, whereas noninvasive BCIs use brain signals recorded with sensors outside the body boundaries.
This article reviews the research concerned with invasive and noninvasive BCIs from the perspective of their clinical usefulness for communication and motor restauration in paralysis. The reviews available on invasive BCI in animals (Nicolelis, 2003; Nicolelis, Birbaumer & Mueller, 2004; Schwartz, Taylor, & Tillery, 2001) describe primarily the performance of single neuronal unit resonse patterns for the reconstruction of movement sequences in healthy animals; if they discuss clinical applications in human patients at all, a science fiction perspective of what may be possible is given without reference to the few published clinical applications. The noninvasive BCI literature overviews (Kübler, Kotchoubey, Kaiser, Wolpaw, & Birbaumer, 2001; Wolpaw et al., 2002) were based on small numbers of clinical cases; meanwhile the database for BCI research in clinical populations has broadened and allows some tentative theoretical and clinical conclusions not available in previous reviews. Remarkably, the clinical applications, particular those of BCIs for communication in completely paralyzed patients, allow a fresh view on some old and still unresolved theoretical questions in psychophysiology:
1What is the role of voluntary motor control and of the feedback following motor responses in goal directed thinking and imagery and verbal behavior?
2What are the consequences of a loss of complete or virtually complete loss of motor behavior on emotional responding at the subjective and the physiological level?
3What is the nature and extent of brain reorganization after complete cessation of voluntary motor response systems? What are the consequences of compensatory brain reorganization on behavior?
This review addresses these questions in the context of BCI research and tries to illustrate once again the usefulness of a union between clinical and experimental approaches in psychophysiology for the reformulation of some basic scientific problems in the field.
History of BCI Research
Hans Berger, who discovered the human EEG, speculated in his first comprehensive review of his experiments with the “Elektrenkephalogramm” (1929) about the possibility of reading thoughts from the EEG traces by using sophisticated mathematical analyses. Grey Walter, the brilliant EEG pioneer who described the contingent negative variation (CNV), often called the “expectancy wave,” built the first automatic frequency analyzer and the computer of “average transients” with the intention of discriminating covert thoughts and language in the human EEG (Walter, 1964). Fetz (1969) published the first paper on invasive operant conditioning of cortical spike trains in animals. Only the recent development of BCIs, however, has brought us a bit closer to the dreams of these pioneers of EEG research.
Invasive and noninvasive BCIs originate from different research traditions, though both have their roots in animal experiments. Invasive BCIs consist of implanted multielectrode grids in the motor cortex of paralyzed patients (Donoghue, 2002), premotor cortex of monkeys (Carmena et al., 2003), or parietal motor command areas (Schwartz et al., 2001). They try to reconstruct intended skilled movements from neuronal firing patterns online. Based on “sparse coding” approaches to motor learning (Riehle & Vaadia, 2005) and directional coding vectors of motor neurons (Georgopoulos, Schwartz, & Kettner, 1986), automatized complex movements can be reconstructed online from relatively few motor neurons using simple algorithms: Nicolelis' group (Carmena et al., 2003) demonstrated in monkeys after extensive training of a reaching and grasping movement that firing patterns of 32 neurons are sufficient to execute that movement directly with an artificial limb. Chapin, Moxon, Markowitz, and Nicolelis (1999) trained rats to move a lever with an artificial arm in a Skinner box for reward with extracellular firing of cortical cells without any actual movement. The neuronal firing pattern that used to precede and accompany the lever pressing response alone was able to operate on the lever delivering the reward.
Operant Conditioning of Autonomic Functions
The second root of BCI research is intimately tied to the tradition of biofeedback and instrumental-operant learning of autonomic functions. During the late 1960s and early 1970s, Neal E. Miller and collaborators opposed the traditional wisdom of the autonomous nervous system (ANS) as autonomous and independent of voluntary control of the somatic central nervous system (CNS). Miller (1969), in a landmark paper in Science, challenged that view that voluntary control is acquired through operant (instrumental) conditioning whereas modification of involuntary ANS functions is learned through classical (Pavlovian) conditioning, a distinction first emphasized by Skinner (1953; Holland & Skinner, 1961).
Miller presented experimental evidence in curarized and artificially ventilated rats showing that even after long-term curarization of several weeks, the animals learned to increase and decrease heart rate, renal blood flow, and dilation and constriction of peripheral arteries in an operant conditioning paradigm rewarding the animals for increases and decreases of these specific physiological functions. These studies stirred an enormous interest in the scientific and clinical community, particularly in psychosomatic medicine and behavior modification.
The results suggested that instrumental (“voluntary”) control of autonomic functions is possible without any mediation of the somatic-muscular system. Operant training of any internal body function seemed possible, opening the door for psychological and learning treatment of many medical diseases such as high blood pressure, cardiac arrhythmias, vascular pathologies, renal failure, gastrointestinal disorders, and many others. In the clinic, biofeedback of these functions replaced the operant conditioning in rats, the feedback from the specific physiological variable constituted the reward (for an overview of these years' enthusiasm, see the Aldine series on Biofeedback and Self-Control; Kamiya, 1971).
During the next two decades, Miller and his students at Rockefeller University tried to replicate their own findings. Figure 1 shows the steady decline of the size of the conditioning effect with each replication. Finally, by the mid-1980s, it was impossible to replicate the previous effects. Barry Dworkin, Neal Miller's last and most prolific student, continued to try and build the most sophisticated “intensive care unit” for curarized rats, but again, operant training of autonomic function or nerves in the curarized rat was impossible.
In contrast, classical conditioning succeeded even in single facial nerve fibers (Dworkin, 1993; Dworkin & Miller, 1986). Dworkin attributed the failure of operant techniques to the missing homeostatic effect of the reward: The reward acquires its positive effect through homeostasis-restoring effects (i.e., ingestion of food restores glucostatic and fluid balance). In the curarized rat (and the completely paralyzed respirated and fed patient?), where all body functions are kept artificially constant, the homeostatic function of the reward is no longer present because imbalances of the equilibrium do not occur.
The chronically curarized rat and the completely paralyzed, artificially ventilated and fed locked-in patient share many similarities; difficulties in communicating with these patients may be understood based on these similarities.
The most spectacular and popularized results in the emerging field of biofeedback (or “physiological regulation” as it is presently called) were the self-regulation of brain waves (Kamiya, 1971). Increase and decrease of alpha frequency of the EEG were supposed to create “meditative” states with many beneficial effects in the periphery and on behavior. Theta wave augmentation and reduction had profound effects on vigilance and attention (Birbaumer, 1977). Slow cortical potentials (SCP) control allowed anatomically specific voluntary regulation of different brain areas with area specific effects on behavior and cognition (for an overview, see Rockstroh, Elbert, Birbaumer, & Lutzenberger, 1989). Warning voices such as experiments by Mulholland and his group (Mullholland & Evans, 1966) demonstrating perfect control of alpha waves through manipulation of the oculomotor system and decoupling of eye fixation went largely unheard.
Sterman (Sterman, 1981; Sterman & Friar, 1972) was the first to propose self-control of epileptic seizures (Elbert et al., 1984) by an augmentation of sensorimotor rhythm (SMR). SMR in human subjects is recorded exclusively over sensorimotor areas with frequencies of 10 to 20 Hz and variable amplitudes. Pfurtscheller and colleagues (2005) localized the source of human SMR in the sensorimotor regions following the homuncular organization of the motor and somatosensory cortical strip. Imagery of hand movement abolishes SMR over the hand region; imagery or actual movement of the legs blocks SMR in the interhemispheric sulcus. Pfurtscheller called this phenomenon event-related desynchronization and synchronization (Pfurtscheller et al., 2005).
On the basis of careful animal experiments (Sterman and Clemente, 1962a, 1962b), Sterman demonstrated incompatibility of seizures in motor and premotor areas in the presence of SMR. Cats exhibited maximum SMR during motor inhibition and various sleep stages. Presence of spindles during different sleep stages, particularly during rapid eye movement (REM) sleep indicated recruitment of inhibitory thalamo-cortical circuits and blocked experimentally induced seizures. Sleep spindles and SMR share identical physiological mechanisms. Epileptic cats and humans were trained to increase SMR, and, after extensive training ranging from 20 to more than 100 sessions, Sterman (1977) was able to demonstrate seizure reduction and complete remission in some patients with drug-resistant epilepsy. It is important to note that SMR is often called mu-rhythm following a suggestion of Gastaut (Gastaut, 1952; Gastaut, Terzian, & Gastaut, 1952) who noted its abolition in some types of seizures. However, it is not clear whether the neurophysiological bases of the two phenomena are really comparable and therefore I recommend that the term SMR as used by Sterman et al. be retained because of its well-defined theoretical and experimental background.
It is not accidental that SMR operant control is achieved through activation and deactivation of the central motor loops. Again, successful voluntary regulation of a physiological variable is tied to the regulation of the motor system. The results of SMR control in animals and patients seem to demonstrate that manipulation (mediation) of the peripheral motor efferents is not a necessary requirement of SMR control, at least on the basis of EMG recordings of the arm muscles showing no measurable variation during motor imagery with central nervous system event-related desynchronization (Pfurtscheller et al., 2005). The successful brain regulation of SMR in completely paralyzed patients reported below confirms that changes of the peripheral motor system do not mediate CNS activity responsible for SMR origin. The notion of the critical role of CNS activity in voluntary action and thought remains.
Beginning in 1979, our laboratory published an extensive series of experiments that demonstrated operant control of slow cortical potentials in the EEG. These demonstrations differed from previous brain biofeedback work as they documented the following in well-controlled experimental paradigms:
1Strong and anatomically specific effects of self-induced cortical changes on behavior and cognition;
Of particular interest in the context of CNS motor mediation of voluntary control of brain activity was the fact that SCPs originating from posterior parietal sources were resistant to operant learning whereas central and frontal SCPs could be brought under voluntary, operant control after one to five training sessions (Lutzenberger, Roberts, & Birbaumer, 1993). Several clinical studies confirmed the critical importance of the anterior brain systems for physiological regulation of CNS functions: Lutzenberger et al. (1980) showed that patients with extended prefrontal lobe lesions were unable to learn SCP control despite intact intellectual functioning. Disorders with prefrontal dysfunctions such as attention deficit disorder (ADD; Birbaumer, Elbert, Rockstroh, & Lutzenberger, 1986) and schizophrenia (Schneider et al., 1992) exhibited extreme difficulties in acquiring SCP control, and attentional improvement after SCP or SMR neurofeedback training required long training periods (Strehl, Leins, Goth, Klinger, & Birbaumer, in press). Again, peripheral motor function played no role in SCP conditioning (Birbaumer & Kimmel, 1979), but intact prefrontal systems seemed to be a prerequisite for successful brain control. Figure 2 shows the results of a study where healthy subjects learned SCP control, and fMRI (BOLD response) was recorded simultaneously during training.
Subjects received visual feedback of positive and negative SCPs of 6 s duration and were rewarded for the production of target amplitudes (Hinterberger et al., 2004; Hinterberger, Birbaumer, & Flor, 2005; Hinterberger, Veit, et al., 2005). As illustrated in Figure 2, successful voluntary brain control depends on activity in premotor areas and the anterior parts of the basal ganglia. Birbaumer et al. (1990) had proposed earlier that physiological regulation of SCP and attention depends critically on anterior basal ganglia activity regulating local cortical activation thresholds and SCP in selective attention and motor preparation. Braitenberg (Braitenberg & Schüz, 1991) created the term “thought pump” (“Gedankenpumpe” in German) for this basal ganglia–thalamus–cortical loop. Taken together, the extensive literature on the SCP also suggests that operant-voluntary control of local cortical excitation thresholds underlying goal-directed thinking and preparation depends on an intact motor or/and premotor cortical and subcortical system.
Encouraged by the reliable and lasting effects of brain self-regulation on various behavioral variables and by Sterman's case demonstrations, Birbaumer and colleagues conducted several controlled clinical studies on the effect of SCP regulation on intractable epilepsy (Kotchoubey et al., 2001; Rockstroh et al., 1989, 1993). Based on their neurophysiological model of SCP regulation, patients with focal epileptic seizures were trained to down-regulate cortical excitation by rewarding them for cortical positive potentials and perception of SCP changes. After extremely long training periods, some of these patients gained close to 100% control of their SCPs and seizure suppression, tempting Birbaumer and colleagues to apply cortical regulation as a BCI for paralyzed patients: Given that epileptic patients suffering from a dysregulation of cortical excitation and inhibition and consequent brain lesions learn to control their brain responses both within the laboratory and in daily life, it is not unreasonable to ask whether a paralyzed patient could learn to activate an external device or computer in order to move a prosthetic arm or to convey messages to a voice system.
Noninvasive BCIs for Communication in Paralysis
Amyotrophic Lateral Sclerosis (ALS) is a progressive motor disease of unknown etiology resulting in a complete destruction of the peripheral and central motor system but only affecting sensory or cognitive functions to a minor degree (Norris, 1992). There is no treatment available; patients have to decide to accept artificial respiration and feeding after the disease destroys respiratory and bulbar functions for the rest of their life or to die of respiratory problems. If they opt for life and accept artificial respiration, the disease progresses until the patient loses control of the last muscular response, which is usually the eye muscle or the external sphincter. The resulting condition is called completely locked-in state (CLIS). If rudimentary control of at least one muscle is present, we speak of a locked-in state (LIS). Other conditions leading to a locked-in state are subcortical stroke and other extended brain lesions, Guillain-Barre syndrome, some rare cases of Parkinson disease, and Multiple Sclerosis.
The SCP-BCI needs long training periods, sometimes months, in the home of the patient (all patients were artificially respirated and paralyzed), and letter selection speed is slow, usually one minute per letter. However, speed is not an issue in artificially respirated paralyzed patients devoting all their cognitive and emotional energies to communication (Birbaumer, Strehl, & Hinterberger, 2004). The SCP-BCI needs professional attention and continuous technical support; easy application by family members or nonprofessional caretakers was possible in only one patient.
Wolpaw and colleagues at the Wadsworth Laboratories at Albany, New York, did an extensive series of experiments mainly with healthy persons using SMR rather than SCP as the target brain response (Wolpaw et al., 2002). In a group of patients, two with high spinal cord lesions, Wolpaw and McFarland (2004) demonstrated that multidimensional control of a cursor movement on a computer screen can be learned in just a few sessions of training: The subjects were able to move a cursor within 10 s into one of eight goals appearing randomly at one of the four corners of the screen. The flexibility, speed, and learning performance is generally equal to that seen when invasive multielectrode BMI systems are tested in animals. The Wolpaw and McFarland (2004) preparation consisted of a simple electrode montage covering the hand and foot area with a linear online filtering and detection algorithm used for data reduction and quantification. Most subjects employed right and left hand and feet imagery to reach the target goals in SMR-BCI.
The Albany and Tübingen group joined forces in an NIH-funded project and compared the feasibility and performance of the SCP-BCI, the SMR-BCI, and the P300-BCI developed by Farwell and Donchin (1988) in seven pre-LIS ALS patients in a balanced within-subject design. The results were clear-cut: All patients achieved sufficient performance rates (more than 70% of the trials correct) after 20 sessions with SMR-BCI training, four of the seven could spell with the P300-BCI, but none of the patients achieved acceptable performance rates with the SCP-BCI despite significant differentiation between negative and positive SCP. It can be concluded that in ALS patients with functioning vision and eye control, SMR-BCI and P300-BCI shows the most promising results. The project continues to follow these patients into complete paralysis and eventually into the complete locked-in state. Figure 3 gives examples of the training situations for the three BCIs.
SCP-BCIs need more extensive training than other BCI modes but may have the best stability and independence of sensory, motor, and cognitive functioning necessary for its application to the LIS and the CLIS patients. The patients described earlier (Birbaumer et al., 1999) had high success rates with SCP-BCI training but only after many more sessions.
Together with the introduction of controlled clinical trials to document comparative BCI performance, the Albany–Tübingen group created a Web site, BCI 2000 (http://www.bciresearch.org/BCI2000/bci2000.html; Schalk, McFarland, Hinterberger, Birbaumer, & Wolpaw, 2004) providing free software modules for BCI applications in research and clinic. More than 100 laboratories are now regular contributors to the BCI 2000 Web site, improving both the hardware and software modules. The aim is an inexpensive, FDA and CE approved, easy-to-use, universal, noninvasive BCI that will allow SCP, SMR, P300, and other possible oscillatory brain activities (i.e., gamma band in ECoG) in a world wide net of participants whose data collection and analysis will contribute to the continuous improvement and validity of BCI applications.
Long training periods, noisy signals, the continuous professional attention necessary, slow spelling speed, electrode and skin problems with long recording times, and the controlled attention focus during spelling makes the invasive BCI approach an attractive alternative, at least at a theoretical level.
Invasive BCIs for Communication
Kennedy, Kirby, Moore, King, and Mallory (2004) published several single cases with ALS in different stages (none either LIS or CLIS), with a cortically implanted glass microelectrode filled with a neurotrophic growth factor. The axon of the cell targeted by the electrode grows into it and allows recording of the spike activity. Some of the patients learned to spell using the spike activity mainly by turning it on and off in a “yes” or “no” fashion. From the published material, it is difficult to judge the usefulness of this preparation because death and medical complications interrupted communication in several cases (one case reportedly used the device on a more continuous basis). None of the patients were in urgent need of the device because all had rudimentary motor control.
More than 100 scientists attending the 2005 BCI conference in Rennselearville, New York, were asked for their opinion on the future of BCI applications. The majority of the BCI researchers present at the conference believed that the noninvasive BCI showed the most promise for development during the next decade. The main argument against noninvasive BCIs was their limited capacity to represent more than two signal alternatives (“yes,”“no,”“select,”“ignore,” etc.), and this limitation would prohibit their use for motor restoration or motor neuroprosthesis applications (Carmena et al., 2003; Taylor, Tillery, & Schwartz, 2002). This argument was recently countered experimentally by Wolpaw and McFarland (2004), who demonstrated two-dimensional cursor control over the sensorimotor rhythm of the scalp EEG. Even high-level motor control of complex movements combined with sophisticated prosthesis design can be exerted with a two-dimensional command system. In earlier papers by Elbert et al. (summarized in Birbaumer et al., 1990), healthy participants were trained to produce differential frontal, central, parietal, and left-right hemispheric negative and positive slow cortical potential shifts, allowing them at least several degrees of freedom for cursor or prosthesis control (see Birbaumer at al., 1990, for a review).
A further argument against widespread use of noninvasive BCIs for motor control and communication consists of the long training periods required and the high error rates that are observed even after extensive training. Patients often need weeks to learn to produce a particular brain response voluntarily in order to select letters or words reliably above chance. Although healthy persons may achieve brain control within one or two sessions, patients need a minimum of 20 sessions to achieve more than 70% correct selections at least with sensorimotor rhythm or slow cortical potentials (Kübler, Nijboer, et al., 2005). The incorporation of more sophisticated algorithms for EEG classification did not improve the situation substantially (Hinterberger, Kübler, Kaiser, Neumann, & Birbaumer, 2003; see results of the BCI competition in the IEEE Transactions in Biomedical Engineering; Nicolelis et al., 2004). Papers by Hinterberger, Veit, et al. (2003) and Piccione et al. (2006) illustrate this point nicely; they report equivalent results for BCI control with different classification algorithms (Hill et al., in press).
In humans, there are two published reports, in addition to the already mentioned attempts by Pfurtscheller's group, on invasive BCIs with epileptic patients. In these experiments, subdural macroelectrodes were implanted over frontal regions, and patients attempted spelling or they performed imagery tasks (Lal et al., 2005; Leuthardt et al., 2004). In a single session with these patients, it was possible to differentiate imagination of hand, tongue, and mouth movement using the ECoG. Figure 4 shows the perfect nonoverlapping classification of hand and tongue movements at the sensorimotor cortex (Support Vector Machines, SVM, were used as classification algorithms; see Lal et al., 2004; Schröder et al., in press), allowing the patient to select letters at a speed of several letters per minute after a 20-min training session. Patients spelled by selecting letters with imagery of finger movement (green field at cortex in Figure 4) and rejecting a letter by imagery of tongue movement (red field at cortex of Figure 4).
This indicates, not surprisingly, that with subdurally implanted macroelectrodes, degrees of freedom, precision of classification, and success rates may substantially improve. The first implantation of 100 microelectrodes in the motor cortex of a high spinal cord patient by Donoghue et al. (personal communication) and Hochberg, Mukand, Polykoff, Friehs, and Donoghue (2005) seems to allow improved BCI performance. However, of 17 ALS patients in our sample, all in the final stage of the disease and all artificially respirated and fed, only 1 agreed to implantation of subdural macroelectrodes (Wilhelm et al., 2006). Even when informed about the possibilities and advantages of the surgical implantation, 16 patients refused the procedure and preferred the slow and error-prone noninvasive device. An important argument of patients was that time is not an issue if one is completely paralyzed (Birbaumer et al., 1999, 2004; Kübler et al., 2003; Kübler, Nijboer, et al., 2005).
It is fair to conclude, therefore, that noninvasive BCIs using different types of EEG signals such as slow cortical potentials, P300, or SMR oscillations at present are and will remain the method of choice for communication in paralyzed and hopefully also in completely locked-in patients with ALS and other debilitating neurological diseases (subcortical stroke, Guillain Barré, extensive brain damage). If patients, their families, and the local ethical committees agree, implantations of micro- or macroelectrodes subdurally or in brain tissue should be considered. However, the database of invasive BCIs for communication purposes in paralyzed patients at present is too small to judge their efficacy, and the willingness of patients and their families to agree to implantation is weak as long as the noninvasive BCIs are available and functioning. The slow spelling speed and high error rate (even in highly trained patients rarely above 80% trials correct) of noninvasive EEG-based BCIs is well tolerated by paralyzed patients with a different life perspective and an urgent need to communicate.
Operant Learning, Thinking and BCI Control in the Complete Locked-in State
As mentioned above, none of the ALS patients starting BCI training after entering the complete locked-in state acquired stabile communication (n=17). Again, one of these patients was implanted with subdural electrodes over the left frontal cortex. Despite clean ECoG recordings and extensive learning attempts over several weeks, no communication was achieved.
The most frequent argument explaining the lack of communication in the complete locked-in state assumes that with progression of ALS or Guillain-Barré Syndrome deterioration of cognitive functions prevents learning and communication (see Sellers & Donchin, 2006, for a discussion of the problem). It is difficult to reject this argument empirically because neuropsychological testing for cognitive functioning is impossible in a completely paralyzed person. We therefore developed an ERP test with an extensive series of cognitive experimental paradigms ranging from simple oddball-P300-evoking tasks to highly complex semantic mismatch N400 and personalized memory tasks eliciting late cortical positivities (Hinterberger, Birbaumer, et al., 2005; Kotchoubey et al., 2005).
More than 100 patients in responsive and nonresponsive vegetative state and 24 ALS patients at different stages of the disease were tested. The relationships between the complexity of a cognitive task and the presence or absence of a particular component are rather inconsistent (Kotchoubey et al., 2005; Kotchoubey, Lang, Bostanov, & Birbaumer 2002), meaning a patient may show absent early cortical components such as N1 but normal P300, or absent P300 to simple tones but intact P600 to highly complex verbal material. With one exception, all CLIS patients had ERP responses to one or more of the complex cognitive tasks, indicating at least partially intact processing stages in the complete locked-in state (Hinterberger et al., 2005). Patients in the more advanced stages of ALS show slowing of waking EEG sometimes into the theta band. This slowing may be, at least in part, caused by episodes of anoxia due to inadequate functioning of artificial respiration. It is often difficult to decide whether the patient is awake or in sleep stage 1 or 2. One CLIS patient gave informed consent to implantation of electrodes in the brain over a two-session period by answering “yes” with imagery of milk taste and “no” by imagining lemon taste, and measurement of the pH level in mouth cavity mucosa served as the dependent variable (Wilhelm et al., 2006). Responding with BCI and the pH device was lost again after implantation in this patient. Slowing of the ECoG and complete absence of gamma-band activity characterizes the recordings.
These ERP data neither prove nor disprove normal information processing in CLIS but suggest some intact “processing modules” in most ALS patients with CLIS despite a reduced general arousal. Three of the remaining 12 patients of our sample entered LIS and continued to use the SCP-BCI for verbal communication, indicating transfer of learning from rudimentary motor control (mostly eye movements) to LIS and probably to CLIS also.
Assuming partially intact processing in ALS patients who are completely locked in and possible transfer of already acquired BCI communication to CLIS, the question of why the patients who entered the CLIS before learning BCI use did not acquire control of their brain signals (SCP-BCI and SMR-BCI was tried on this CLIS group) remains. Figure 1 demonstrating the failure to replicate operant (“voluntary”) learning of visceral functions (see Dworkin & Miller, 1986) may provide an answer to this question: Chronically curarized rats and people with longer time periods in CLIS may lose the contingency between the required physiological behavior (SMR decrease or heart rate increase) and its consequences (brain stimulation reward in the curarized rat and letter selection in the patient). Extinction sets in due to there being so few reinforced learning trials in the rat and in the completely locked-in patient. No contingency remains at all: Thoughts and intentions are never followed by their anticipated consequences in one's own behavior or in the behavior of others, and thoughts and imagery and goal-directed feelings are extinguished.
Theories of consciousness come to a conclusion similar to learning theory accounts of extinction of thinking. In a Hebbian tradition, associative binding between distinct stages of neural activity was postulated as the crucial mechanism behind conscious experience and perception of sensory and motor events (Singer & Gray, 1994/1995). The presence of localized gamma-band responses in the cortex functions as an electrophysiological indicator of associative binding of cell assemblies into meaningful percepts; its absence seems incompatible with conscious percepts and “Gestalt” formation (Kaiser, Lutzenberger, Preissl, Ackermann, & Birbaumer, 2000). Psychophysiological and psychophysical experiments comparing self-induced voluntary actions with the same but involuntary movements caused by transcranial magnetic stimulation (TMS) or external agents demonstrate that conscious decision and perception of “will” depends on the close contiguity in time between the decision and the response. Voluntary action and thoughts and their consequences are attracted together in time; involuntary externally initiated and attributed responses and their effects are experienced as more distant in time (Haggard, Clark, & Kalogeras, 2002; Libet, Gleason, Wright, & Pearl, 1983). They are consequently not interpreted as a conscious unit but separate cognitive elements incapable of acquiring any contextual meaning. Virtually all thought–action–consequence contingencies in a completely paralyzed person become externally induced by patient-independent agents, usually the caretakers. The resulting cognitive state and remaining information-processing capacities remain unclear until the first CLIS patient communicates.
Under the assumption that passive-sensory information processing remains intact in completely locked-in patients (see above), the failure to control autonomic functions with operant learning in the curarized rat (see Dworkin & Miller, 1986) and the described experiments on transcranial magnetic stimulation and voluntary movement seem to provide converging evidence for the following: In the complete locked-in state, the fact that intentional thoughts and imagery are rarely followed by a rewarding or punishing stimulus (i.e., attention from others for that thought) creates an extension of the subjective time perception of the interval between a response (thought) and eventual consequences. Therefore, the probability for an external event (e.g., attention of a family member) to function as a perception of a causal contingency between the response (thought) and its consequence becomes progressively smaller, and after a long CLIS it may vanish altogether. What fills the subjective world may consist only of the few remaining external auditory and tactile and visceral sensations bearing no contextual relationship between them. With the lack of reinforcing contingencies controlling the maintenance of the stream of thoughts, they extinguish slowly. As demonstrated by Haggard et al. (2002), it is this lack of motor control consisting of intention (“will”), preparation, execution, and sensory and external feedback that determines the deteriorating subjective time estimation between response and its consequence.
Donchin (personal communication) assumes that “fooling” the system by providing artificial stimulation such as TMS or electric brain stimulation contingent after a particular neural respose may delay the extinction of goal-directed thinking. The motor control factor responsible for the cessation of voluntary cognitive activity and goal-directed thinking in the completely locked-in patient and the curarized animal lends support to a “motor theory of thinking” similar to that discussed by William James (1890).
Another consequence of response–consequence separation was described as “learned helplessness” that characterized depression at the affective level and deficits in problem solving at the cognitive level (Seligman, 1975). Surprisingly, the commonsense prediction that complete paralysis accompanied by the loss of most positive reinforcers should result in depression and despair was not confirmed. But common sense and folk psychology often result in egregious errors.
Emotion and Quality of Life in ALS and Paralysis
Most ALS patients opt against artificial respiration and feeding and die of respiratory problems. In many countries, doctors are allowed to assist the transition with sedating medication to ease respiration-related symptoms. If doctor-assisted suicide or euthanasia is legal, as it is in the Netherlands and Belgium, very few patients vote for continuation of life. The vast majority of family members and doctors (usually neurologists) believe that the quality of life in total paralysis is extremely low and continuation of life constitutes a burden for the patient and that it is unethical to use emergency measures such as tracheostomy to continue life. The pressure on the patient to discontinue life is enormous.
The facts on end-of-life issues and quality of life do not support hastened death decisions in ALS, however, and the scientific literature and our own studies challenge the pervasive myth of helplessness, depression, and poor quality of life in respirated and fed paralyzed persons, particular with ALS (Albert, Rabkin, Del Bene, Tider, & Mitsumoto, 2005; Quill, 2005). Most instruments measuring depression and quality of life such as the widely used Beck or Hamilton depression scales are invalid for paralyzed people living in protected environments because most of the questions do not apply to the life of a paralyzed person (“I usually enjoy a good meal,”“I like to see a beautiful sunset”). Special instruments had to be developed for this population (Kübler, Winter, et al., 2005). In studies by Breitbart, Rosenfeld, and Penin (2000) and by our group (Kübler, Winter, et al., 2005) only 9% of the patients showed long episodes of depression, most of them in the time period following the diagnosis and a period of weeks after tracheostomy. Figure 5 shows the results for depression (A) and for quality of life (B) rated by patients and family members and caretakers. As can be seen, ALS patients are not clinically depressed. In fact, they are in a much better mood than psychiatrically depressed patients without any life-threatening bodily disease. Likewise, patients rate their quality of life as much better than their caretakers and family members do, even when these patients are completely paralyzed and respirated. None of the patients of our sample (some of them in LIS) requested hastened death.
It could be argued that questionnaires and interviews reflect more social desirability and social pressure than the “real” behavioral–emotional state of the patient. The social pressure in ALS, however, directs the patient toward death and interruption of life support. The data, therefore, may underestimate the positive attitude in these groups. This hypothesis is strongly supported by a series of experiments with ALS patients at all stages of their disease using the International Affective Picture System (IAPS; Lang, Bradley, & Cuthbert, 1999). Lulé et al. (2005) and Lulé et al. (in press) using a selection of pictures with social content, found more positive emotions to positive pictures and less negative ratings to negative pictures in ALS than in matched healthy controls. Even more surprising are the brain responses to the IAPS slides (Figure 6). FMRI measurement in 13 patients with ALS and controls demonstrated increased activation in the supramarginal gyrus and other areas responsible for empathic emotional responses to others comparable to the “mirror neuron network” identified first by Rizolatti and colleagues (Gallese, Keysers, & Rizzolatti, 2004). Furthermore, brain areas related to the processing of negative emotional information such as the anterior insulae and amygdala show less activation in ALS. These differences become stronger with progression of the disease 6 months later.
One is tempted to speculate that with progression of this fatal disease, emotional responding on the behavioral and central nervous system level improves toward positively valenced social cues, resulting in a more positive emotional state than in healthy controls! The positive responding and positive interaction of the social environment and caretakers to a fatally ill, paralyzed person may, in part, be responsible for the prosocial emotional behavior and for the modified brain representation of the “observer” depicted in Figure 6 as predicted by social learning theory (Bandura, 1969). Taken together, the results on emotional responding and quality of life in paralyzed ALS patients suggest a more cautious and ethically more responsive approach toward hastened death decisions and last-will orders of patients and their families. The data reported here also speak pervasively for the usefulness and necessity of noninvasive BCI in ALS and other neurological conditions leading to complete paralysis.
The preceding sections were devoted to BCIs designed for verbal communication in completely paralyzed persons unable to use muscular or autonomic responses to activate an assisted communication device. The second major field of BCI research concerns restoration of movement in patients with paralysis, mostly spinal cord lesions, chronic stroke, and other movement disorders. It is certainly an attractive possibility to build a direct connection between voluntary movement command centers in the brain and the periphery isolated from these regions by a central, spinal, or peripheral lesion.
Invasive and Noninvasive BCIs for Restoration of Movement
Brain–computer interface research received its impetus from animal research reconstructing movement from microelectrode-recorded spike trains or synaptic field potentials (Donoghue, 2002; Nicolelis, 2001). After extensive training and the implementation of learning algorithms (for an exception, where animals learned rapidly, see Serruya et al., 2002), monkeys move cursors on screens toward targets or an artificial hand moves in four directions directed by spike activity, demonstrating the possibility of translating cellular activity into simple movements online. After such training, even complex movement patterns can be reconstructed from an astonishingly small number of cells located in the motor or parietal areas (Musallam, Corneil, Greger, Scherberger, & Andersen, 2004; Nicolelis, 2001; Schwartz et al., 2001; Taylor et al., 2002). The plasticity of the cortical circuits allows learned control of movements directly from the cellular activity even outside the primary or secondary homuncular representations of the motor cortex (Taylor et al., 2002).
A multielectrode array recording spike and field potentials simultaneously was implanted in a single quadriplegic patient's motor hand area (2004) by Donoghue's group (personal communication, April 2005). Within a few training sessions, the patient learned to use neuronal activity from field potentials to move a computer cursor in several directions comparable to the tasks used for multidimensional cursor movements in the noninvasive SMR-BCI reported by Wolpaw and McFarland (2004). None of the invasive procedures allowed restoration of skillful movement in paralyzed animals or people in everyday-life situations. The animals studied in BMI research (Nicolelis, 2003) were all intact animals who learned to move an artificial device or curser for food reward without moving their intact arm in highly artificial laboratory situations. Any generalization from the invasive animal BCI approach to paralyzed people is premature.
In contrast to the invasive approaches, SMR-controlled BCIs developed by Pfurtscheller and colleagues (Pfurtscheller, Neuper, et al., 2003; Pfurtscheller et al., 2005) allowed control of reaching and grasping in high spinal cord lesioned patients. Pfurtscheller, who was the first in testing and implementing SMR-based BCIs for motor paralysis, demonstrated convincingly the potential usefulness of noninvasive BCIs for motor restoration, more clearly than the widely acclaimed and cited animal experiments using implanted microelectrodes. In one preparation, Pfurtscheller, Neuper, et al. (2003) used the SMR signals to activate electric stimulation electrodes attached to the paralyzed arm and hand muscles in order to reach and grasp objects in a quadriplegic patient. These data suggest that, with intelligent prosthetic devices and orthoses, electrical muscle stimulation, and EMG feedback from the target muscles, noninvasive BCIs may have promise for highly complex movement reconstruction. Neuper, Müller, Kübler, Birbaumer, and Pfurtscheller (2003) demonstrated successfully that the same SMR-based BCI used for motor control can be used as a communication device in a paralyzed cerebral palsy patient and that training and measurement may be performed even from laboratories located at long distances from the patient. However, none of the paralyzed patients reported in the literature is using the motor BCI in everyday-life situations as long as voluntary upper face and shoulder movements can activate an artificial limb. Therefore, in spinal cord lesioned patients, invasive and noninvasive BCIs (BMIs) may be useful in the future for the few patients with extremely high spinal cord lesions only.
Another obstacle for real-life daily use of BCIs regardless of the type of application is their demand on attention. Whereas simple motor commands in the intact adult organism are executed with a minimum of cognitive resource allocation, the voluntary production of brain signals irrespective of the type of signal needs more and continuous attentional resource mobilization than highly automatized skills because automatization of brain control is slow and probably never complete (Neumann et al., 2004). In addition, the noninvasive BCIs allow relatively undisturbed slow verbal communication, but production of movement with brain activity inevitably generates movement-related artifacts difficult to eliminate online. Particularly in patients with spasticity and uncontrolled movement episodes, attempts to produce motor action from EEG signals are often punished by the presence of these artifacts and cause frustration and decline in motivation (Birbaumer et al., 2003, 2004; Kübler, Winter, & Birbaumer, 2003). For these special cases, the implantation of electrodes may constitute a viable alternative. Whether the electrodes need to penetrate hundreds to thousands of neurons as some maintain (Nicolelis, 2003) or only small samples of critically important neurons responsible for directional tuning, for example, is an unresolved question.
Birbaumer, Weber, Buch, Neuper and Cohen (in press) at the National Institute of Neurological Diseases and Stroke (NINDS) together with the Tübingen group (Lal et al., 2006) developed a BCI system for chronic stroke that may solve most of the problems of noninvasive BCIs devoted to motor restoration and may constitute a sensitive alternative to invasive approaches. In this preparation, patients with no residual hand movement are trained with a magnetoencophalography (MEG)-contolled hand orthosis (Figure 7).
For the first 10 to 20 training sessions in the MEG and after successful hand opening, closing, and grasping using feedback and modulation of central SMR magnetic-field oscillations, the patient is switched to a mobile EEG-SMR-based BCI wearing the same orthosis. Because brain magnetic fields are not attenuated and distorted on their way from the cortical generators to the MEG dewar containing the recording SQUIDs, MEG provides a much larger and more localized SMR response, allowing control of even single fingers (Braun, Schweizer, Elbert, Birbaumer, & Taub, 2000). The head of the patient is fixated in the dewar and the fingers attached to the orthosis open and close the hand contingent on SMR increase and decrease. The patient receives visual and proprioceptive feedback from his/her own movement and simultaneously watches a screen with an up or down moving cursor that indicates the amount of SMR present in the appropriate cortical region 7 s before the self-produced SMR moves the orthosis attached to the hand. Figure 8 depicts the SMR magnetic field localization and training performance of a patient with long-standing chronic stroke and complete immobility of the affected hand. As a positive side effect, the patient experienced complete relief of hand spasticity after several training sessions.
The primary aim of the MEG-BCI training in chronic stroke is not only restoration of movement but cortical reorganization and compensatory cerebral activation of nonlesioned brain regions through voluntary brain-controlled hand movement of the paralyzed limb and reduction of contralesional hemispheric inhibition. Duque et al. (2005), Murase, Duque, Mazzocchio, and Cohen (2004), and Ward and Cohen (2004) have shown in a series of transcranial magnetic stimulation (TMS) experiments that the strong inhibitory effect from the healthy hemisphere on the lesioned hemisphere may be responsible for the lack of reorganization and insufficient recovery of the stroke-affected brain area. Consequently, the MEG-BCI training is targeted toward a “strenghthening” of the ipsilesional brain regions around the destroyed tissue and “weakening” of the homotypical regions in the opposite hemisphere. This is achieved by using SMR oscillations (from 10 to 20 Hz) as a movement-directing source originating in the immediate neighborhood of the lesion and simultaneous interruption of feedback and orthosis control with contralesional coactivation. Cortical reorganization is measured before and after training with fMRI of imagined and executed hand and lip movements as described by Lotze et al. (Lotze, Braun, Birbaumer, Anders, & Cohen, 2003; Lotze, Grodd, et al., 1999; Lotze, Montoya, et al., 1999). Whether the training results in improved hand mobility with or without orthosis is the question of the ongoing clinical experiments. Chronic stroke with no remaining finger mobility is resistant to treatment and shows no spontaneous recovery; any improvement through BCI training therefore constitutes a success. Again, invasive implantation of large quantities of electrodes with the many risks and uncertainties involved may be superfluous or reserved for the few most difficult cases.
Future Directions: The Metabolic Whole Brain BCI
Weiskopf et al. (2003) for the first time demonstrated convincingly that healthy persons are able to regulate BOLD (blood oxygen level dependent) responses from circumscribed cortical and subcortical brain regions using online functional magnetic resonance imaging (fMRI-BCI). These authors and others (DeCharms et al., 2005) demonstrated substantial effects of BOLD-response BCI training on behavior: Pain, emotional arousal, and memory were investigated and astonishingly strong effects on the behavioral variables after short training periods with fMRI-feedback training were shown. This is not surprising, considering that vascular changes in brain arteries and veins responsible for metabolic responses such as BOLD and brain blood flow may allow superior voluntary (operant) control because of the vascular-motor component of the physiological target response. Dilation and contraction of vascular changes are sensed by the brain and regulated by neural structures with closely coupled autonomic and somatic-motor functions, allowing access to voluntary control (Dworkin, 1993).
Clinical application of fMRI-BCI is presently unrealistic and unlikely, considering the cost and technological difficulties involved in real-time fMRI. It will, at present, remain reserved for research purposes and experiments intending to demonstrate effects of learned local blood-flow changes on emotional and motivational behavior. A clinically more realistic new metabolic BCI system has been proposed and tested recently by Sitaram et al. (in press). These investigators used near-infrared spectroscopy (NIRS) and measured, with optical recording devices, changes in cortical oxygenation and deoxygenation. Using the reflection of light in living tissues with high circulation density such as the brain, NIRS is completely noninvasive (Coyle et al., 2004). NIRS devices are also relatively inexpensive (price equivalent to that of a multichannel EEG) and commercially available. Another virtue of NIRS is portability, allowing, for example, the training of young children. Sitaram et al. (in press) demonstrated online operant control of sensorimotor brain areas in five healthy subjects and spelling of letters with NIRS-BCI with an accuracy of 70%–95% after only two training sessions and with information transfer speed comparable to EEG-BCI.
Brain–computer interfaces or brain–machine interfaces are intended to translate “thought into action” with brain activity only. The research devoted to this goal has raised many fascinating questions about brain–behavior relationships without achieving its ultimate practical goals: communication with the completely paralyzed and restoration of movement in paralysis. But the reformulation of the problem of how brain cells and their output create observable behavior applied to an existential problem of human suffering will focus the questions we ask in cognitive neuroscience and psychophysiology. BCI research stimulates long-held hope and expectation of thought and emotion detection and translation from brain states. And true to the old Yiddish saying, “Fur lojter hofenung wer ich noch meschugge” [I am crazy with hope].