An interesting theoretical property of common coding theory is that it combines representational and dynamic processing in an elegant fashion (Chandrasekharan & Osbeck, in press). By promoting common coding at the neural level during learning, the theory is unabashedly representationalist. However, one effect of this coding is that during run-time, the activation of movement in one modality (e.g., perception of movement) automatically, that is, directly or instantaneously, activates the others (imagination/execution of movement), thus translating an external movement into covert movement in body coordinates immediately. The traces generated by this direct/instantaneous translation also supports the covert activation of the motor system in a temporally extended fashion, such as in imagining a mental rotation.
Common coding is thus a representationalist account, but it proposes a representation that supports a motor simulation mechanism, which can be activated across different timescales—instantaneous simulation of external movement, and also extended simulations of movement. The latter could be online, that is, linked to an external movement (as in mental rotations while playing Tetris, see Kirsh & Maglio, 1994), or can be offline (as in purely imagined mental rotation). As both “direct” and offline simulations can be activated by perception of external movement, the model offers a common body coordinate–based representation that connects external models with internal models (when both possess movement).
4.2. Role of construction
Common coding also shows why building an external model (construction) is important in generating novelty. I will lay out this feature as a set of points.
Building a system that exhibits the same behavior as a target system involves thinking of the system in component terms, and then imagining how these components would interact dynamically to produce the target behavior. This process automatically generates a flurry of internal movements in imagination. These are detailed movements that are not generated by just observing the target system. An example would be thinking of how to build a bicycle, as opposed to watching a passing bicycle or imagining a bicycle you owned. The latter two would not generate internal movements of, For example, the intricate and interconnected movements between the pedals, chain, gears, and the wheel, or ball bearings.
As the external system is built, it generates its own movements, which may be in conflict with the imagined internal model of movement patterns. This leads to revision—of either the internal or external model, or both.
The system is built component by component, and this provides more fine-grained control over the way the different components interact and the ability to manipulate each component. This generates a further level of fine-grained movement parameters, which are not provided either by observing the target system or by the imagined model of the target system.
Building an external model of behavior involves a synthesis of a large set of component movements. So the building process also provides integrated information—about how movements influence each other and global movement patterns. Also, this shows how the properties of different components relate to each other in dynamic terms, such as heat generated/attractive-repulsive forces/centrifugal-centripetal effects that arise during movement.
Building is based on primary actions, while imagination is based on secondary, stored, actions. This means building movements have more impact compared to imagined movements, and they can thus help override imagination’s existing grooves of movement (functional fixedness, which arise from traces of previous actions and perceptions).
These features provide a way of thinking about how external facsimile models generate novelty: They generate external movements that “perturb” the internal models in a seamless and integrated fashion. Imagination by itself can perturb the internal models in many ways, but the changes it can generate are constrained by factors such as memory, cognitive load, repertoire of existing movements, use of standard manipulations (“functional fixedness,”Adamson, 1952;German & Defeyter, 2000; German & Barrett, 2005), etc. At the other extreme, imagination running by itself can lead to unconstrained changes in parameters (“flights of fancy”).
The external model provides more freedom and flexibility, as it retains its states, thereby lowering memory and cognitive load. It also provides ways of escaping fixedness, as the building process generates more movements, or possibilities of movements. The model serves as a more flexible “external imagination,” as it can generate a wide variety of movements that are not supported or activated easily by imagination. This external imagination is also tightly constrained by executable possibilities. Common coding ensures that these external movements seamlessly perturb the internal model, almost as if external manipulation is the same as imaginative manipulation. Just as making a compatible physical action speeds up mental rotation (Wexler et al., 1998) and planning incompatible movements interfere with mental rotation (Wohlschlager, 2001), the act of construction could perturb the internal model.
There are a few strands of evidence that point to this role of external models. Martin and Schwartz (2005) showed that manipulating physical pieces facilitated children’s ability to develop an interpretation of fractions. They also report that children who learned by adapting unstructured environments transferred to new materials better than children who learned with “well-structured” environments. Based on this, they argue that the opportunity to adapt an environment permits the development of new interpretations that can advance learning. In a related vein, German and Defeyter (2000) report that children 5 years old and younger are less prone to functional fixedness compared to children 7 years old and older. It is possible that this is because the movements of younger children are less “set” than the older ones, and hence more amenable to novel combinations. Craig, Nersessian, and Catrambone (2002) report a study using the radiation problem, where a doctor has to devise a way to kill a tumor using radiation, but any radiation strong enough to kill the tumor would also kill the surrounding tissue. The correct strategy is to use weak radiation from many directions. Before trying to solve the problem, students were first exposed to an analogical story (a general capturing a fortress using small groups). One group of students then recalled this analogy using sketches, another by manipulating wood blocks, and the third just verbally. While solving the radiation problem, the wood block group performed better than the sketch group, and both performed better than the verbal recall group, suggesting that the physical activity allowed them to better connect the analogy story and the radiation problem.
4.3. Epistemic coordination
The common coding approach helps in understanding how external models allow groups of people to reach a consensus on models, concepts, and control structures, and provide groups with a common way of thinking about problems and solutions. In existing work on epistemic coordination, distributed cognition theory considers as given the common conceptual structure of an external artifact across team members; it does not ask the question how this structure arises across people. Further, the theory does not specify whether the representations traded among team members are amodal or modal (Barsalou, Simmons, Barbey, & Wilson, 2003) in nature. If they are amodal, such a shared structure arising across people is a mystery, because amodal representations are considered to have arbitrary relations to entities in the world, and there is no reason why each person, particularly people new to the workspace, should have the same representation of the artifact.
On the other hand, in a movement-based model, the motor system “resonates” to an external movement automatically, which means every detectable movement is picked up and replicated in some fashion by the system, even though the movement is not always overtly performed. The level of arbitrariness in this replication is much more limited, as the replication of a movement by the motor system would be broadly similar in everyone. While there would be variation in the way the movement is integrated into the internal model, the common resonance of the movement generated by an artifact or model system provides a base structure that is shared by everyone in the workspace.
Further, such a replication of the artifact/system’s movements, combined with replication of the actions generated by humans interacting with the artifact/system, could also support second-order understanding—based on repeated movement patterns, and their automatic replication across the group, everyone “knows,” or has a sense that, there is a common understanding of the artifact/system. These first- and second-order shared internal structures based on the artifact/system’s movements support an implicitly shared, and considered as shared, internal model of the artifact/system. This leads to the coordination of workflow around the artifact/system and also supports the emergence of common approaches to solving problems. Note that this “hub” quality, combined with the ability to generate new movements, leads to the model system working as an “imagination hub” as well, as the shared internal model of the system allows people to generate similar or overlapping ideas and follow, support, or reject novel ideas generated by others easily. Work within common coding theory has recently started examining such sharing of movements across subjects during joint tasks (Sebanz et al., 2005; Welsh et al., 2007), and results show that such sharing indeed emerges.
Closely related to this “hub” notion is social learning, particularly imitation-based learning of techniques in science and engineering labs. Such learning is crucial to understanding distributed cognitive systems such as laboratories that evolve over time (Nersessian et al., 2003). One of the primary areas of application of common coding theory is in the understanding of imitation (the primary method of social learning), where it is argued that imitation is based on the automatic activation of others’ actions by the observer’s motor system (see Brass & Heyes, 2005; Hurley & Chater, 2005; for reviews). Our observation of the two biomedical engineering labs showed that lab members mentor juniors initially in techniques, based on an imitation-based apprentice style of learning (Alac & Hutchins, 2004; also see Becvar et al., 2007). Later on, the juniors pick up a style of learning we have termed “agentive learning” (Newstetter, Kurz-milcke, & Nersessian, 2004), which is an opportunistic and self-motivated learning that weaves together people, systems, and environmental affordances in an integrated fashion to solve problems. The common coding framework allows us to examine this style of learning in more detail, examining the combination opportunities provided by different system movements and internal movements, and how these could relate to such agentive learning.
4.4. Common coding and the dish system
How does the common coding framework help us answer the questions raised about the case study? In the following section, I will take up the questions raised in section 1 (with reference to how D11’s computational model changed D4’s notion of bursts-as-noise) and lay out how the common coding framework could address these questions.
Why was it not possible to think of the spatial nature of the activity before the computational model, even though the group used global and spatial concepts, such as population vectors, and was precisely interested in network-level learning?
Obviously, the answer is visualization. But visualization viewed traditionally is a representation, where the “standing in” relation it has to the underlying data is what is crucial. In the common coding framework, movements in the visualization are also a way of generating equivalent movements in body coordinates. By this resonance system, the movements in the visualization influence the internal model directly. This means visualization is not just a user-friendly mapping of the data generated by the system, it is a way of transferring the system’s behavior to the internal model of the system.
However, both the in-vitro and the computational model used visualizations, so why did not the former lead to a breakthrough? The in-vitro dish’s output (MEAScope graphs, see Fig. 1) did not have global movement-generating properties, so it was not able to generate global spatial concepts such as CAT and burst types. For one, the MEAScope visualizations were detached from the dish itself and showed activity at each node using graphs of spikes. So while there were movements on screen, they were spiking movements detached from the dish, and the spatial activity pattern of the spikes was hidden among the different graphs. Further, the spiking movements required a mapping function to link it to the internal (network) model of the dish. The spikes thus acted as both a limitation (submerging the global movement patterns) and a form of fixation (activating spike-based internal movements more than other possible ones, such as network-based internal movements).
The computational model, on the other hand, used a visualization where the activity moved across the dish network itself, thus allowing an easy integration of the network activity seen on screen with the internal model of activity in the dish, in movement terms. The internal model (movement of electrical activity through neuronal connections) was linked to a specific pattern of such movement, namely a movement with a spatial focus, but “jumped” around (CAT). The combination of the internal model of the dish and the spiking activity in the computational model generated a model of dish activity that had spatial parameters and movement behavior across the dish, leading to the new spatial concepts.
How did seeing the spatial pattern lead to a change in the internal model?
Seeing the spatial pattern generated novel motor simulations, which pushed the internal model out of the spike-based movement groove (local minima) it was in, and raised the types of manipulative movements available for the internal model (see availability heuristic, Kahneman & Tversky, 1982). Also, the spatial movement patterns and the stop-go control were related, which led to more correlated movements. This helped in getting out of the spike-based minima as well. Since the structure of the dish and the network were integrated in the visualization, the spatial movement pattern could influence the internal model’s movements more directly. This integrated visualization also helped lower memory and cognitive load, as the external model retained its states, could be stopped at any time, and there was no mapping required between the movements in the visualization and the movements in the internal model of the dish.
How did control contribute to the shift in perspective?
The primary contribution of stop-go control is its ability to generate more fine-grained movements. This is particularly useful in cases where imagination is constrained by more “available” parameters. In such cases, the external model makes different parameters available for movement. D11 could stop and start the network activity as he pleased, thereby raising the number and types of manipulations his, and others’, imagination could execute on their internal model of the network. In our particular case, this was very important, because this large number of varied movements generated the case for types of bursts, and movements of such bursts across the system, and these two ideas ultimately led to discarding the notion of burst as noise. If the number of movements generated were limited and less varied (in terms of both output and input), the classification based on types and location would not arise, as the results from the limited trials would be unrecognizable from noise, and thus would not generate these patterns.
The second contribution of control, in common coding terms, is the activation of an “interventional stance,” where the agent actively tracks the external movements and generates intervention plans. An example would be a video game, where the player constantly generates and revises action plans, and tracks the environment actively in relation to them. In contrast, watching an action movie only generates replication of the external movements. Control leads to constant tracking of the simulation, both in surface activity as well as internal activity. This helped in qualitatively judging the rate of activation (of the CSIM neurons), and then judging the rate of propagation of activity through the network, both in terms of movement. This, in turn, supported judging the spatial extent, and comparing the extent of different patterns that are generated, leading to the idea of burst types. Stop-go control allowed building stronger correlations between stimulation and activation of the model. The fine-grained movements provided by such control, together with the tracking activity involved, helped in detecting and standardizing common movements and their effects (thus forming causal relations).
What type of structure underlying an internal model would allow such a broad change to occur so quickly across people, based just on observed patterns?
The fact that the visualization of movement changed the internal model so quickly is an indirect indication that the underlying structure of the internal model has movement properties. If the internal model has an amodal structure, for example, with quantifiers and algebraic notations, it would not change as quickly, across people, based on visualization of movement. The case of D11 is not an isolated instance of conceptual change based on such visualization. Entire methodologies, disciplines, and phenomena challenging existing models have been built just from observed movements on computer screens. These include Complexity Theory (Langton, 1984, 1990), Artificial Life (Reynolds, 1987;Sims, 1994), models of plant growth (Prusinkiewicz, Lindenmayer, & Hanan, 1988; Runions et al., 2005), computational bio-chemistry (Banzhaf, 1994; Edwards, Peng, & Reggia, 1998), and computational nanotechnology (reported in Lenhard, 2004; Winsberg, 2006a). All these novel areas of exploration are based on movement systems and could not exist without visualizing movements. From a common coding perspective, the ability of such visualizations to challenge existing models is not a mystery, because most internal models in science involve movement, and visualized movements can interact with these internal models in a seamless way, across the population who possess such internal models. Before the advent of the computer, such external movements were approximated by diagrams and giving directions to the audience to move the elements in the diagram in imagination, in specific ways (Nersessian, 2002a, 2008), sometimes using arrows. This process is very close to the simulation of actions from end-point movements such as writing and drawing (Viviani, 2002).
How can building lead to innovation? How does building external facsimile models contribute to discovery?
The process of building a model requires both executing novel movements and generation of novel movements by the model. These movements alter internal movement-based models of a target system. Since the objective is to build a working model, it sets constraints on which internal movements are activated. Building external models is thus a way of perturbing the imagination in a focused and constrained fashion, and this perturbation is one way in which facsimile models can generate novelty. A related way in which such “perturbation” could contribute to discovery is by generating random combinations of internal and external movements, and thereby “connecting” brain regions that are not activated together ordinarily, while just imagining movement (see Schubotz & von Cramon, 2004, for imaging evidence for such a “thread” across different perceived movements).
A second way in which building leads to innovation is via the role it plays in judgment, where other group members decide whether the novel concept is worth pursuing, and whether it would address the problems they are facing. This judgment is easier to do with a built and “manifest model” (Nersessian & Chandrasekharan, 2009) than an internal model, because the manifest model allows group members to perform manipulations and thus form common movement representations of the proposed concept. The manifest model also improves group dynamics. One need not say “you’re wrong” only that “the model doesn’t support that claim.”
What desirable cognitive features should such external models have to support discovery?
Based on the common coding framework, one desirable feature would be lots of movement—building and manipulation activity. Another feature would be a range of ways to generate movements in the built system. The movements need not always be visualizations. For instance, protein structure has been generated as music (Dunn & Clark, 1999), and scanning microscope output has been used to generate haptic feedback (Sincell, 2000).
A third desirable feature would be more control, to generate more fine-grained movements. Four, a closer integration between the movement generation mechanism and the internal model of the target system, similar to D11’s use of the dish to display network activity.
4.5. Objections to a movement-based approach to discovery
In this section, I will consider a set of objections that have been raised to a movement-based explanation of the coupling between internal and external models, and the role of movements in discovery.
In the common coding model, all transactions between internal and external models happen through motor activation. Are all the internal models of the world we possess coded in movement terms? What about objects and colors and labels? Static images and symbols (such as graphs and histology stains) do exist in science, and they provide information. How are they incorporated into internal models?
In the common coding view, the brain is a control mechanism that evolved to coordinate actions and movements in the world. There are two ways in which static images and symbols could be integrated with such a movement-based account. One is a model of representation where the “standing-in” relation between a symbol and the world arises out of actions and movements (see Chandrasekharan & Stewart, 2007). In this view, the static nature of a symbol is an illusion—the symbol is part of a dynamic sense-action network, and what appears static is the part that remains constant across actions and movements. A close metaphor would be the persistent core of a bee swarm or a tornado. A related possibility is that static images and symbols are starting points for internal simulations, as in the use of the Two-thirds power law to generate movement patterns from drawings (Viviani, 2002). This second option comes for free if the first view is accepted (for details see Chandrasekharan & Stewart, 2007). Generally speaking, any system designed to process dynamic structures can process static structures, for example, as equilibrium states or time slices. At the level of neural systems, there are recent efforts to explore how object movements could be coded by body movement areas (see Schubotz, 2007; Schubotz & von Cramon, 2004).
For our purposes here, it is highly likely that for a large majority of models in science and engineering, movement is a central component, given the focus on causation and co-variation, and the high use of dynamics as an explanatory device in science. This means external movements could lead to changes in such models via the ideomotor effect. As for objects and colors, Noe (2004) has made a persuasive case that vision, particularly object perception and color, requires movement components, either self-generated (such as eye movements or internal simulations) or object-generated. On labels, research into processing of concepts and sentences shows that processing of labels involves movement components. However, there could still be elements processed in a static, purely pattern-matching, fashion. The common coding view does not deny this possibility.
If any movement can activate the motor system, wouldn’t all movement influence internal models?
In a general sense, all movements in the world do influence the brain, and the representations it contains. However, changes to specific internal movement modules (models) can occur only when these modules are active. For instance, the motor areas involved in dancing and piano playing are not usually active when dancers and piano players are, for example, driving. So the movements generated in driving are not influencing the dancing/piano-playing module. However, when they are watching a dance/piano concert, or planning one while driving, the module is activated. And any movements on stage, or movements encountered while doing the planning, have the ability to influence the internal model. Similarly, the external movements can contribute to the internal models of scientists and engineers only when they are imagining or interacting with the model.
The common coding experiments show only that movements outside and movements inside influence each other. The influence could be positive (as in speeding up mental rotation) or negative (as in slowing down mental rotation). Given this, why should construction, and external movements in general, always play a positive role?
In the view I have outlined, the process of construction serves to generate more movements than is possible in imagination. That by itself is always positive, as more movements help in overcoming functional fixedness. The negative role arises only in the interaction of the externally generated movement with the internal model. It is highly probable that such negative integration or interference does happen (as in the noise interpretation of bursts by D4), but as the construction process moves further, it leads to other movements, which can dislodge such interpretations. One way to think of the construction process is to think of it as working similar to a genetic algorithm, where the system can settle into local minima, but random mutations always dislodge the system from such states. The movements generated by the construction process are similar to these random mutations. Also, ultimately, all integrations of movements into the internal model are “grounded” by the building of the facsimile system, which need to exhibit the behavior of the target system. So misinterpretations are dislodged by this requirement as well. Note also that the building process involved in developing facsimile models just perturbs internal models; it does not guarantee new discoveries and insights. Perturbation is just a strategy—it is not a method—so it does not always have positive effects.
Does all building lead to new understanding?
No. Kirlik (1998) describes how humans built projectiles thousands of years before we understood dynamics and mechanics. If building always leads directly to understanding, ancient societies would have developed abstractions and theories that accelerated their progress, building more sophisticated projectiles. Since this did not happen, it is clear that sophisticated internal theoretical models need to exist for any building to contribute to understanding. It is the interplay between the building process and the internal model that leads to novel understanding and discovery.