should be sent to Kurt Stocker, Institute of Cognitive and Brain Sciences, University of California, 3210 Tolman Hall #1650, Berkeley, CA 94720–1650. E-mail: email@example.com
This article provides the first comprehensive conceptual account for the imagistic mental machinery that allows us to travel through time—for the time machine in our mind. It is argued that language reveals this imagistic machine and how we use it. Findings from a range of cognitive fields are theoretically unified and a recent proposal about spatialized mental time travel is elaborated on. The following novel distinctions are offered: external versus internal viewing of time; ‘‘watching” time versus projective ‘‘travel” through time; optional versus obligatory mental time travel; mental time travel into anteriority or posteriority versus mental time travel into the past or future; single mental time travel versus nested dual mental time travel; mental time travel in episodic memory versus mental time travel in semantic memory; and ‘‘seeing” versus ‘‘sensing” mental imagery. Theoretical, empirical, and applied implications are discussed.
In the late 19th century, H. G. Wells developed in his fictional writings the idea of a sophisticated kind of machinery that allows one to travel through time, an idea that he eventually coined time machine. While today the idea that we can physically project ourselves over time still belongs to the realm of fiction or to the realm of unsolved problems in physics, the idea that we can mentally project ourselves over time—we might also say, the idea that we are equipped with a mental time machine—has been investigated within psychology and cognitive science quite fruitfully in a variety of ways. Yet a comprehensive and systematic conceptual account of the sophisticated mental machinery that is needed to carry out the remarkable task of mental projection over time is still missing. It is the aim of this article to provide such an account—an account of the time machine in our mind. As we will see, we make use of this machine in ubiquitous ways—for example, every time we utter or listen to a sentence that contains a future or a past tense.
A central term used in this article is ‘‘the imagery structuring of time.” By this I refer to an invisible spatial scaffolding in our mental imagery across which temporal material can be splayed, the existence of which will be proposed in this article. At times, it will be quite natural to assume that a space-to-time mapping in the sense of conceptual metaphor theory (Boroditsky, 2000; Casasanto & Boroditsky, 2008; Clark, 1973; Lakoff & Johnson, 1999) is involved in the structuring of this invisible scaffolding. However, spatial imagery structuring of time can also quite naturally be proposed as underlying linguistic expressions that are often not considered to be metaphorical, although they may have had a metaphoric origin etymologically. It is thus for the present investigation more coherent to assume that mental time is basically constructed out of ‘‘spatialized” mental imagery—‘‘spatialized” is another central term that I use in this article. I use it in the sense that it is neutral as to whether some of the imagery might be transferred via space-to-time mappings or whether some of the imagery might relate to space-to-time mappings only in an etymological sense. An example of temporal constructions that are readily characterized in terms of spatialized temporal imagery structuring are the conceptualizations underlying the use of before and after, conceptualizations that are often treated as having autonomous temporal status and as relating only etymologically to space (e.g., Clark, 1973, p. 50; Moore, 2000, pp. 73–74; Núñez & Sweetser, 2006, p. 413). The current investigation can refine this view somewhat, by postulating that spatialized temporal structures still play a very vital role in the imagery structuring underlying before and after (see Section 4).
Three relevant investigative lines for the present work have been offered by Endel Tulving, Leonard Talmy, and Herbert Clark. Tulving has characterized mental projection over time as ‘‘mental time travel,” the idea that one can project oneself into one's own subjective past (remembering) or imaginally into one's own subjective future (autobiographical temporal imagining). Tulving characterizes such temporal mental actions as carried out by a ‘‘self.”‘‘But an ordinary self will not do,” as Tulving says: It must be a self that can exist ‘‘in subjective time” and is capable of ‘‘traveling” within it (Tulving, 2002, p. 2). Tulving's idea of mental time travel and associated episodic memory has led to a wealth of empirical investigations. In these investigations, it is often (implicitly or explicitly) assumed that mental time travel involves mental imagery. Such studies might, for example, include rating scales that say: 1 = saw event through my own eyes; 5 = saw myself from an external perspective (Addis, Pan, Vu, Laiser, & Schacter, 2009, p. 2226; cf. also Eich, Nelson, Leghari, & Handy, 2009). Talmy characterizes mental projection over time with a temporal perspective point (PP) from where one can have a retrospective or prospective viewing in a temporal frame (2000, pp. 72–76). Talmy describes such cognitive temporal viewings as ‘‘most readily characterized in visual terms as, in effect, pertaining to where one places one's ‘mental eyes’ to ‘look out’ upon a referent structure” (2000, p. 68). These ‘‘mental eyes” are sometimes also characterized as belonging to what Talmy calls a cognitive ‘‘conceiver” (2000, p. 313) or ‘‘ceiver” (2000, p. 158). Although Talmy's work on mental temporal perspective and mental viewings through time is in my view a highly significant contribution to the study of mental time, it has not received much discussion until the present article. Clark characterizes mental projection over time as a space-to-time metaphorical scenario where ‘‘time can be viewed as a highway consisting of a succession of discrete events” and where ‘‘we are moving along it [this highway], with future time ahead of us and the past behind us” or where ‘‘the highway is moving past us from front to back” (Clark, 1973, p. 50; cf. also McTaggart, 1908, p. 470). These are of course Clark's early characterizations of the moving ego and moving time metaphors, which have led to much subsequent theoretical work (e.g., Lakoff & Johnson, 1980, 1999; Traugott, 1975, 1978) and empirical work (e.g., Boroditsky, 2000; Boroditsky & Ramscar, 2002; Ramscar, Boroditsky, & Matlock, 2009). Clark characterizes such temporal mental actions as carried out or experienced by an ‘‘ego,” a term that is still used by many researchers doing work in the conceptual metaphor field (e.g., Boroditsky, 2000; Moore, 2006; Núñez & Sweetser, 2006), whereas some prefer the rather imagistic term ‘‘observer” (e.g., Lakoff & Johnson, 1980, 1999). Clearly, the moving time and moving ego metaphors also require mental projection over time (like the conceptualizations by Talmy and Tulving do): for example, a projection into a metaphorical aheadness to conceptualize a moving or stationary future time. Additionally, Clark—as well as McTaggart (1908, p. 470)—systematically bring in the idea that the ‘‘self” (McTaggart) or ‘‘ego” (Clark) is projected over spatialized time. Furthermore, McTaggart says that the self is ‘‘pictured” in these spatialized temporal landscapes.
It is of course not the case that Tulving, Talmy, and Clark have all covered the same aspects of mental projection over time. Tulving's (1985, 2002) mental time travel is set exclusively within an autobiographical realm and is exclusively concerned with the present moment as a ‘‘temporal deictic center.”1 Additionally Tulving's (and memory researchers’ in general) conceptualization of mental time travel is to view it as an essential feature of episodic memory—our capacity to remember past events that additionally also allows us to construct personal future events—and not additionally as a feature of semantic memory, a memory system that is viewed to contain generalized knowledge (e.g., Tulving, 2002).
Talmy's work on prospective and retrospective directions of viewing is largely neutral to whether these viewings take place in an autobiographical or fictive realm and also includes temporal deictic centers other than the present moment (2000, p. 86). This neutrality in terms of the temporal deictic center allows us to characterize mental projection over time not only into the future and the past but also more generically into ‘‘earlierness” (anteriority) and ‘‘laterness” (posteriority) (cf. also Núñez & Sweetser, 2006, p. 404), a topic that will frequently reoccur throughout the article. Additionally Talmy's conceptualization of temporal retrospective or prospective viewings cannot only be applied to conceptual thought that memory researchers would classify as ‘‘episodic,” but—as will be shown with various specific examples in this article—also to conceptual thought that most memory researchers would probably classify as ‘‘semantic.” So Talmy's conceptualization of mental projection over time can be said to be ‘‘episodic–semantic neutral.”
Clark's (1973) work on moving time and moving ego metaphors, while also largely neutral to whether the mental conceptualization is set in an autobiographical or fictive realm, mainly includes only the present moment as a possible temporal deictic center and seems to mainly involve episodic memory. But despite their differences, the temporal conceptualizations of Tulving (1985, 2002), Talmy (2000), and Clark (1973) all involve the basic idea of mental projection over time and these three investigative lines are all, either explicitly or implicitly, sometimes also characterized in terms of mental imagery.
No one ever proposed a fundamental distinction between these proposals by Tulving (1985, 2002), Talmy (2000), and Clark (1973). The differences in terminology rather seem to stem from the fact that these analyses grow out of different research traditions or research backgrounds. But if the field of cognitive science is to move on in relation to investigating the properties of mental time, the time has perhaps come to theoretically unify them to a certain degree. Consequently, I will unify these three investigative lines for the current article in two points: (a) the term self is used in the sense of Tulving, but additionally as what otherwise might also be referred to as ego (conceptual metaphor theory), observer (conceptual metaphor theory), or conceiver or ceiver (Talmy); and (ii) the term mental time travel is also used in the sense of Tulving and other memory researchers, but additionally as what otherwise might also be referred to as a viewing through mental time in a prospective or retrospective direction (Talmy, as discussed above) or what might involve a projection, for example, into a metaphorical aheadness to conceptualize a moving or stationary future time (conceptual metaphor theory, as discussed above). Additionally as mental projection over time can also be identified in semantic memory, I use the term mental time travel in episodic–semantic neutral terms (as just discussed above). This means that in this article the term mental time travel is used quite generally for the idea of mental projection over subjective time—no matter if this projection takes place in an autobiographical or entirely fictive realm and no matter if the deictic center is the present moment or some temporal point other than the present moment and no matter if the episodic or semantic memory system is involved. If a need for these specifications arises, I simply speak of autobiographical mental time travel versus fictive mental time travel; or, in relation to the temporal deictic center, I speak of mental time travel into the past or the future (present moment as deictic center) versus mental time travel into anteriority or posteriority (deictic center other than the present moment); or in relation to memory systems, I speak of episodic mental time travel versus semantic mental time travel.
One result of unifying the work of Tulving (1985, 2002), Talmy (2000), and Clark (1973) is the idea that mental time travel involves travelling through spatialized time (more precisely, this idea already results when one unifies the ideas of Tulving and Clark, whereas many other novel ideas proposed in this article require the additional consideration of Talmy's ideas). The idea of spatialized mental time travel has recently in relation to episodic memory also been expressed by Merritt, Casasanto, and Brannon. In this context, they speak of the ‘‘unexplored link between the human capacity to spatialize time and our ability to form episodic memories” (2010, p. 201). Section 5.4 discusses specifically how the current investigation can conceptually refine this thus far unexplored link.2
With this unification of the work of Tulving (1985, 2002), Talmy (2000), and Clark (1973) in hand, in the rest of this article I am now able to propose a comprehensive account of the spatialized imagery structuring of time that is proposed to make up the time machine in our mind. A basic theoretical framework that can take on the challenge to describe this sophisticated mental imagery machinery in a concise way is what I refer to as Talmyan concept structuring (Talmy, 2000). The properties of Talmyan concept structuring that are needed for mental time are briefly outlined in Section 2. We then (Section 3) first use linguistic examples of two distinct events temporally related within a single clause—linguistic cross-related events in a single clause (Talmy, 2000, pp. 347–349)—as an entree to an underlying imagistic mental machinery that allows us to travel through time. This leads to temporal conceptualizations that I term mental time watching and optional mental time travel. Next (Section 4), we use linguistic examples of two distinct events temporally related across two clauses—linguistic cross-related events in a complex sentence—as an entree to underlying time travel. This leads to a temporal conceptualization that I term obligatory mental time travel. Linguistic cross-event relations (Talmy, 1978, 2000, Chapter 6) are a useful entree to mental time because such linguistic temporal relations are additionally also one possible entree to more complex, embedded mental time structures (Sections 3.4 and 4.2). In Section 5, the mental time machine account is examined in relation to the linguistic tense/aspect distinction, specific empirical predictions in relation to cognitive and neural processing, human versus animal cognition, and applied clinical potential.
The proposed spatialized temporal imagery structuring system is to a certain degree akin to what Kosslyn and colleagues call (albeit in relation to spatial imagery per se and not in relation to spatialized temporal imagery) categorical spatial relations (Kosslyn, 1987, 1994; Jacobs & Kosslyn, 1994). To give one example: a spatial relation like ‘‘being behind” can occur in the categorical spatial relations system (cf. Laeng, Okubo, Saneyoshi, & Michimata, 2011) and (in spatialized temporal form) in the temporal relations described in this article (see Section 3.2). However, this article is basically set in a Talmyan theoretical framework (Section 2) and to unify the current findings with the theoretical work by Stephen Kosslyn goes beyond the scope of the present investigation, although a unification of this sort might indeed be a promising endeavor. At this point, let it just be stated that the readiness with which mental time conceptualization is described in terms of imagery structuring opens up the possibility that the imagistic qualities of this temporal structuring system are not merely epiphenomenal but play a functional role. This is somewhat in contrast with Kosslyn, who states that ‘‘the categorical spatial relations encoding subsystem produces a ‘spatial code’ that specifies a categorical relation between two or more objects . . . These spatial codes, unlike the coordinates produced by the coordinate spatial relations encoding subsystem, are propositional representations” (Kosslyn, 1994, p. 194). In the current work, temporal relations are proposed to involve abstract spatialized imagery and no need arises to assume a propositional ‘‘temporal code” or ‘‘spatialized time code.” Some ideas for experiments that could test if the proposed mental time imagery structuring does have functional, nonepiphenomenal mental status will be presented in Section 5.
2. A theoretical apparatus for the projection of the self over time: Talmyan concept structuring
Based on decades of research, Talmy has sketched a comprehensive theory of concept structuring with several cognitive schematic systems, the main ones being configurational structure, perspective, attention, and force dynamics (Talmy, 1988, 2000, Chapter 1). Each schematic system in turn consists of various schematic categories. The projection of the self over time—our concern—can mainly be covered with the Talmyan schematic categories perspectival location, perspectival distance, and direction of viewing (from the perspective system); degree of extension (from the configurational structure system); and Figure, Ground, and ReferenceFrame (from the distribution of attention system) (Talmy, 1988, 2000). The descriptions in this section of these schematic systems (including the linguistic examples) mainly draw from Talmy (2000). The overall hierarchical schematic structure is outlined in Fig. 1. Note that this is a selective representation. Only the schematic systems and schematic categories that are mainly needed to describe the projection of the self over time have been outlined. In this section, they are briefly described and related to mental imagery. We could also say that in this section we are assembling the imagery structuring parts that we need to describe the mental time machine and its use by the self.
Talmy introduces the schematic system perspective in the following way:
The present schematic system consists of the perspective that one can have on [a referent] entity, as this is specified by closed-class forms. This system thus establishes a conceptual perspective point [PP] from which the entity is cognitively regarded. While this schematic system is presumably neutral to particular sensory modalities, it is most readily characterized in visual terms as, in effect, pertaining to where one places one's ‘‘mental eyes” to ‘‘look out” upon a referent structure.
The perspective system covers several schematic categories. Included among these categories are ones pertaining to: a perspective point's [PP] spatial or temporal positioning within a larger frame, its distance away from the referent entity, its change or lack of change of location in the course of time and the path it follows with change, and the viewing direction from the perspective point [PP] to the regarded entity (2000, p. 68).
These ‘‘mental eyes” are sometimes also characterized by Talmy as belonging to a cognitive ‘‘conceiver” or ‘‘ceiver” (Talmy, 2000, p. 313 and 158)—to which we refer as self in the present theoretical approach (cf. Section 1). In the unified theoretical framework of this investigation, perspective then involves the temporal positioning of the self and its PP within a larger temporal frame, from where the self uses its ‘‘mental eyes” to look out on a temporal referent structure. In order to be able to characterize perspective in relation to spatialized mental time imagery structuring, we mainly need the perspective categories perspectival location, perspectival distance, and direction of viewing.
2.1.1. Perspectival location
The schematic category perspectival location has two main member notions: exterior and interior (Talmy, 2000, pp. 68–69). Looking at how Talmy has characterized this in relation to the domain of space will later on (from Section 3 onward) assist us in describing spatialized temporal perspectival location. Talmy observes cognitive calculations that
. . . appear to combine a rule of English with geometric knowledge. Though often breached, an apparent general rule in English is that if the initiator of an event is visible, it must be included in the clause expressing the event, but if not visible, it must be omitted. Thus, if a glass I am holding slips from my hand, I can felicitously say to a bystander I dropped the glass, but not The glass fell (2000, p. 69).
In this example, Talmy shows that these cognitive calculations can involve visual perception, involving the visibility of the event-initiator (the glass dropper) and the event (the dropping of the glass). But with the following examples Talmy also shows that the same cognitive calculations can occur in visual mental imagery underlying language (2000, p. 69):
1Spatial perspectival location
a. Exterior: Two men slowly opened the lunchroom door and walked in.
b. Interior: The lunchroom door slowly opened and two men walked in.
In (1a), the event initiators (the door openers) are mentioned in the same clause as the event (the opening of the door), so it induces imagery where the event-initiators are visible to the self when the event is initiated: The only perspectival location (assuming opaque walls and door) where door-opening initiators who are entering a room can be visible at the same time as the door starts to open is a location that is outside the room. In (1b) on the other hand, the door-opening event initiators are not mentioned in the same clause as the door-opening event, so it induces imagery where the event-initiators are not visible to the self when the event is initiated: The only perspectival location where door-openers who are entering a room cannot be visible at the same time as the door starts to open is a location that is inside the room.
2.1.2. Perspectival distance and degree of extension
The schematic category perspectival distance (from the perspective system) and the schematic category degree of extension (from the configurational structure system) frequently correlate in specific ways (Talmy, 2000, pp. 61–62 and 69–70); for this reason, they are presented together here. An imagistic entity can be visible to the self from three distances: medial, distal, and proximal. A medial perspective correlates with a median degree of extension, a distal perspective with a reduced degree of extension, and a proximal perspective with a magnified degree of extension. Talmy illustrates the interaction of these categories in relation to the domain of time with the following examples (2000, pp. 61–62):
2Interaction of temporal perspectival distance and temporal degree of extension
a. Medial/bounded: She climbed up the fire ladder in 5 minutes.
b. Distal/point: Moving along on the training course, she climbed the fire ladder at exactly midday.
c. Proximal/unbounded: She kept climbing higher and higher up the fire ladder as we watched.
The event referent of climb a ladder seems to basically consist of a bounded temporal extent. The temporal boundedness is manifested by the fact that the event referent can be used with the grammatical form ‘‘in + NPextent-of-time,” as (2a) shows. A temporally bounded event requires a medial perspectival distance: At this perspectival distance the self takes on a PP that has neither zoomed in as much into a bounded event as to obliterate its boundaries, nor has it zoomed out as much from a bounded event as to collapse it to a single point (on mental zooming see, e.g., Kosslyn, Thompson, & Ganis, 2006). Rather the self establishes a PP at a distance that allows for mental perception of both ends of the bounded temporal event.
In language (by the means of using corresponding closed-class forms), we can also observe that an event with a bounded extent is shifted toward a conceptual schematization as a point in time (Talmy, 2000, pp. 61–62). This reconceptualization process is called reduction.3 Reduction in relation to (1b) can, for example, be performed in conjunction with the grammatical form ‘‘at + NPpoint-of-time” (and different contextual specifications, as (2b) shows). Reduction typically involves the adoption of a distal perspective: An event normally stretched out from a beginning to an end boundary in time is imagistically regarded by the self from a distance that reduces the extent to no more than a speck.
Additionally, the cognitive operation that allows a shift in the cognized extension can also go the opposite way from reduction: The event referent with the unmarked form of a bounded extent can conceptually be reschematized to a referent with an unbounded extent, a cognitive operation called magnification (Talmy, 2000, p. 62).4 In language, magnification can, for example, be observed in relation to grammatical forms such as ‘‘keep-ing,”‘‘-er and -er,” and ‘‘as + S,” as (2c) shows. Magnification typically involves the adoption of a proximal perspective: An event normally stretched out from a beginning to an end boundary in time is imagistically regarded from such a proximity that the existence of these boundaries falls outside of view and attention.
2.1.3. Direction of viewing
In the domain of time, the schematic perspectival category direction of viewing involves the conceptual possibility of the self viewing in temporal directions from an established temporal PP (Talmy, 2000, pp. 72–76). Talmy has identified three directions: direct, prospective, and retrospective. A helpful way to illustrate direction of viewing in relation to these examples is to use the diagrams that Talmy provides in relation to them. They are presented in Fig. 2 and they represent faithful reproductions of Talmy's (2000, p. 73) own diagrams.5 The corresponding example sentences in (3) have also been taken over.
3Temporal direction of viewing
a. Direct/prospective: I shopped at the store before I went home.
b. Retrospective/direct: After I shopped at the store, I went home.
Although Talmy (2000, p. 73) has not explicitly declared these diagrams to represent mental imagery structuring, he has done so at least implicitly. Recall according to Talmy ‘‘mental eyes” are meant to ‘‘look out” from the PP upon a ‘‘referent structure” (2000, p. 68). Accordingly, the arrows coming out of PP are meant to represent the mental ‘‘looking out,” called a ‘‘line of viewing” in the context of these diagrams by Talmy (2000, p. 73). The looked-upon ‘‘referent structure” is the timeline and the events thereupon (events A and B). In accordance with the theoretical unification outlined in Section 1, we can assume that the self is carrying out these lines of viewing.6
The example of diagram 2a shows how the temporal subordinate conjunction before is involved in the imagistic conceptual structuring of time (Talmy, 2000, p. 73). A temporal deictic center is established in relation to the event in the main clause (event A). From there, the self first directs a line of viewing at event A itself. This viewing is ‘‘temporally direct,” as the temporal deictic center and event A coincide temporally. The self then directs a second line of viewing from the temporal deictic center in a prospective direction, ahead to event B. In the terminology of the current article, the self has undertaken mental time travel into posteriority, an issue that we take up systematically in Section 4. The prospective viewing is grammatically achieved by putting the event in a temporal subordinate clause; the temporal conjunction before gives the indirectness its prospective direction. The example of diagram 2b shows the concept structuring imagery of time in relation to the temporal subordinate conjunction after (Talmy, 2000, p. 73). Here, a temporal deictic center is established at event B. From there, the self first directs a line of viewing retrospectively at event A (the indirect viewing is again grammatically achieved by putting the event in a temporal subordinate clause, but this time the temporal conjunction after gives the indirectness not a prospective, but a retrospective direction). The second line of viewing is this time the direct viewing (grammatically again achieved by putting the event in the main clause). Let us also note that the diagrams of (3) are simplified in relation to temporal PPs (here as well as in the original diagrams of Talmy). As the events are set in the past tense, they additionally involve a PP at the present moment of speech, with a retrospective look back to the deictic center in the past (cf. Talmy, 2000, p. 86). Such embedded structures (here embedded temporal PPs) are one possible form of cognitive nesting structures (Talmy, 2000, pp. 86–87). In the theoretical framework of this article, this is actually a case of ‘‘mental time travel within mental time travel,” as the self once travels prospectively from a temporal deictic center to a more posterior point in time (time travel underlying before) and once retrospectively from the present moment to a point in the past (time travel underlying the past tense). A closer look at the imagery of embedded mental time travel is taken in Section 4.
2.2. Figure, Ground, and Reference Frame
Central to the schematic system distribution of attention are the ‘‘fundamental cognitive functions” (Talmy, 2000, p. 311) of Figure and Ground.7 Talmy has specified the basic functions of Figure and Ground for the domains of both space and time (2000, Chapter 5). In space, the Ground is a reference object relative to which the path, site, or orientation of another object, the Figure, is determined. In time, the temporal Ground is a reference event relative to which the location in time of another event, a temporal Figure, is determined. Talmy also observes that temporal Figure and Ground always additionally relate to a Reference Frame.8 Let us nevertheless also look at how Talmy has characterized the Reference Frame in relation to spatial Figure and Ground, as this will be directly relevant when we look at how a temporal Reference Frame is spatialized in relation to cross-related events in a single clause (see Section 3). Consider the following spatial example by Talmy (2000), p. 311):
4Figure in motion, Ground, and Reference Frame The pen rolled off the table.
(4) involves a Figure in motion. As Talmy describes using this example, as soon as motion of the Figure is involved some spatial background in addition to the Ground must be considered:
For . . . if an observer (or conceiver) has in sight (or mind) only the Figure object [the pen], she can know only that the object exists, but nothing of change of position. Even when . . . the observer sees both Figure and Ground objects [pen and table]—still without any reference frame, however—she can additionally know only that there is a change from the two object's being together to their being apart, but could not know which object moved (or if both moved), nor whether there is any further motion once the two objects are apart, as there is no way to determine (change of) distance. Only when the observer sees both objects within a framework . . . can she know which object is stationary, which object moves, by how much, and along what path. (Talmy, 2000, p. 313)
This example is used by Talmy to show that for any notion of Figure motion, there must be a Ground and a Reference Frame. The Reference Frame encompasses the region through which the Figure moves (in (4), e.g., a region extending from a point where the pen is rolling on the table to a point where the pen is ‘‘being off the table”). When there is motion of the Figure, we can thus define the Reference Frame as the region that encompasses the Ground as well as the region through which the Figure moves. This spatial definition of the Reference Frame will become relevant for the domain of time when we investigate mental time watching of temporal Figure and Ground in motion (Section 3). However, in time, Figure and Ground—as long as two events are related to each other—always have a Reference Frame, even when there is no motion involved (Talmy, 2000, p. 320).This will become relevant when we investigate obligatory mental time travel (Section 4), as this often involves static temporal Figure and Ground.
In this section, we have been assembling the cognitive parts from the Talmyan concept structuring system that are needed to describe the mental machinery that allows for mental projection over time—the parts that are needed to describe the time machine in our mind. Let us then start to use language about time as a window into this machine.
3. Cross-event relation in a single clause as an entree to mental time watching
In this section, I will analyze sentences containing temporally cross-related events in a single clause as an entree to mental time imagery structuring that is proposed to underlie such linguistic forms. Cross-related events are concerned with sentence structures that relate a Figure event to a Ground event (Talmy, 1978, 2000, Chapter 6). Cross-event relation in a single clause represents, as will be argued, a linguistic entree to a mental imagery conception where the temporal perspectival location of the self is principally unspecified, that is, the self is free to take on either an exterior or interior perspectival location in relation to the temporal Reference Frame. If the self takes on an exterior location it engages in what I term mental time watching, a mental situation where the self stays outside a spatialized temporal extension (outside a spatialized temporal Reference Frame) and from there looks at the spatialized temporal extension and the structure therein. If the self—in the very same spatialized temporal extension—takes on an interior location, it engages in mental activity for which one can use Endel Tulving's term mental time travel (Tulving, 1985, 2002): here characterized as a conceptualization where the self is within a spatialized temporal extension (within a spatialized temporal Reference Frame) and is retrospectively or prospectively looking through mental time (cf. Section 1). This mental time conceptualization underlying cross-event relation in a single clause—where the self is free to either engage in mental time watching or in mental time travel (optional mental time travel)—is quite distinct from the mental time conceptualization that will be discussed in the next section (Section 4). There the mental time conceptualization underlying cross-event relation in a complex sentence is investigated, a conceptualization where the self—at least in the cross-event relations investigated there—cannot take on an exterior perspectival location in relation to the temporal Reference Frame but must take on an interior one and additionally must engage in mental time travel (obligatory mental time travel).
3.1. Mental time unfolding symmetrically before motion applies
Let us start using language as a window to uncover mental time imagery structuring by considering the following sentence9:
5 New Year's follows Christmas.
The temporal sense of follow is generally treated as involving a space-to-time mapping from the spatial sense of follow (Moore, 2006; Núñez & Sweetser, 2006). In accordance with this approach, I will point out some mapping aspects that are particularly relevant to possible underlying mental time imagery. One fundamental observation that linguistic analysis can assist us to make is to note that the sense of spatial follow, which is mapped onto time basically involves the following conceptualization: Figure and Ground move in the same direction, at roughly the same speed, one being behind the other (Moore, 2006; Núñez & Sweetser, 2006; the relevant analysis supporting this claim also follows below in this section). As Talmy observes (2000, pp. 336–337), for cases where Figure and Ground move in the same direction at roughly the same speed, we can at the most basic level, despite the motion, identify a static component, namely the constant distance between the moving Figure and the moving Ground. Thus, although the conceptualization of follow involves motion, it nevertheless has a static base. This static base has been mapped from the spatial sense of follow (as described above) to the temporal sense of follow (dynamic properties have also additionally been mapped; but this is considered in the next section). Considering Talmyan degree of extension (cf. Section 2.1.2), this static base of follow involves schematizing Figure and Ground to nothing more than points; that is, there is nothing in the static base of follow that would require that Figure or Ground is extended. Linguistic evidence can be found in the spatial sense of follow10:
a. One dot followed another dot across a screen.
b. *The speck lay across the ruler.
c. *The matchstick lay across the speck.
(6a) is an acceptable English sentence because the concept of a dot can be schematized to a point. follow can thus accommodate a Figure point and a Ground point and nothing more needs to be assumed about the basic Figure and Ground geometry that follow calls for. This is in contrast with, for example, a conceptualization underlying an expression such as lay across where neither Figure nor Ground can be schematized to a point (cf. Talmy, 2000, pp. 189–192), as examples (6b) and (6c) show. In relation to the Reference Frame—for the static base of follow—nothing more needs to be assumed than a line: A line is enough to encompass the region of two points (see also Herskovits, 1986, p. 51). Note that it is not a directed timeline, since in the static base nothing calls for the need to schematize direction. The need to schematize direction arises as soon as motion is additionally considered; this is the issue we will turn to in the next section (Section 3.2). The proposed mental imagery structuring of the static base of follow (as mapped from space onto time) is shown in Fig. 3.
In accordance with the linguistic analysis just undertaken, Fig. 3 proposes the mental existence of the following mental time imagery structuring: a mental line with two mental points on it. In this abstract static base, we can observe mental time unfolding symmetrically: There is no notion of before or after nor of the past or the future—yet. Temporal asymmetry arises only once the motion component is added to the structure (see next section).11 The line functions as a temporal Reference Frame and lets mental time unfold on it as a temporal apartness between the Figure event and the Ground event. Having adopted the assumption that the line and the two points of Fig. 3 are nonepiphenomenal, functional imagery structures (cf. Section 1), we must address the issue that the mind's eye normally never ‘‘sees” such abstract structuring units. A solution is to adapt the Talmyan term of sensing to capture this mental phenomenon (Talmy, 1996, 2000, pp. 146–153). According to Talmy a sensed entity
. . . is experienced as present in association with other entities that are seen at the fully concrete level, but it itself is intangible and nonmanifest, as well as vague or indefinite and relatively faint. It has little or no ostension, and with no quality of direct visibility. In viewing a scene, one's experience is that one does not ‘‘see” such an entity explicitly but rather ‘‘senses” its implicit presence. (2000, p. 146)
Talmy additionally points out that such entities, before they are sensed in a given perceptual situation, often exhibit topological properties:
[For instance] one may sense as equal instantiations of an ‘‘across” schema both the path of an ant crawling from one side of one's palm to the opposite side, and the path of a deer running from one side of a field to the opposite side. This visually sensed ‘‘across” schema would then exhibit the topological property of being magnitude neutral. Comparably, one may equally sense an ‘‘across” schema in the path of a deer running in a straight perpendicular line from one boundary of a field to the opposite boundary, and in the path of a deer running from one side of the field to the other along a zigzag slanting course. The visually sensed ‘‘across” schema would then also exhibit the topological property of being shape neutral. (2000, p. 149)
As these quotations show, Talmy uses the term sensing exclusively in relation to concurrent online sensory stimulation (so, e.g., one can visually sense the across schema while actually looking at an ant that is crossing one's palm). Clearly, however, the term of sensing can also be applied advantageously in mental imagery that occurs independently of online sensory stimulation. If, for example, one imagines an ant crossing one's palm, one might mentally see the ant crossing the palm but might also additionally mentally sense the across schema in the visual imagery. Thus, I use the term online sensing for the sensing described by Talmy and introduce the term mental sensing for sensing, that takes place in mental imagery. As an umbrella term for both types of sensing, I simply use sensing (thus, this term is used here more generically than it is used by Talmy). Note that visual sensing implies that the structures thus sensed are invisible (‘‘with no quality of direct visibility,” as Talmy says in the quotation above): While they are spatially relevant (since they act as a spatial scaffolding across which contentful material can be splayed), they are neither detectable by the physical nor the mental eye.
Whereas in schematic space sensing can occur in relation to concurrent online sensory stimulation as well as without it (as has just been exemplified by across), schematic time is seemingly never directly produced by online sensory stimulation.12 From this, it follows that the sensing of mental time can only co-occur with mental seeing (and not with any form of online seeing). We can also note that mental time structuring exhibits the topological property of being magnitude neutral. The temporal sense of follow can demonstrate this point:
7 Magnitude neutrality of temporal concept structuring, exemplified by follow
a. New Year's follows Christmas.
b. The Mesolithic era follows the Paleolithic era.
Let us now see what happens when motion is added to the static base in follow.
3.2. Mental time unfolding asymmetrically into anteriority and posteriority after motion applies
follow demands of its Figure and Ground to be both in motion. Any attempt to stop one of them or both of them from moving results in semantically unacceptable results, as the spatial examples in (8b–d) demonstrate.
8 Obligatory motion of Figure and Ground in follow
a. I followed him on skis as he was running through the snow.
b. *I followed him, standing in the grass.
c. *While he stood still, I followed him.
d. *While we both stood still, I followed him.
Additionally encoded in follow (in space and mapped onto time) seems to be that the two objects in motion keep their distance between them roughly constant as has already been suggested above. Linguistic evidence for this comes from the observation that follow does not seem to be compatible with constructions where the distance between the moving Figure and the moving Ground changes. If such a distance changes, we must revert to different linguistic means, such as using the phrase catch up:13
9 Constant distance between the moving Figure and the moving Ground in follow
a. He caught up with the leader on the last lap of the run.
b. *He followed up with the leader on the last lap of the run.
As Talmy (2000, pp. 336–337) has demonstrated, when Figure and Ground move in the same direction at roughly the same speed they form (together with the static path between them) a Composite Figure that as a whole moves in relation to yet another Ground.14 This second Ground functions at the same time as the Reference Frame. In spatial follow that Reference Frame is the path that Figure and Ground (as a whole) follow along—the temporal Reference Frame for follow is discussed below in this section. The motion in follow furthermore brings in the notion that Figure and Ground are conceived as having fronts and backs. It is well established that observing an isotropic object in motion—such as a point or a cube—imputes (in the onlooker) a front onto the leading part of the moving object (and correspondingly a back onto the trailing part of the moving object), although an isotropic object lacks an inherent front (e.g., Núñez et al., 2006). This is a case of imputed asymmetric geometry (cf. Talmy, 2000, pp. 210–211); the asymmetry (here frontness and backness) is not inherent in the structure but arises as a perceptual phenomenon only when an additional factor (here motion) comes into play. The motion not only imputes fronts and backs to the inherent isotropic structure of the Figure and the Ground, on a Figure-and-Ground interrelational level it also establishes that the leading event is conceived as being ahead of the following event. Thus, the Figure (the follower) is following behind the Ground (the followed). Talmy has characterized such behindness involving Figure–Ground interrelations as cases where the Ground's ‘‘back”‘‘extends out” and the Figure ‘‘is localized” in relation to this extension (Talmy, 2000, p. 210). As these two cognitive processes play a prominent role in the current investigation and as Talmy has not introduced technical terms for them, I introduce two terms for them: Ground radiation and Figure localization. Thus, in (5) the Ground (Christmas) radiates out a behindness (cf. also Núñez & Sweetser, 2006, p. 404) and the Figure (New Year's) localizes itself in relation to it.
When this imputed asymmetric geometry gets mapped from space to time in follow (and conceptually related words, see further below in this section) frontness signifies ‘‘earlier” (anteriority) and backness signifies ‘‘later” (posteriority) (Moore, 2006; Núñez et al., 2006; Núñez & Sweetser, 2006). Some ideas have been put forward how this mapping of frontness onto anteriority and backness onto posteriority can take place. But the precise mental mechanisms behind this mapping have not been identified yet.15 Because frontness is mapped onto anteriority, Figure and Ground are also conceived as moving toward anteriority (as their fronts move in this direction). For this reason, the timeline is no longer symmetric, as it is in the underlying static base (cf. Fig. 3). Now it is perceived as being directed—toward anteriority in the direction of the forward motion and toward posteriority in the opposite direction. Note, however, that this anteriority–posteriority timeline cannot serve as a Reference Frame as the nondirected timeline underlying the static base could (Section 3.1). A Reference Frame encompasses all of Figure and Ground (Section 2.2). As Figure and Ground now (in the perceptual phenomenology of the self) have imputed fronts and backs, they have to be schematized at least as two-dimensional structures, as fronts and backs cannot be conceptualized within a one-dimensional extent. Thus, there is a need for a two-dimensional Reference Frame that can encompass these structures. The most parsimonious schematization is to assume that Figure and Ground move through a ‘‘time ribbon”—that is, a plane with two parallel line edges that have an anteriority/posteriority-directed timeline as their axis. Figure and Ground move along this axis and their fronts and backs create a minor extension into the second dimension, which results in an overall ribbonal geometry of the Reference Frame. To use an analogy, it is to a certain degree as if the timeline is a wire and Figure and Ground are beads that move along this wire. The beads cannot move in the wire; they need some minor space around the wire, space that at least extends as far as the beads themselves do in diameter. The beads-moving-on-a-wire analogy works quite well, except that it assumes a three-dimensional format, whereas the temporal Figure-and-Ground-moving-on-the-timeline conceptualization can be schematized to a two-dimensional format. This proposed two-dimensional mental imagery structuring underlying follow—revealing mental time unfolding asymmetrically into anteriority and posteriority after motion applies—is shown in Fig. 4.
In accordance with the linguistic analysis undertaken in this section, Fig. 4 then proposes the existence of the following invisible mental time imagery structuring (once mental motion applies to the static structure of Fig. 3): A mental axis of a mental ribbon that is directed from anteriority to posteriority, two front/back-imputed moving dots with Figure and Ground functions, a mental ribbon functioning as a Reference Frame, and some mental Figure-and-Ground interrelational structures that also presuppose an unfolding in spatialized mental time (Composite Figure, Ground radiation, and Figure localization). Note that the lines depicting the perpendicular distance between the two parallel line edges of the ribbon are not meant to indicate that the time ribbon is temporally bounded. If the distance between the lines is not pictorially represented, then, especially in more complex diagrams still to follow in this article, it would be hard to still recognize the two parallel line edges as belonging to a ribbon. This is the only reason why they are depicted in the diagram. The linguistic analysis that I am undertaking in this article never forces me to choose whether the timeline and the time ribbon are bounded. Thus, the question of the degree of extension of the timeline and the time ribbon itself—rather than the question of the degree of extension of temporal regions on the timeline (cf. Section 2.1.2)—is left intentionally vague (cf. with the comments of Comrie, 1985, pp. 2–3).
Figure 4 then shows what is proposed that the self senses when cognizing the temporal meaning of follow (as in New Year's follows Christmas). But where is the self in all this? This is the question we turn to next.
3.3. The self's options between mental time watching and optional mental time travel
As we have seen, in a temporal conceptualization underlying follow (and comparable linguistic structures, see below in this section) one event is temporally located in relation to another event. This means that the self is not an inherent part of the temporal Reference Frame, as it is, for example, the case in a conceptualization underlying the use of the past tense, where the self naturally is located at the present moment (cf. Comrie, 1985; Talmy, 2000, pp. 86–87; see also Sections 3.4 and 4.2). That is why the few researchers who have investigated this particular type of conceptual time metaphor—a metaphor where an event is defined in relation to another event without reference to the self's positioning in time—call such constructions ‘‘Ego-free” (Núñez et al., 2006) or ‘‘perspective-neutral” (Moore, 2006). Using the unified theoretical framework outlined in Sections 1 and 2 allows us to refine these observations of being ‘‘Ego-free” or ‘‘perspective-neutral” to some degree. We can state that in these constructions the self can either take on a perspectival location that is inside or outside the temporal Reference Frame and that there is no a priori way of predicting if the self will be exterior or interior. The only prediction one can make is that the self must be somewhere (otherwise there would be no cognizer) when temporal follow evokes a mental representation in a listener. The only two basic perspectival locations that a self can adopt in relation to a temporal Reference Frame are to be either inside or outside this Reference Frame.
First, we look at the possibility that the self can take on a perspectival location that is outside the temporal Reference Frame. In such a case, the Reference Frame is indeed ‘‘Ego-free,” as Núñez et al. (2006) put it (what we could call ‘‘self-free” in the current terminology of this article). This possibility is shown in Fig. 5. This imagery proposes the mental existence of a self that can be outside a sensed mental ribbonal extent that functions as a Reference Frame. From this exterior vantage point, the self can look at contentful material and at the same time sense the temporal scaffolding across which the contentful material is splayed. In case the contentful material is of propositional format (rather than of a depictive imagistic format) then the self might not mentally see any content. However, according to the present theory that imagery structuring underlies concept structuring of language, the temporal scaffolding should still be sensed, even when the contentful material is propositional. This claim leads to specific predictions for mental simulation and neural processing (see Section 5). In Fig. 5, the mental visual field that allows for mental looking at content (in case there is any depictive content), and also at the same time allows for concurrent mental sensing, is symbolized by the shaded area coming out of the self. When the self is establishing an exterior perspectival location and from there looks at a temporal Reference Frame (and its internal structure and content), I term this mental time watching.
While it has just been demonstrated that a temporal conceptualization underlying follow (and underlying quite generally cross-related events in a single clause, see below) can be ‘‘Ego-free,” as Núñez et al. (2006) put it, there is no principled reason why the same temporal conceptualization could not also be ‘‘Ego-occupied” (self-occupied). The correct observation that the self serves no temporal reference function in such constructions does not provide sufficient logical grounds for this assumption. The self might still be within the temporal construction even if it serves no reference function. If we take the prototypical example of this section—the sentence New Year's follows Christmas—we can, for example, imagine someone saying this sentence with establishing New Year's Day as a temporal deictic center—since we know that the Figure event must sometimes even serve as the temporal deictic center in cross-related events (see Section 4). I cannot think of principled reasons why it should not be possible that the self can co-locate itself with the Figure event in a nonobligatory situation. If we allow for this possibility, then we can assume that a listener, when processing the sentence New Year's follows Christmas, can also co-locate his or her self with the temporal New Year's location. The self would then from there—and through mental time—retrospectively view the Christmas location. Note that in the current theoretical framework (cf. Section 1) this means that the self in such a case mentally travels by mentally looking through time (travelling mentally to the Christmas location by using New Year's as a temporal deictic center). As mentioned, the interior positioning—and thus also the corresponding mental time travel—is optional. When the self is within the spatialized temporal extension of Fig. 4, as shown in Fig. 6, the self uses the spatialized temporal extension, the temporal Reference Frame and its internal structure, as a mental time machine. Looking along it—that is, mental positioning at New Year's and from there looking at Christmas—means to mentally travel through it. It is in this way that the self can use the imagery structuring of Fig. 4 as a mental time machine: The structuring is imagistically constructed in a way that it enables the self to carry out mental time travel by ‘‘entering it” and looking along it. Note that in terms of memory systems this account allows to identify mental time travel in semantic memory, as New Year's follows Christmas expresses generalized knowledge about time and not a specific episodic event (for a further example of mental time travel in episodic memory, see Section 4.1).
As just discussed, the self could just as well have stayed outside the temporal Reference Frame and thereby engage in mental time watching (watching mental time unfolding from an exterior vantage point which requires no mental time travel). Núñez et al. (2006) have not offered a principled account for their concept of ‘‘Ego-freeness” of temporal conceptualization where a Figure event is located in relation to a Ground event, so it seems more appropriate to assume that both ‘‘Ego-free” (self is outside the temporal Reference Frame) and ‘‘Ego-occupied” (self is inside the temporal Reference Frame) temporal conceptualizations are possible. Figure 6 shows in imagistic terms one possibility how the self can be proposed to be within a temporal Reference Frame without taking on a temporal reference function.
Figure 6 proposes mental imagery structuring where the self can be inside a sensed mental ribbonal extent that functions as a temporally directed Reference Frame. One possible interior location, as discussed and shown in this figure, is that the self co-locates itself with the Figure event and from there looks at the Ground event. In Fig. 6, for example, this results in a retrospective viewing. When the self takes on an interior perspectival location and from there looks either retrospectively or prospectively along the directed axis of the time ribbon—and when there is no principled reason why the self could not also have cognized the same temporal structure from an exterior perspective—I call this optional mental time travel. Next we will look at obligatory mental time travel.
3.4. Mental time watching or optional mental time travel embedded within obligatory mental time travel
If we take away the use of the past tense for a moment (‘‘Her response follow_ my response”), we get the same underlying imagery structuring as the one proposed in the last section underlying the temporal sense of follow (see Fig. 4) with different contentful material splayed across (here the Figure point is draped with her response and the Ground point is draped with his response). But what happens in the temporal mental landscape if we additionally have to consider the use of the past tense? The use of the past tense establishes a PP of the self at the present moment, which functions as the origin of the retrospective direction of viewing to the past event (cf. Talmy, 2000, pp. 86–87). It is generally acknowledged in episodic memory research, that if one mentally looks into the past, one mentally sees oneself in the past event, looking out from that perspective either in a first person (field) perspective or in a third person (observer) perspective (e.g., Addis et al., 2009; Eich et al., 2009). In the current theoretical framework, the self at the present moment also mentally sees or at least senses the self in the past. But according to the current theoretical framework (see Section 1) such a mental scenario is evoked every time we utter or listen to a sentence that contains the past tense (or other forms in which we might linguistically indicate a concept of past or anteriority), that is, upon hearing a sentence that contains the past tense we mentally simulate the temporal conceptualization (mentally simulate temporal information in the sense of, e.g., Bergen & Wheeler, 2009 and Zwaan, 2008). Our mind simulates the conceptualization of the past, involving the self in the past, no matter if the sentence we are processing relates to our own autobiography or not. In the terminology of the current article, this proposes that processing a past-tense sentence evokes in the listener the simulating of mental time travel into the past. Likewise the theory predicts that linguistic indication of the future concept (in English, e.g., the use of will, going to, etc.) evokes mental time travel into the future in the listener (even within a strict autobiographical realm the concept of mental time travel is usually applied for travel into both the past and future; Addis, Wong, & Schacter, 2007; Addis et al., 2009; Schacter & Addis, 2007a, 2007b; Tulving, 1985). Note that the use of the past or future tense is predicted to obligatorily evoke mental time travel (provided the listener has cognized them in the intended way and processes the relevant temporal information presented to her). It is not like the conceptualization underlying follow, where the self can nonobligatorily be engaged in mental time travel or alternatively be engaged in mental time watching, as discussed in the last section. The difference with the use of the past tense (and the same applies to future forms) is that it requires the self to take on a reference function (the self necessarily co-locates itself with the present moment), whereas in the structure underlying follow there is no obligatory temporal reference function for the self. Of course, if one hears the sentence Her response followed his response in a nonautobiographical context, then one will typically not see oneself (neither in a first- nor in third-person perspective) as part of the temporal conceptualization—one was not there after all. However, the current theory predicts that hearing the sentence, even if it does not relate to an autobiographical event of the listener, nevertheless evokes a mental landscape where the listener's self is ‘‘there” in its own subjective way—cognizing the imagery of Fig. 4 from either an interior or exterior perspectival location. Hearing a sentence such as Her response followed his response might, for example, evoke in the listener a mental image of a woman giving a response after a man gave a response, but the listener herself will in such a case typically not be part of this mental image. But somehow she must still have seen the mental image; a possible explanation is that she ‘‘was there” in a sensed way. In Talmy's words, such a self can be present ‘‘as a fictive projection” (2000, p. 86), which also means that the self can only be mentally sensed, but not mentally seen. Figure 7 shows the proposed mental imagery structuring if the self in the past takes on an exterior perspectival location—which would thus result in mental time watching embedded within mental time travel. The use of the past tense then forces the self to use the mental time machine: The self must enter or be within the spatialized temporal extension and look along it. Underlying the past tense is an extension that—unlike in the last section—does not have an anteriority/posteriority axis, but a past–future axis. The past–future Reference Frame can also be schematized to a ribbonlike extension (a one-dimensional extent would not suffice since the self then could not take on an exterior perspectival location).
In this section, I have mainly analyzed the temporal sense underlying the use of the word follow (without and with the additional use of the past tense). But the same basic imagery structuring (Figs. 3 and 4) and the same possibilities of how the self cognizes these structures (Figs. 5–7) could also be said to underlie cross-related events in a single clause that involve conceptualizing a temporal behindness/aheadness quite generally. follow encoded a notion of temporal behindness. If a notion of temporal aheadness is encoded, this would again basically involve the imagery proposed in Fig. 4, but with Figure–Ground reversals, that is, in such constructions the moving event that is ahead is the Figure event and the one behind is the Ground event (Moore, 2006; Núñez & Sweetser, 2006; for a corresponding diagram, see Fig. S2). Linguistic examples of the latter include The house was completed ahead of time, We will be in Paris in the days ahead of Christmas,17 and so on.18
4. Cross-event relation in a complex sentence as an entree to mental time travel
In this section, the mental time conceptualization underlying cross-event relation in a complex sentence is investigated. In relation to the subordinators before and after, the two structures analyzed here, this leads to a temporal conceptualizaton where—unlike in the last section with follow—the self cannot take on an exterior perspectival location in relation to the temporal Reference Frame: It must take on an interior one and additionally must use the mental time machine to engage in mental time travel (obligatory mental time travel). Just as with optional mental time travel or with mental time watching, the mental time travel underlying before and after can also be further embedded into mental time travel, which has the present moment as its departure point.
4.1. Mental time travel from a temporal deictic center other than the present moment
11 Around here the wind starts blowing before the sun sets.
The following aspects of the temporal imagery structuring underlying the use of before have already been identified by Talmy (2000, pp. 72–76) as well as in Section 2.1.3 of this article: that the self establishes a temporal deictic center that is temporally co-located with the Figure event (the wind's blowing); that the self views the temporal Figure point in a direct viewing and the temporal Ground point in a prospective viewing; that Figure and Ground are located on a timeline that is directed toward posteriority. All that we need to add to this already carried-out description is the Reference Frame. The Reference Frame—as in the last section—opens up into the second dimension. While in the last section the imputed fronts and backs called for a second dimension, it is this time the behavior of the ‘‘self itself” that suggests that we need two dimensions. One first thing to note is that perspectival distance (cf. Section 2.1.2) must be distal, as this is the perspectival distance that allows us to perceive Figure and Ground as points. Only when the events are reduced to points (i.e., if they were not punctual to begin with) can the self from one position observe two events (if the self, say, adopted a medial perspectival distance, then it could only observe one event). Note now that the self co-locates itself temporally with the Figure event from this distal perspective. This means that the self is within the temporal Reference Frame, as it takes on a specific temporal location in order to carry out the direct temporal viewing, but at the same time it is also positioned a ‘‘distal distance” away from the timeline. This removedness from the timeline, which at the same time involves a distal connection to points on the timeline, is what makes the second dimension necessary. The proposed overall imagery structuring underlying (11) is shown in Fig. 8. As example (11) shows, the current theoretical framework also leads to the possibility that the self can be involved in obligatory mental time travel even when there is no future or past involved: Cognizing before in (11) leaves the self no choice but to travel to the posterior event by directing a line of viewing in a posterior direction. Note that in terms of memory systems this means that we—as in Section 3.3—have identified mental time travel in semantic memory, as (11) expresses generalized knowledge about time and not an episodic experience.
4.2. Obligatory mental time travel embedded within obligatory mental time travel
If we—as we have done once before in Section 3.4—take away the use of the past tense for a moment (I shop_ at the store before I go_ home), we get the same underlying imagery structuring as the one proposed in the last section, the one of Fig. 8. Adding the past tense again results in the self positioning itself at the present moment and retrospectively looking back at the past event from there. This then results in the overall mental imagery structuring suggested in Fig. 9. We are basically led to imagery structuring where an anteriority–posteriority Reference Frame is embedded within a past–future Reference Frame. Putting the focus somewhat differently, we can also characterize this temporal conceptualization as mental time travel into posteriority embedded within mental time travel into the past, which indeed shows how complex the time machine in our mind can become. As mentioned, Talmy calls such embedded mental structures ‘‘nested” structures (2000, p. 86). Thus, we might want to refer to ‘‘mental time travel embedded within mental time travel” as nested dual mental time travel in order to make a novel distinction between single mental time travel and nested dual mental time travel. We have encountered single mental time travel in the imagery structuring shown in Figs. 6–8 (one more example of single mental time travel is presented in Section 5.1).
In this section, I have analyzed the projection of the self over time underlying the use of before. The projection of the self over time underlying the use of after is very similar except for the temporal direction of viewing. A sentence such as Around here the wind starts blowing after the sun rises21 involves the same imagery structuring as the one shown in Fig. 8, except that the direct viewing at the Figure event (the blowing of the wind) is not followed by a prospective viewing toward the Ground event, but by a retrospective viewing toward the Ground event (the rising of the sun)—cf. also Section 2.1.3. Similarly, except for these same changes the sentence I went home after I shopped at the store22 involves the same imagery structuring as the one shown in Fig. 9 (for corresponding after diagrams, see Figs. S4 and S5).
In the previous sections a rather comprehensive account of the time machine in our mind and of how this machine is used by the self has been developed by adding the following novel distinctions: external versus internal viewing of time; ‘‘watching” time versus projective ‘‘travel” through time; optional versus obligatory mental time travel; mental time travel into anteriority or posteriority versus mental time travel into the past or future; single mental time travel versus nested dual mental time travel; mental time travel in episodic memory versus mental time travel in semantic memory; and ‘‘seeing” versus ‘‘sensing” mental imagery. I will now discuss the time machine and its use by the self in relation to the linguistic distinction between tense and aspect, specific empirical predictions about mental simulation and neural processing, human versus animal cognition, and applied clinical potential.
5.1. Tense and aspect and mental imagery structuring of time
Future research could investigate the question if the distinction developed in this article between mental time travel and mental time watching is enough to basically capture the conceptual distinction between tense (Comrie, 1985; Pinker, 2007) and aspect (Comrie, 1976; Engelberg, 2004; Pinker, 2007; Sasse, 2002). For example, the conceptual structure underlying a sentence such as She climbed up the fire ladder (cf. Section 2.1.2) also seems—like some of the cross-related events investigated in this article—readily described in terms of mental time watching and mental time travel. Relating to the notion of aspect is the observation that the event seems to be of a bounded temporal duration: In the current framework, this can be characterized as the self engaging in mental time watching from a medial perspectival distance. Relating to the notion of tense is, of course, the observation that the sentence is set in the past tense: In the current framework this can be characterized as the self being at the present moment from where it carries out a retrospective viewing. Thus, a sentence such as She climbed up the fire ladder is readily described as mental time watching (underlying aspect) embedded within mental time travel (underlying tense). Future research could investigate if the conceptual structure underlying tense and aspect is also in general readily characterized as involving spatialized imagery structuring of time. In relation to the tense system, we have already analyzed the future and past tense in this way in the current article. But an interesting question is also if the tense system as a whole, including, for example, formally and conceptually complex forms such as the pluperfect, could be analyzed as involving underlying spatialized imagery structuring of time.
5.2. Mental simulation and mental time travel
Matlock (2004) showed that people mentally simulate motion when processing sentences such as An earthquake fault runs across the valley although the referent described contains no explicit motion (see also Matlock et al., 2005; Ramscar, Matlock, & Dye, 2010). Thus, Matlock has shown that implicit motion (in this case fictive motion; see Talmy, 1996, 2000) is mentally simulated. This suggests that people, for example, conceptualize the motion in the motion verb run upon hearing a sentence such as An earthquake fault runs across the valley in an abstract way. The precise mental mechanisms that are involved in mentally simulating fictive motion are still largely undetermined (cf. Matlock, 2004, p. 1396). Note that the concept of implicit motion (mental motion with no corresponding motion in the spatial referent) could also be applied to motion in temporal imagery structuring, the theoretical account that has been developed in this article. A sentence such as A birch tree stood here twenty years ago does not involve explicit spatial motion either (and not even implicit spatial motion). One single static spatial point is enough for the spatial structuring that is needed to conceptualize the mental space for this scene. However, according to the current framework when one conceptualizes the same scene in relation to mental time, then implicit motion is involved: The self establishes a PP at the present moment from where it looks back to a subjective past moment where the self sees itself (or possibly only senses itself) looking at the tree. The subtle implicit motion to be detected in this conceptualization is the ‘‘traveling” of the retrospective viewing: The viewing is predicted to start off at the present moment and to then move on until it reaches the past event. The implicit motion is basically a form of mental scanning (e.g., Kosslyn et al., 2006), but a mental scanning through time and not through space. The current theory of mental imagery structuring of time, as developed in this article, allows for the prediction that this subtle implicit motion is mentally simulated. In such a case, it should take subjects longer to process sentences with temporally more distant points—longer to process sentences such as A birch tree stood here two thousand years ago or A birch tree stood here a long, long time ago than sentences such as A birch tree stood here two years ago or A birch tree stood here not too long ago. Carefully designed experiments with response time paradigms could investigate if this prediction holds true or not. If it does not hold true (or if it does not hold true in all cases), it is also possible that something like ‘‘mental time jumping” is a further option how the self can be projected over time. Rather than looking along a temporal extension by incremental shifts of attention (scanning), a search for a temporal location within this temporal extension might also involve to mentally look right to the sought temporal point, without any need of scanning along the extension. A comparable phenomenon of ‘‘skipping scanning” can indeed be found in relation to mental space; Kosslyn calls it imagery ‘‘pop out” (1994, pp. 339–341).
5.3. Neural correlates of mental time travel
Little is known about the neural correlates of autobiographical mental time travel, as most studies on episodic memory have not isolated autobiographical mental time travel from other components of episodic memory (Nyberg, Kim, Habib, Levine, & Tulving, 2010). The preliminary findings of a recent fMRI study by Nyberg et al. (2010) suggest that the following regions might be involved in (real or imagined) autobiographical mental time travel: left lateral parietal cortex near the intraparietal sulcus (Brodmann areas 39, 40, and 7), right cerebellum, a midbrain region extending into the bilateral thalamus, and left middle frontal gyrus (cf. also Buckner & Carroll, 2007; Spreng, Mar, & Kim, 2009; Szpunar, Watson, & McDermott, 2007). If the proposal of the current article that mental time travel is mentally simulated whenever we conceptualize a past or future event is true—be it autobiographical or not—then many of the same areas (or whatever the neural correlates of autobiographical mental time travel turn out to be) should also be activated in nonautobiographical mental time travel. Even listening to a totally fictive sentence—say Peter Pan flew to Neverland—should activate many of the same regions, since according to the current framework processing the past tense of this sentence should activate the mental simulation of mental time travel. Carefully designed neuroimaging studies (e.g., fMRI) could provide evidence for this prediction or falsify it. If the prediction holds true, then a sentence with a true autobiographical memory (like I was in X yesterday, where subjects would fill in the X with a place where they really were yesterday) and a nonautobiographical fictive sentence such as Peter Pan was in Neverland yesterday should—once the different conditions such as ‘‘I,”‘‘Peter Pan,” and so on have been controlled for with corresponding subtraction tasks—yield a very similar pattern of neural activation. One would also have to control for perspective (first-person vs. third-person perspective, cf. Eich et al., 2009). The easiest solution would probably be to instruct subjects to see themselves (‘‘I”) in third-person perspective, as this is also the likely perspective from which they would ‘‘see” Peter Pan in their fictive mental time landscape (cf. Section 3.4).
5.4. The debate over mental time travel in animals
It is a question for future comparative research whether animals that exhibit a greater capacity for mental time travel also show a greater dependence of time on space in their more basic temporal representations. Such a finding would suggest an unexplored link between the human capacity to spatialize time and our ability to form episodic memories . . .
The asymmetric relationship we find in humans supports theories of metaphorical mental representation, according to which abstract domains like time are structured, in part, by mappings from more concrete domains like space. By contrast, the symmetric relationship in monkeys is most consistent with ATOM [Walsh's A theory of magnitude], which suggests that space and time are represented by a common metric for analog magnitudes. Together, these data raise the possibility that the capacity to represent abstract magnitudes metaphorically may be uniquely human. (2010, p. 201)
Merritt and colleagues’ conceptualization of mental time travel involves the idea that ‘‘spatial paths can be traveled forward or backward. Once time is conceptualized as a spatial path, then time can be traveled forward or backward—at least in our imaginations” (2010, p. 200; cf. also Casasanto & Boroditsky, 2008; Casasanto et al., 2010). So when they in the quotation above state ‘‘that the capacity to represent abstract magnitudes metaphorically may be uniquely human,” they—in the context of spatialized time—ultimately question if animals can have spatialized temporal paths in which they can travel into a subjective past or subjective future. The current investigation can conceptually elaborate on the idea of Merritt, Casasanto, and Brannon that there is an ‘‘unexplored link between the human capacity to spatialize time and our ability to form episodic memories,” the idea of a spatialized temporal path that can be traveled forward or backward. Refining this thus far unexplored link with the theoretical framework developed in this article, the question is if there are animals that in their minds can enter a spatialized temporal extension (a spatialized temporal Reference Frame) and can mentally look along it (carry out a prospective or retrospective viewing from the present moment) as a means to travel through it. Put succinctly, the question is if there are animals that have a time machine in their minds.
5.5. Cognitive challenges in the mental realm of time and the time machine
In a variety of cognitive disabilities, problems with the acquisition of the past tense have been reported—for example, in children with specific language impairment (Eadie, Fey, Douglas, & Parsons, 2002; Rice, Wexler, & Cleave, 1995; Rice, Wexler, & Hershberger, 1998; Rice, Wexler, Marquis, & Hershberger, 2000; Rice, Warren, & Betz, 2005), Down syndrome (Eadie et al., 2002), autism combined with language impairment (Roberts, Rice, & Tager-Flusberg, 2004), and Williams syndrome (Thomas et al., 2001; Jacobson & Smith Cairns, 2010). However, the attempts to understand these problems almost always take on either a formal (morphological or syntactical) or phonological perspective. The comprehensive conceptual account of the time machine in our mind, as it has been developed in this article, would now also allow a systematic investigation if in some cases not only formal or phonological problems might be involved but also conceptual ones. Perhaps, some of these children with these or other cognitive disabilities might also have problems with acquiring parts of the time machine. Such conceptual problems might (e.g.) relate to the ability to mentally spatialize the unfolding of time as a directed line, the ability to mentally spatialize the present moment as a point on that line, the ability to mentally look back through subjective time from this point (from the present moment), the ability to mentally see or sense oneself again in the past as one looks backward from the present moment, the ability to mentally look from the this past point to a temporal point or extent that is on the timeline.
The theoretical strategy, to use linguistic expressions about time as an entree to conceptual structures about time that seem deeper than language itself, has been applied quite fruitfully, as it has allowed for the development of a rather comprehensive and precise conceptual account of the time machine in our mind. The theory is not an ad-hoc theory, as linguistic conceptualizations cannot be interpreted in a totally arbitrary way—for example, language does not allow us to assume that a sentence such as I shopped at the store before I went home means that first the going home took place and then the shopping. In this respect, the theory is to some degree already a data-guided theory, as linguistic expressions are data. However, the proposal of the theory that language has helped us to uncover a specific system of spatialized imagery structuring of time can only be evaluated by carrying out corresponding psychological (cognitive and neurocognitive) experiments and some ideas for such experiments have been presented. As the time machine in our mind is a deeply fascinating apparatus, I am confident that theoretical and empirical investigations will continue to explore it. Perhaps, some of these studies will also investigate the account of the mental time machine as it has been developed in this article.
‘‘Deixis” and ‘‘deictic center” are terms from linguistics. In its most basic form, deixis refers to a situation where the speaker positions the perspective point (PP) of his or her self at the current location of the speaker (and correspondingly invites the listener to mentally also take on the speaker's location). The term deictic center extends this concept to cover any location—including remembered or imaginal locations—to which the self projects its PP (Zubin & Hewitt, 1995). Generalizing away from exclusive use in linguistics, I use the term ‘‘temporal deictic center” to cover any temporal location to which the self projects its mental PP, no matter if the temporal projection involves verbal or nonverbal (purely imagistic) forms.
Incidentally, it might be noted that the novel idea—that mental time travel requires to travel through spatialized time—has been developed independently by Merritt and colleagues and by myself. They presented the idea in the context of analyzing differences between human and animal cognition. In my case, this idea was a rather natural result of bringing together the work of Tulving, Talmy, and Clark. I learned of Merritt and colleagues’ work when I met one of the authors (Daniel Casasanto) at a workshop in 2010, where in an informal discussion we discovered that we both (i.e., he and his colleagues and I) had come up with this idea independently. Note also the difference that Merritt, Casasanto, and Brannon propose spatialized mental time travel solely in relation to episodic memory, whereas the present account, as has just been discussed, proposes spatialized mental time travel in relation to episodic and semantic memory.
We can note that there are, of course, also events that are cognitively perceived as being temporally punctual per se—that is, that do not have to undergo reduction to become punctual, but that are punctual to begin with. But we should also note that actually not all scholars would agree with this claim, since some challenge the need for a punctual–durative distinction, a distinction that has been extensively employed in linguistic research on aspect. However, Engelberg (2004, pp. 62–65) presents a number of convincing arguments in favor of the punctual–durative distinction, showing a need, for instance, to account for restrictions of valence alternation, restrictions of the progressive, co-occurence restrictions that relate to bounded temporal extents, and iterative interpretation restrictions (see Engelberg, 1999, 2004 for specific examples).
Analogous to the remarks on punctuality in the previous note, there are, of course, events whose degree of extension are unbounded in the unmarked (default) form—that is, that do not have to undergo magnification to become unbounded, but that are unbounded to begin with (Jackendoff, 1991; Pinker, 1989; Talmy, 1988, 2000).
In the diagrams here only one minor adjustment in relation to the original diagrams has been undertaken: The arrows signifiying cosequentiality or antisequentiality have been removed since cosequentiality and antisequentiality, important as they are, need not be discussed in the context of the present article (on cosequentiality/antisequentiality see Clark, 1971; Talmy, 2000, p. 74).
Similar before diagrams that additionally also incorporate the self diagrammatically—and not just the PP of the self—will be shown in Section 4.
Talmy uses the Figure and Ground terms differently from the way they are used in Gestalt psychology. The main difference is that the Gestalt sense involves a perceptual relation and the Talmyan sense a conceptual relation (Herskovits, 1986, p. 197). For some discussion of the differences between the Talmyan and Gestalt analyses, see Talmy (2000, p. 313). I use the terms in the Talmyan sense (as outlined in this section) and therefore I take over the convention from Talmy to write these terms with initial capital letters to distinguish them from the use in the Gestalt sense.
As the term ‘‘reference frame” is used in quite a variety of ways in the cognitive science literature, I use (analogously to Talmy in relation to the terms Figure and Ground) capitals to refer to this term, whenever I use it in direct relation to Figure and Ground.
The generated example sentences that start here have been approved as a set by one native American English speaker and have—where it seemed necessary—been corroborated by others.
Adopting a widely used convention in linguistics, I mark semantically or grammatically unacceptable sentences with an asterisk.
Note that a symmetrical temporal Reference Frame cannot only exist as an abstract base that then turns asymmetrical once motion applies to it. A symmetrical temporal Reference Frame can also exist in its own right and as such can make it all the way to the linguistic surface level. Such a symmetrical temporal Reference Frame directly underlies, for example, a sentence such as Christmas is close to New Year's. Here the Reference Frame only involves temporal proximity (and not temporal direction). The symmetry can be demonstrated by showing that a Figure-and-Ground reversal can still refer to the same temporal scene: for example, one can have the same Christmas and New Year's (say the ones in the year 2010) in mind in Christmas is close to New Year's and in New Year's is close to Christmas (for a corresponding diagram, see Fig. S1). In asymmetric forms—such as conceptualizations underlying follow once motion applies—one cannot have the same Christmas and New Year's in mind after a Figure-and-Ground reversal has occurred: New Year's follows Christmas allows for a conceptualization where both events take place in the same year (say 2010), whereas Christmas follows New Year's only makes sense when the time span of two consecutive years is considered. Of course even in Christmas is close to New Year's we know that Christmas comes before New Year's within a given year. However, we cannot deduce this information from the conceptual structure underlying the temporal use of close; the asymmetry must be added additionally (presumably by adding cultural specific knowledge).
It is Talmy that makes this point, but only in relation to ‘‘the conceptual category of ’tense,’ with such specific member concepts as ’past,’’present,’ and ’future’” (2000, p. 155). However, the point probably holds true for all concept structuring of time.
One can of course ‘‘follow up” concepts that allow one to pursue a solution or conclusion as in I followed up on his progress. But this sense of follow up does not involve the sense of follow that is of concern here (to move in the same direction behind a moving person or a moving object).
‘‘composite figure” is a term from Talmy. He writes it with small initial letters. However, for consistency I write it with capital initial letters, just as is generally done with the terms Figure and Ground (cf. Section 2.2).
Moore might be on a very promising track by theorizing that space-to-time mapping underlying follow is experientially motivated by the following spatial scenario: ‘‘Two entities are going in the same direction on the same path and one is ahead of the other. Wherever they go, the one that is in front arrives first, and the one that is behind arrives later” (Moore, 2006, p. 220). According to Moore, this experiential motivation (on experiential motivation see, e.g., Lakoff & Johnson, 1999) can ‘‘plausibly account for the appearance” (p. 219) of the mapping of frontness onto earlierness (anteriority) and the mapping of backness onto laterness (posteriority). However, the scenario offered by Moore clearly additionally involves a notion of arrival. But schematizing the mental time imagery structure that underlies a sentence such as New Year's follows Christmas leads to no notion of arrival. It is not that the event Christmas arrives somewhere first and stops there; it is rather that the two events, Christmas and New Year's, pass by on a temporal path (the Reference Frame), without us being able to make out a particular notion of coming to a destination (cf. the imagery in Fig. 4). What is still missing in Moore's account then is an explanation of how the notion of arrival gets ‘‘deleted,” in case it is this scenario that gets mapped from space to time. Once such an explanation is added, Moore's account can perhaps indeed fully account for the mental phenomenon of mapping frontness onto anteriority and behindness onto posteriority.
Example slightly adapted from an example given by L. Talmy (personal communication, 2010).
Note that in American English (for one) we can say sentences such as ‘‘Tokyo time is five hours ahead of Moscow time,” meaning that in Tokyo it is 5 h later than in Moscow (e.g., 10 p.m. in Tokyo and 5 p.m. in Moscow). Likewise we say that ‘‘Moscow time is five hours behind Tokyo time,” meaning it is an earlier time in Moscow. Note that this construction involves just the opposite of the temporal conceptualization that we have analyzed in this section: the later time (10 p.m. in Tokyo) is ahead of the earlier time (5 p.m. in Moscow)—so the later time is not behind as in the examples in this section. Likewise the earlier time (5 p.m. in Moscow) is behind the later time (10 p.m. in Tokyo). Núñez, Motz, and Teuscher (2006) (as well as Núñez & Sweetser, 2006) do not recognize this distinction: On the one hand, they correctly characterize sentences such as Wednesday follows Tuesday or expressions such as ahead of time as instances where ‘‘earlier times are in front, ahead of, later times” (Núñez et al., 2006, p. 4)—expressions which involve the same temporal conceptualization discussed in this section and thus can be said to involve the imagery structuring proposed in Fig. 4 or a Figure–Ground reversal version of it. On the other hand, Núñez and colleagues also erroneously characterize sentences such as Greenwich Mean Time is lagging behind the scientific standard time or Boston time is three hours ahead of San Francisco time in this way—instances where, as has just been exemplified by the sentences involving Tokyo and Moscow time, the opposite is the case: Earlier times are behind later times. The imagery structuring underlying these expressions is of course different from the imagery structuring proposed in Fig. 4 (e.g., in Boston time is three hours ahead of San Francisco time the Figure event and Ground event move toward posteriority rather than toward anteriority). For a corresponding diagram, see Fig. S3.
This example has been adapted from an example given by L. Talmy (personal communication, 2010).
Sentence from L. Talmy (personal communication, 2010).
Sentence taken from Talmy (2000), p. 74). As mentioned before, the issue of sequentiality (cosequentiality/antisequentiality) does not need to be considered for the current analysis (cf. note 5).
Most of all I thank Len Talmy for many hours of very valuable and helpful discussion. Many thanks also to Steve Pinker for pointing out some very important issues in our discussions. Additionally, I also thank Ben Bergen, Lera Boroditsky, Lutz Jäncke, Kevin Moore, Rafael Núñez, and Dan Schacter for taking some time to discuss specific aspects of this article and thereby improve these aspects. The research carried out for this article is part of a larger research project on the conceptual structure of mental space, time, and causality, carried out at the Institute of Cognitive and Brain Sciences (ICBS) at UC Berkeley and financially supported by the Nachwuchsförderungskommission der Universität Zürich (Funding for Individuals and Projects, University of Zurich, Switzerland). I thank the involved ICBS faculty members as well as the members of the Nachwuchsförderungskommission for their support, too.