Automatic detection of irregular vanishing and reappearing parts of objects in two interwoven sequences: A visual mismatch negativity study

The cognitive system automatically develops predictions on the basis of regularities of event sequences and reacts to the violation of these predictions. In the visual modality, the electrophysiological signature of this process is an event‐related potential component, the visual mismatch negativity (vMMN). So far, we have no data, whether the system underlying vMMN is capable of dealing with more than one event sequence simultaneously. To disclose this aspect of the capacity of the system, in a passive oddball paradigm, we presented two interwoven sequences. The stimuli were objects (diamond patterns with their diagonals), one of the sequences was presented to the left side and the other to the right side of the visual field. From time to time, two parallel lines of the diamonds disappeared (OFF event) and then reappeared (ON event). The frequently vanishing pair of lines on the left side (standard) was identical to the rarely vanishing lines of the objects on the right side (deviant) and vice versa. We found that deviant ON events elicited vMMN only for left‐side deviants and deviant OFF events elicited vMMN only for right‐side deviants. The standardized low resolution brain electromagnetic tomography (sLORETA) source localization showed vMMN sources both in posterior visual structures and in anterior locations, and activity was stronger in the hemisphere contralateral to the deviant event. According to the results, the system underlying vMMN is capable of dealing with two sequences, but within a sequence, it detected only one type (either OFF or ON) of deviancy.


| INTRODUCTION
In this study, we investigated whether the capacity of an automatic system capable of following environmental regularities was large enough to deal with two interwoven sequences of visual events simultaneously. Within the sequence of physically or categorically identical frequent events (standards), the appearance of infrequently appearing different events (deviants) is automatically detected. In the visual modality, the signature of detection of such nonattended changes is the visual mismatch negativity (vMMN) component of event-related potentials (ERPs). vMMN is the difference between the ERPs to the standard and to the deviant events. This difference potential emerges within the 90-to 350-ms range after the onset of the deviant stimulus, usually at electrode locations over the posterior scalp (Stefanics et al., 2014). vMMN can be elicited by deviant visual features like colour Winkler et al., 2005), spatial frequency (Heslenfeld, 2003), movement direction (Lorenzo-L opez et al., 2004) orientation,  object-related features (Müller et al., 2010), visual categories (Athanasopoulos et al., 2010), facial expressions (Astikainen & Hietanen, 2009;Kreegipuu et al., 2013;Li et al., 2012), gender (Kecskés-Kov acs et al., 2013), the age of individuals on photographs (Csizmadia et al., 2021), and semantic categories Hu et al., 2020). This list of deviants capable of eliciting vMMN shows that the content of the memory system is sensitive to both simple and complex stimuli. vMMN is also elicited by the violation of sequential rules. As an example, vMMN emerged when the rule is that two identical colour gratings (AA) is followed by two gratings of a different colour (BB), and this rule was violated by the appearance of a third identical grating sharing the same colour (AAA) (Czigler & Pat o, 2009;Stefanics et al., 2011).
The present study investigates another aspect of the capacity of the system. So far, vMMN studies have investigated single oddball sequences with one or more standard and deviant events. In the present study, we investigated whether the system is capable of dealing with two simultaneously presented (interwoven) but independent sequences, where the standard in one of the sequences is identical to the deviant of the other sequence and vice versa. To better understand what happens when the brain has to follow two event sequences happening at the same time could be important as in everyday life, outside of the laboratory, people are usually presented with a vast amount of information coming from the nonattended part of the visual space. This study is a step towards understanding exactly how much and what type of events can the brain "pick up on" without directly attending to them. Our further intention with this study was to develop a task that we can later use to study hemispatial neglect, a neuropsychological syndrome that develops most often after right hemisphere stroke. Neglect is traditionally defined as a set of symptoms where the patient does not attend, react, or orient to the contralateral (usually left) side of space (Vallar, 1998). Deficits in neglect are usually linked to disturbances in the attention networks (Corbetta & Shulman, 2011); however, currently, we know less about how intact the "preattentive" processes are during this syndrome. So, the current task was designed to explore these underlying mechanisms of the neglect syndrome.
In our study, we presented simultaneously one event sequence to the left side and another sequence to the right side of the visual field. There were standard and deviant events in both sequences. The sequences were unrelated to the ongoing task, that is, we conducted a passive oddball paradigm. We swapped the standarddeviant relationship in the various experimental blocks; therefore, in this reverse control design, we compared ERPs with physically identical stimuli in the role of standard and deviant. We adopted the stimuli developed in studies with similar stimuli and design (Czigler et al., 2019;File et al., 2018;Sulykos et al., 2017). Within a sequence, there were three kinds of stimuli: (1) diamond patterns with their diagonals, (2) diamond patterns without the parallel lines of left strokes, and (3) diamond patterns without parallel lines of right strokes. Within a sequence, either (2) was the standard and (3) the deviant, or vice versa. The whole diamond pattern, and the patterns without a pair of lines, alternated without an interstimulus interval; therefore, two parallel lines of the diamonds disappeared (OFF event) and subsequently reappeared (ON event). An advantage of this paradigm is to have control of low-level adaptation, because all lines of the patterns are present for a longer period (ON event). Therefore, the deviant minus standard difference potential is related to the deviant-related additional activity and, in a less extent, to the adaptation-related decreased activity to the standard. On both sides, the stimuli were presented in the lower half-field, because from several studies, it was proven that lower half-field stimulation resulted in a more robust vMMN (Amenedo et al., 2007;Berti, 2009;Clifford et al., 2010;Czigler et al., 2004;Müller et al., 2012;Sulykos & Czigler, 2011). Like in the majority of vMMN studies, we also measured the exogenous ERP components. The emergence of these components, in our case, the canonical P1, N1, and P2, indicates that the visual system was able to detect the nonattended, ERP-related stimuli (Di Russo et al., 2002). Thus, the emergence of the P1, N1, and P2 can indicate that the stimulus arrangement is adequate, and the visual system can process all that is necessary to subsequently produce the difference-related vMMN.
Because no previous study has investigated the possibility of simultaneous detection of more oddball sequences in the visual field, we had no a priori hypothesis on such a possibility. An important feature of the OFF-ON method is that the deviant OFF events necessarily violate the sequential rule. In case of the representation of the diamond as a whole, the appearances of the ON events are not equivocal. Either after standard or deviant OFF events, the diamond as a whole returned, that is, this event is considered not unexpected. In fact, in our previous experiments (e.g., Sulykos et al., 2017), we obtained no relatable ON-related vMMN. However, in case of a capacity limit of the system underlying vMMN, it is possible that the onset of the deviant lines is not connected to the representation of the diamond as a whole. If this is the case, the infrequent appearance of the two parallel lines (ON events) is treated as a deviant event, and it can elicit the vMMN. Presenting two interwoven sequences, we cannot preclude this opportunity. Therefore, in our analyses, we treated the ERPs to the OFF and ON events separately, similarly to the previous studies (e.g., File et al., 2018).
However, in case of having enough capacity to fully process the two sequences like in the previous studies (e.g., Sulykos et al., 2017), we expected dominant vMMN to the OFF events. In case of limited processing, the possibility of reduced OFF-related vMMN was expected. Reduced sensitivity to the OFF events may result in enlarged ON-related vMMN. In order to increase the possibility of a mismatch response in both sequences, the two sequences were presented to different sides of the visual field. In this way, the sequences stimulated different hemispheres; therefore, we expected an easier differentiation of the sequences.

| Participants
There were 21 participants, who were paid students (12 female, mean age: 22.00 years, SD = 1.78). Participants had normal or corrected to normal vision (measured with Snellen cards). The study was conducted in accordance with the Declaration of Helsinki and approved by the Ethical Review Committee for Research in Psychology, Hungary (EPKEB).

| Stimuli and procedure
Experiments were conducted in an electrically and sound isolated chamber. The stimuli were presented on a 24-in. LCD monitor (Asus VS229na) with a 60-Hz refresh rate.
To prevent attention to the visual mismatch-related stimuli, participants performed a reaction time (RT) task with the task-relevant stimulus presented in the central part of the screen. The task-irrelevant stimuli appeared in the lower left and right quadrants of the visual field. Participants were instructed to fixate on the central cross, to not look anywhere else during the task, and to react to the changes being made in that task-related fixation cross. The central fixation cross randomly changed its shape, with the unequal vertical and horizontal lines (0.34 and 0.68 ) being reversed randomly between 5 and 15 s. Participants had to press the space bar in response to the changes, allowing us to then be able to measure their RT. Behavioural data were defined as the average RT and the number of missing detections of the changes.
The task-irrelevant, vMMN-related stimuli were diamonds with their diagonals. During the task, either the 45 or 135 parallel lines of the diamonds vanished for 400 ± 16.6 ms (OFF events) and then reappeared for 400 ± 16.6 ms (ON events), but the probability of which direction of the parallel lines would vanish was biased, thus creating frequent events (standards) and rare events (deviants). There were no interstimulus intervals. There were two independent, but simultaneously presented event sequences, one on the left side and one on the right side of the lower half of the visual field. Within a session, there were six blocks, each consisting of 100 changes from whole diamonds to diamonds with vanished sides, from which 85% were standards and 15% were deviants, in each of the event sequences. Figure 1 illustrates the vMMN-related stimuli and the event sequences. From the 1.6-m viewing distance, the size of a diamond was 1.57 deg. The distance of the diamonds from the imaginary vertical midline of the screen was 4.96 deg. The vertical distance of the centres of diamonds was 1.96 deg. The line thickness of the diamonds was 10 pixels. Luminance of the screen and the lines of the diamonds were 47 and 177 cd/m 2 , respectively.

| Measurement of electrical brain activity
Electrical brain activity was recorded from 27 locations according to the extended 10-20 systems (F7, F3, FZ, F4, F8, FC3, FC4, T7, C3, CZ, C4, T8, CP5, CP6, P7, P3, PZ, P4, P8, PO7, PO3, POZ, PO4, PO8, O1, OZ, and O2) with BrainVision Recorder (ActiChamp amplifier, Ag/AgCl active electrodes, EasyCap, Brain Products GmbH, sampling rate: 1000 Hz, DC-70 Hz online filtering). The reference electrode was placed on the nose tip and the ground electrode on the forehead (AFz). Horizontal and vertical electrooculograms were recorded with bipolar configurations between two electrodes (placed lateral to the outer canthi of the two eyes and above and below the left eye). The electroencephalography (EEG) signal was bandpass filtered offline with a noncausal Kaiser-windowed Finite Impulse Response filter (low pass filter parameters: 30 Hz of cut-off frequency, beta of 12.265, and a transition bandwidth of 10 Hz; high pass filter parameters: 0.1 Hz of cut-off frequency). The EEG data were processed with MATLAB R2015a (version 2015a, MathWorks, Natick, MA). Independent component analysis was used to remove eye-movement artefacts. Epochs with larger than ±100-μV voltage change at any electrode site were considered artefacts and rejected from further processing. Table 1 shows the mean number of averaged epochs for each event.
Stimulus onset was measured by a photodiode, providing an exact zero value for averaging. Epochs were extracted for further analysis ranging from À100 to  400 ms for both the ON and OFF events. Then, we averaged these events separately and, in addition, the left-side and the right-side deviants.
To measure the amplitude and latency of the ERP components and the emergence of vMMN, we constructed a left-and a right-side regions of interest (ROIs) from PO3 and O1 and PO4 and O2 locations, respectively, based on one of our previous studies (File et al., 2018). Exogenous component (P1, N1, and P2) latencies were measured as the latency of the largest amplitude values within the 50-to 100-, 100-to 170-, and 170-to 300-ms ranges, respectively. On the standard OFF and ON events, amplitudes of the exogenous components were measured as the mean of the ±10-ms range around the latency values. To measure these components, our aim was only to compare the activities of the OFF and ON events. Latencies and amplitudes were compared in analyses of variance (ANOVAs) with factors of Event (OFF or ON), Side (left and right stimuli), and ROI (right or left).
To investigate vMMN, we calculated difference potentials in the two ROIs. Although the same stimulus configuration appeared as deviant in a sequence and as standard in another sequence, difference potentials were calculated from physically identical ERPs. Considering the results of previous studies (Czigler et al., 2019;Sulykos et al., 2017), we expected the deviant minus standard difference in a range of approximately 90-200 ms. Inspection of the difference potentials corresponded to the expectation. We divided the 90-to 180-ms range into three epochs: 90-120, 120-150, and 150-180 ms. We evaluated the emergence of vMMN at the two ROIs within the main values of the three epochs by calculating t-tests against zero. Due to the multiple testing, probabilities were corrected using the Benjamini-Hochberg procedure (Benjamini & Hochberg, 2000).
To compare the possibility of lateral differences of the vMMNs, we calculated ANOVAs on the average amplitudes of the difference potentials with factors of Side (left and right stimuli) and ROI (left or right). ANOVAs were calculated only in cases where the t-tests indicated vMMN in at least over one of the hemispheres. For post hoc paired comparison, we used Tukey honest significant difference (HSD) tests. In these tests, the significance level was at least p < 0.05. We used the Statistica package (Version 13.4.0.14, TIBCO Software Inc.) for statistical analyses.
We illustrated the deviant minus standard activity differences in surface maps. As an attempt to localize the source of the difference activity, the standardized low resolution brain electromagnetic tomography (sLORETA) distributed source localization analysis (Pascual-Marqui, 2002) was applied. The sLORETA gives a solution for the EEG inverse problem by applying a weighted minimum norm estimation with spatial smoothing and standardization of the current density map. The forward model was generated on a realistic boundary element method head model (Gramfort et al., 2010) by applying a magnetic resonance imaging (MRI) template (ICBM152; 1 mm 3 voxel resolution) with template electrode positions. The Q14 reconstructed dipoles (pA/m) were determined for every 15,002 sources in three orthogonal directions (unconstrained solution). Only brain regions were considered where the computed χ 2 -test value showed significance in at least 20 voxels and in at least one of the conditions. The sLORETA analysis was performed with Brainstorm (Tadel et al., 2019), which is documented and freely available for download online under GNU general public license.

| Behavioural results
The average hit rate of the change of fixation was 95.70% (SD = 4.63); and it was similar in the conditions with left-and right-side deviant stimuli, 95.46% (SD = 5.73) and 95.97% (SD = 4.53), respectively. The average RT was 407.10 ms (SD = 68.02); and it was similar in the conditions with left-and right-side deviant stimuli, 475.63 (SD = 63.33) and 476.89 (SD = 74.94), respectively. There were no significant differences between the conditions, measured by t-tests. Accordingly, we obtained no performance difference. Figure 2 shows the ERPs to standard and deviant stimuli, separately, for OFF and ON events to left-and right-side stimuli at PO3, O1, PO4, and O2 locations. Stimuli elicited the positive P1, negative N1, and positive P2 exogenous components, and the corresponding latency and amplitude values are shown in Tables 2 and 3.

| Exogenous ERPs
For the latency of the P1 component, we obtained a significant Event Â ROI interaction (F(1,20) = 4.88, ε = 0.20, p < 0.05). According to the Tukey HSD test, at the right ROI, the latency of the OFF events was shorter. For the amplitudes, we obtained no significant effect. For the N1, there was neither significant latency nor amplitude differences. For the P2 latency, both the Event (F(1,20) = 5.63, ε = 0.22, p < 0.05) and ROI (F(1,20) = 4.91, ε = 0.20, p < 0.05) main effects were significant. Latency was shorter for the OFF events, at the right ROI.
F I G U R E 2 Event-related potentials to standard and deviant stimuli. Separately for OFF and ON events to left-and right-side stimuli at PO3, O1, PO4, and O2 locations with 95% confidence intervals (CIs) indicated.

| Deviant-related ERPs
The deviant minus standard difference potentials and the surface distribution (120-to 150-ms range) are shown in Figure 3. Table 4 shows the main amplitude values of the difference potentials in response to the OFF and ON events for left-and right-side stimuli at the left and right ROIs. According to the t-tests, significant differences emerged for OFF events at right-side stimulation in the 120-to 150-and 150-to 180-ms ranges. For the ON events, left-side stimulation elicited vMMN in all investigated latency ranges.
Difference potentials were compared in two-way ANOVAs with factors of Side (left or right F I G U R E 3 Difference potentials and the surface distribution of the difference potentials in the 120-to 150-ms epoch. (a) Difference potentials for left-and right-side stimuli in the right and left ROIs. (red) OFF events with 95% confidence interval (CI) indicated and (blue) ON events with 95% CIs indicated. (b) The surface distribution of difference potentials for the leftand right-side stimuli in the OFF and ON conditions in the 120-to 150-ms range. We present surface distributions only for the 120-to 150-ms range, because we obtained OFF-related visual mismatch negativity only in this range. ROI, region of interest. stimulation) Â ROI (right or left hemisphere), for the OFF events in the 120-to 150-and 150-to 180-ms epochs and for the ON events in all three epochs. For the OFF events in the 120-to 150-ms epoch, we obtained a significant Side main effect (F(1,20) = 6.43, ε = 0.22, p < 0.05) and Side Â ROI interaction (F(1,20) = 6.83, ε = 0.25, p < 0.05). Left-side stimuli elicited a larger negativity on the right side, but according to the Tukey HSD test, the ROI difference was not significant. (Note that OFF stimuli presented to the left side did not elicit reliable vMMN.) In the 150-to 180-ms epoch, the Side main effect was significant (F(1,20) = 5.60, ε = 0.22, p < 0.05). Negativity was larger for the right-side stimuli.
In summary, OFF events elicited vMMN at right-side stimulation, and ON events elicited vMMN at left-side stimulation. Figure 4 and Table 5 show the results of sLORETA calculations. As Figure 4 shows, deviant minus standard activity difference emerged both in posterior and anterior T A B L E 4 The OFF and ON events to left-and right-side deviancies at the left and right ROIs. F I G U R E 4 Standardized low resolution brain electromagnetic tomography (sLORETA) source localization for the left ON and right OFF difference potentials (visual mismatch negativity) within the 90-to 120-, 120-and 150-, and 150-to 190-ms epochs. Note that for the 90-to 120-ms epoch, only the ON-related effects are shown, because we did not identify OFFrelated visual mismatch negativity in this epoch. The darker pink colour indicates the brain regions of increased deviant-standard difference.
locations, and the activity concentrated on the hemisphere contralateral to the deviant event. Data on Table 5 reinforced this observation. Within the posterior cortex, activity concentrated on locations with functions of higher order processing (lateral occipital areas, fusiform, and lingual gyrus) and at the location of both the ventral and dorsal stream of visual pathways. Concerning the anterior cortex, activity appeared in various parts of the frontal cortex. However, sLORETA identified some locations with functions difficult to attribute to automatic deviant detection (precentral gyrus and pars opercularis). Table S1 shows a more detailed version of our results of sLORETA calculations.

| DISCUSSION
As the main question of this study, we investigated whether the system underlying the vMMN was capable of recording the regularity of two interwoven sequences and to detect deviant events within both sequences. To this end, we presented two oddball sequences, one to the right side and the other to the left side of the visual field. The study applied the stimuli of the OFF-ON (disappearanceappearance) method . This method controls the low-level adaptation of the visual features; furthermore, and more importantly, it involves two events: Parts of an object temporally vanish (OFF event), and subsequently, they reappear (ON event). In the oddball sequences, there were frequently and infrequently disappearing parts, that is, standard and deviant events. Accordingly, within a sequence, there could be two possible vMMNs, one for the OFF deviant and another for the ON deviant. In the present study, we presented two interwoven oddball sequences, one to the left and another to the right half of the visual field; thus, a maximum number of recorded vMMNs were four. However, for deviant OFF events, vMMN appeared only in the right-side sequence (in the 120-to 180-ms range) and vMMN to ON events appeared only in the left-side sequence (in the T A B L E 5 Number of voxels in brain regions in which the differences between the deviant and standard task-irrelevant stimuli were the largest within the 90-to 120-, 120-to 150-, 150-to 180-ms epochs. 90-to 180-ms range). Therefore, the answer to the question whether the system underlying vMMN is capable of dealing with two parallel sequences is "YEA and NAY," because deviants in both sequences elicited vMMN but for only one of the two parallel sequences, this being either OFF or ON deviants alone.
In previous studies with the OFF-ON method (File et al., 2018;Sulykos et al., 2017), vMMN emerged only to OFF-related stimuli, or else, vMMN was smaller to ON deviants (Czigler et al., 2019). In discussing the results of these previous studies, we argued that the vanishing and the reappearing parts were treated as the change of unitary objects. Infrequent vanishing of particular lines (deviant OFF) was an unpredicted event, whereas ON events just set back the unitary object (i.e., it was not unexpected). It seems that the present more complex design, two interwoven oddball sequences in the two sides of the screen, disrupted the unity of the objects. Therefore, OFF and ON events were treated separately.
One can interpret the results in a different way. Presenting standard OFF events, the objects in the two sides were different, in contrast with the deviant OFF events, where the objects in the two sides were identical. Therefore, OFF deviancy was equal to bilateral "sameness." Following up this explanation, the processing system treats the two sequences differently because "sameness" was detected only if the deviant stimuli originated from the right side; and detection of the reappearing whole elicited vMMN only after left-side deviancy. In other words, with this explanation, the system underlying vMMN is capable of dealing with two sequences.
Our attempt to identify the sources of difference potentials shows that the lateral occipital cortex was active in processing both the deviant OFF and ON events in all epochs with significant vMMN amplitudes. Extrastriate cortex activity is frequent in sLORETA calculations on vMMN (File et al., 2017; and also using variable resolution electromagnetic tomography (Müller et al., 2012) and brain electrical source analysis (Urakawa et al., 2010) methods. Concerning the other visual areas, the deviant OFF events elicited wider structures, including the cuneus, fusiform gyrus, precuneus, and pericalcarine areas. Other posterior areas within the parietal and temporal cortex were active in both OFF and ON deviants. Similar posterior activities were recorded by previous studies (File et al., 2017;Jack et al., 2015;Kimura et al., 2012;Li et al., 2012;Müller et al., 2012;Shi et al., 2013;Wang et al., 2014), and also anterior sources were identified in some studies on vMMN (Müller et al., 2010(Müller et al., , 2012. In the present experiment, sLORETA also identified various structures of the frontal lobe. In the auditory MMN, the involvement of the frontal cortex is fairly obvious (Deouell, 2007), and an animal study (Casado-Rom an et al., 2020) indicated frontal top-down influence on the modality specific structures. However, in the present study, the ON-related localization to the precentral gyrus (i.e., a somatosensory area) raises the possibility of ghost localization, even if similar areas were also described in other studies (Shi et al., 2013;Wang et al., 2014). As for the other parts of the frontal cortex, the structures involved in deviant-related activity were wider (superior and orbital frontal cortices) at OFF deviancy.
One of the reasons behind this study was to develop a task with which we can test the so-called preattentive processes during hemispatial neglect. As this syndrome is characterized by the difference between attending to left and right sides of (visual) space, based on the results of this study, the task seems promising to be next applied for that purpose.
One of the limitations of this study was that we could not compare the results of testing interwoven sequences with the results of presenting a single oddball sequence within this study; thus, we can only draw indirect comparisons between these two arrangements. Also, the stimuli (diamond patterns) were identical on both sides, which could make it more difficult for the cognitive system to follow both sequences. Although the current study was an exploratory one, further studies on the same matter will be able to rely on the results presented here.
In conclusion, deviant stimuli in two interwoven oddball sequences are capable of eliciting vMMN in both sequences. As the specificity of the present study, the sequences were spatially separated, and the left and right extrafoveal localization facilitated the separate processing of these sequences. However, the processing within the two locations were different; right-side stimulation was sensitive to infrequently vanishing parts of the objects, whereas left-side stimulation was sensitive to onset deviancy. Further studies are needed to disclose, whether the system underlying vMMN is not only capable of dealing with two sequences presented to the same hemisphere but also of processing two sequences constructed from highly different objects.