The impact of acute asymmetric hearing loss on multisensory integration

Humans have the remarkable ability to integrate information from different senses, which greatly facilitates the detection, localization and identification of events in the environment. About 466 million people worldwide suffer from hearing loss. Yet, the impact of hearing loss on how the senses work together is rarely investigated. Here, we investigate how a common sensory impairment, asymmetric conductive hearing loss (AHL), alters the way our senses interact by examining human orienting behaviour with normal hearing (NH) and acute AHL. This type of hearing loss disrupts auditory localization. We hypothesized that this creates a conflict between auditory and visual spatial estimates and alters how auditory and visual inputs are integrated to facilitate multisensory spatial perception. We analysed the spatial and temporal properties of saccades to auditory, visual and audiovisual stimuli before and after plugging the right ear of participants. Both spatial and temporal aspects of multisensory integration were affected by AHL. Compared with NH, AHL caused participants to make slow, inaccurate and unprecise saccades towards auditory targets. Surprisingly, increased weight on visual input resulted in accurate audiovisual localization with AHL. This came at a cost: saccade latencies for audiovisual targets increased significantly. The larger the auditory localization errors, the less participants were able to benefit from audiovisual integration in terms of saccade latency. Our results indicate that observers immediately change sensory weights to effectively deal with acute AHL and preserve audiovisual accuracy in a way that cannot be fully explained by statistical models of optimal cue integration.


| INTRODUCTION
Spatial perception is crucial for orienting oneself towards relevant events in the world and interacting with them.Although information from a single sense can be used to localize objects and events, each sense has its own unique strengths and limitations.Whereas vision is limited by the field of view, it has a remarkably high spatial resolution (near the fovea).Audition allows detecting events all around us but is spatially less precise.Our brain can combine information from different senses to get the best of both worlds overcoming or reducing the impact of sensory limitations through the process of multisensory integration (MSI).MSI allows us to unify sensory inputs and greatly enhances detection, localization, recognition and discrimination of events (Alais & Burr, 2004;Alsius & Munhall, 2013;Ernst & Banks, 2002;Frassinetti et al., 2002;Miller, 1982;Ross et al., 2006;Van der Stoep et al., 2015).Impairments in MSI can therefore greatly hamper a person's ability to perceive and interact with the environment.Alterations or impairments in MSI can, for example, arise after brain damage (Van der Stoep et al., 2019), or occur in neurodevelopmental cases such as attention-deficit/ hyperactivity disorder (ADHD; Panagiotidi et al., 2017) and autism (Baum et al., 2015).Although much has been learned about how our senses collaborate to facilitate perception in various circumstances and in various disorders, knowledge about the impact of common auditory impairments on MSI is currently scarce.Worldwide, about 466 million people suffer from hearing loss (WHO, 2020), but the impact of different types of hearing loss on how the senses work together is rarely investigated (though see Gieseler et al., 2018;Venskytis et al., 2019).Common subtypes of hearing loss are dissociable by their aetiology: sensorineural hearing loss occurs after damage to the inner ear (e.g., hair cells or auditory nerve), whereas conductive hearing loss is a consequence of damage to the middle or outer ear (e.g., perforated eardrum, otosclerosis, malformations or inflammations).
In this study, we investigated the impact of acute conductive asymmetric hearing loss (AHL) on both the temporal and spatial benefits of MSI.In normal hearing and vision, the temporal benefit of MSI is reflected in a decreased latency of (orienting) responses to multisensory events (Frens et al., 1995;Hughes et al., 1994;Miller, 1982;Otto, 2019;Otto et al., 2013;Van der Stoep et al., 2015).Numerous studies have shown that responses to multisensory events can be faster than the fastest response to sound or light presented in isolation.This facilitation effect often surpasses statistical facilitation (becoming faster simply because of a higher probability on a fast response when there is more information to respond to; see Gondan & Minakata, 2016;Miller, 2016;Otto & Mamassian, 2016;Ulrich et al., 2007).This multisensory facilitation effect is very robust and can be observed in manual detection responses and saccadic behaviour (Hughes et al., 1994).
Apart from temporal benefits, multisensory benefits also arise in space; spatial estimates become more precise than unisensory estimates (Alais & Burr, 2004;Battaglia et al., 2003;Ernst & Banks, 2002).In many cases, humans tend to integrate information from different senses in a statistically (near-) optimal fashion by considering the reliability of unisensory information: observers put more weight on the most reliable sense when integrating sensory input (Ernst & Banks, 2002;Rohde et al., 2016).Whether the perceptual system also considers accuracy of sensory input (i.e., potential sensory impairments) is not fully clear (though see Zaidel et al., 2013), but Bayesian optimal cue integration models can often accurately predict multisensory behaviour based on unisensory reliability without considering sensory accuracy (e.g., being agnostic about external accuracy; Alais & Burr, 2004;Battaglia et al., 2003;Ernst & Banks, 2002).Because visual information is encoded directly in a highresolution retinotopic reference frame (Grill-Spector & Malach, 2004), and the auditory system must infer a sound's location from various auditory cues (i.e., binaural and monaural (spectral) cues (Blauert, 1997;Frens et al., 1995;Middlebrooks & Green, 1991), vision typically dominates multisensory spatial perception (Alais & Burr, 2004;Chen & Vroomen, 2013).This dominance can be seen, for example, in the spatial ventriloquist effect, where the perceived location of a sound source is typically attracted by a nearby visual source (Alais & Burr, 2004;Bruns, 2019;Hendrickx et al., 2015;Stekelenburg et al., 2004).
What happens to the benefits of MSI if hearing is impaired?Given that spatial alignment is a prerequisite for successful binding of unisensory components, accurate unisensory localization is of crucial importance for MSI.Yet, AHL distorts spatial localization in the auditory domain, by disrupting the use of binaural cues (cues that require information from two ears) that humans use to localize sounds in the horizontal plane (Middlebrooks & Green, 1991).Interaural time difference (ITD) can still be processed as long as there is some auditory information coming in (regardless of volume), whereas the interaural level difference (ILD) cue is distorted.Sounds that come from the impaired side are perceived as relatively weaker than actually is the case, and because of this imbalance in perceived volume in the left and right ear in AHL, the ILD does not indicate the correct spatial location of sound sources.This typically leads to problems determining the position of sound sources with a bias towards the unimpaired/less impaired ear (Abel & Lam, 2008;Noble & Gatehouse, 2004;Shargorodsky et al., 2010;Slattery & Middlebrooks, 1994).In the case of acute AHL combined with normal (or corrected-to-normal) vision, hearing and vision will provide conflicting spatial information about the location of an audiovisual event.This conflicting information is problematic for MSI, as one of the main requirements for MSI to occur is that auditory and visual stimuli should originate from approximately the same spatial location (i.e., the principle of spatial alignment; Frens et al., 1995;Meredith & Stein, 1986;Spence, 2013).Therefore, when a sensory impairment alters spatial cues and causes a sensory conflict, this may result in a loss of the spatial and temporal benefits of MSI (e.g., missing out on increased localization accuracy and precision, and faster orienting responses; see Gieseler et al., 2018, for a study on temporal aspects of MSI and hearing loss).
To summarize, in NH, MSI facilitates more accurate, precise, and faster orienting responses towards multisensory targets as compared to unisensory conditions.How AHL affects MSI is unclear.By comparing the spatial and temporal benefits of MSI during eye-movement behaviour between NH and acute AHL, we can reveal how changes brought about by sensory impairments such as acute AHL are dealt with by the human perceptual system.Participants were instructed to make eye movements towards spatially and temporally aligned auditory, visual and audiovisual stimuli randomly appearing to the left and right side of a central fixation cross with NH and a plugged ear, simulating acute conductive AHL.Eye movements were chosen because they are fundamental to the spatial perception of the environment and are dependent on activity in the superior colliculi, a structure involved in MSI and saccade generation (Bell et al., 2005;Hughes et al., 1994;Sparks & Nelson, 1987;Stein & Meredith, 1993).First, because acute AHL affects binaural cues (mainly the ILD), we expected sound localization to become biased towards the intact ear.Localization accuracy and precision are expected to diminish for auditory, but not for visual stimuli.Assuming that this will indeed be the case, acute AHL should cause a spatial conflict between the perceived auditory and visual stimulus location of a spatially and temporally aligned multisensory event.Then, if this spatial conflict is large enough, both the multisensory temporal facilitation effect (faster localization), as well as the spatial benefit (more precise, more accurate) should decrease or vanish.We mainly expect a change in accuracy (although changes in precision might also occur because of the plug-induced lessened reliability of the auditory system) and further assess whether potential multisensory facilitation effects after AHL still occur because of MSI or not.
We test this within both domains (temporal, spatial) by using two models commonly used to assess MSI (i.e., race model inequality violation in the temporal domain and maximum likelihood estimation [MLE] in the spatial domain).To further investigate how reliability of sensory information is weighted before and after AHL, we varied the reliability of the auditory (NH vs. AHL) and visual input.With acute AHL (here induced in the right ear), we expected a stronger visual weight because of decreased reliability of auditory localization compared with NH.

| Participants
Thirty participants signed up for the experiment.All participants reported normal or corrected-to-normal hearing.However, before the start of the experiment, each participant first took part in an equal loudness test (using the same high-pass [HP] filtered sounds as those presented during the experiment).The outcome was used to determine whether there were any pre-existing hearing asymmetries.The equal loudness test was repeated after inserting the earplug and at the end of the experiment with the earplug still inserted.The latter was done to check whether the earplug had moved and whether the induced hearing loss remained consistent throughout the experiment.After testing, data from eight participants were excluded from further analysis because of one or multiple of the following reasons: (1) a pre-existing hearing asymmetry >10 dB(A), 1 (2) less than 25 trials left in at least one condition after saccade filtering (see below for more information on saccade filter parameters) in either the NH or AHL condition (e.g., due to too many erroneous saccades or poor recordings) and (3) a change in the amount of hearing loss induced by the plug over the course of the experiment (>10 dB(A) change between the results of equal loudness test before and after the experiment with the earplug inserted; see below).In total, data from 22 participants (19 female, M = 22.8 years, SD = 2.2 years, range = 20-30, one participant did not report age) were further analysed.Participants gave written informed consent prior to the start of the experiment.All participants were compensated for their participation with money or course credits.The 1 Although there is no strict consensus in clinical judgement of AHL (Margolis & Saly, 2008), AHL is generally defined as a ≥10 dB (A) hearing threshold at 0.5, 1, 2 and 4 kHz (Abel & Lam, 2008;Noble & Gatehouse, 2004).study protocols were approved by the faculty ethics committee (FETC) of Utrecht University.

| Test room
The experiment was conducted in a darkened room of $5 Â 2.5 Â 2.8 m.Participants were seated in a chair with their head supported by a chin rest to stabilize their head to facilitate eye-tracking and to maintain a fixed viewing distance of $70 cm between the projection screen and the participants.The setup was placed in the middle of the room (i.e., equally far from the left and the right wall).

| Apparatus
An Acer X138WH projector (60 Hz, 1024 Â 768 px) was used to present visual stimuli on a black soundpermeable screen (71 Â 54 cm).The projector was placed above the participant's head and projected downwards in a small angle.Two speakers (Harman/Kardon HK206, Frequency response: 90-20,000 Hz) were placed directly behind the projection screen at the same location as where the visual stimuli would appear during the experiment to ensure audiovisual spatial alignment (see Figure 1a).For the equal loudness test, a Sennheiser HD 202 headphone (Frequency response: 18-18,000 Hz) was used.An EyeLink 1000 eye-tracker (SR Research Ltd.) was used to measure the left eye gaze position with a sample frequency of 1000 Hz.AHL was simulated by plugging the right ear with an Ohropax Soft Earplug, which induced an attenuation of $32 dB(A) for auditory stimuli used in this experiment.

| Stimuli
To assess the benefits of MSI, three stimulus types were presented: auditory (A), visual (V) and audiovisual (AV).Stimuli were generated using MatlabR2007b.Auditory stimuli consisted of 100 ms high-pass filtered (>3 kHz) noise bursts (10 ms linear rise and fall time) presented at $60 dB(A) and at a reduced volume of $44 dB(A) (as control condition for overall sound intensity reduction induced by the earplug) against a background noise of $37 dB(A) (as measured from the location of the participant's head).The choice for high-pass filtered noise as auditory stimuli was based on pilot results showing that AHL had the most profound effect on auditory localization because of the earplug's effect on ILDs.To investigate the influence of AHL on sensory weighting, unisensory reliability was varied.Because the variance in auditory localization estimates are typically lower than visual localization estimates (Alais & Burr, 2004), we introduced a large difference between the visual stimuli to create one reliable and one unreliable visual stimulus: visual stimuli consisted either of a small (reliable) grey (3 cd/m 2 , SD = 0.67 , trim value = 3 SD, as measured from 70-cm viewing distance) or a large (unreliable) grey Gaussian luminance modulation (3 cd/m 2 , SD = 5.33 , trim value = 3 SD, as measured from 70-cm viewing distance).This translates to sizes of about 4 and 32 , respectively.The sizes (standard deviations) of the visual stimuli were based on pilot data and earlier research (Alais & Burr, 2004;Kowler & Blaser, 1995) showing that increasing the size of the visual information decreased localization accuracy and precision, making the visual information less reliable.Visual stimuli were presented for a duration of 100 ms against a black background (0.4 cd/m 2 , as measured from 70-cm viewing distance).
Stimuli were presented either 15 to the left or right of a grey central fixation cross (3 cd/m 2 , size = 1.65 Â 1.65 , as measured from 70-cm viewing distance).Audiovisual stimuli consisted of the combination of the auditory and visual stimuli presented at the same time and same location.Audiovisual stimulus presentation synchrony was confirmed using an oscilloscope.Acute AHL was simulated by inserting an earplug in the right ear.All stimuli were presented during NH and during acute AHL.Furthermore, all conditions were repeated in a control condition with NH but with a lower volume (see above) to test whether a reduced auditory intensity during NH (like that induced by the earplug) would lead to similar results as in the earplug condition as it is known that auditory localization precision decreases with sound intensity (Ege et al., 2018).

| Equal loudness task
Participants took part in an equal loudness test before the start of the experiment to check eligibility for participation.Participants were instructed to listen to high-pass filtered (>3 kHz) noise bursts of 200 ms that were presented to the left and the right ear consecutively via headphones.The order of presentation was randomized and counterbalanced.Each sound presentation was preceded by a period of silence with a random duration between 750 and 1250 ms.Participants were asked to judge which noise burst they perceived to be louder.The next trial began after the participant had responded with the left or right arrow key to indicate which sound was louder.During NH, the intensity of the left sound was held constant at 40 dB(A), whereas the right sound intensity varied between around 40 dB(A) ± 20 dB(A).The point of subjective equality (PSE) between both ears was computed using a minimum expected entropy staircase procedure consisting of 50 trials (Saunders & Backus, 2006).The binaural hearing test was again performed before the start of the earplug block to measure the effect of the earplug.As the earplug was inserted in the right ear, the right ear baseline level was set to 60 dB(A) but with the same dB range (±20 dB(A)) as used for NH to speed up the convergence on the PSE with the earplug inserted.After the acute AHL eye-tracking experiment, the equal loudness test was completed again to confirm that the earplug remained functional throughout the experiment.

| Eye-tracking task
At the beginning of each experimental block and after every break, the EyeLink 1000 was calibrated for each participant using a 5-point grid.Each trial started with a drift check.Participants pressed the space bar whenever they fixated on a dot presented at eye level in the horizontal centre of the screen to start the trial.The eyetracker was recalibrated whenever participants' eye position was too far off during the drift check.After the drift check, the fixation cross was presented for a random duration of between 750 and 1250 ms.When the fixation cross disappeared, either a target was presented for 100 ms (auditory, visual or audiovisual), or no target was presented (a catch trial; see Figure 1b).The time between the target/catch trial onset and the trial was always 2000 ms.Participants were instructed to remain fixated on the central cross after drift check and to direct their gaze towards a presented target as fast and accurately as possible.Participants completed a practice session of 10 trials before the real experiment was conducted to familiarize them with the procedure.
In total, 26 conditions were tested.Targets could appear to the left or the right of the central fixation cross.In NH, visual targets could be either reliable (4 ) or unreliable (36 ) and auditory targets could either have a normal (60 dB) or a reduced (44 dB) sound intensity.Audiovisual stimuli consisted of a combination of the above, and the auditory and visual stimuli and were always spatially and temporally aligned to create the audiovisual stimulus.In sum, four unisensory visual conditions (i.e., reliable, left/ right and unreliable, left/right), four unisensory auditory conditions (normal, left/right and reduced, left/right) and eight audiovisual conditions (the combination of the above, left/right) were tested in NH.During acute AHL, the conditions were the same, except there was no reduced sound intensity condition (for either the auditory-only or audiovisual conditions, resulting in 10 AHL conditions).Yet, to overcome general sensory biases caused by unequal amounts of targets from a certain sensory modality (e.g., a larger number of visual-only trials may lead to participants paying more attention to vision), auditory stimuli (at normal volume) were presented twice as much in the AHL condition as in the NH condition.This yielded (26 conditions Â 40 trials) + 80 (compensation for auditory-visual balance)) = 1120 trials that were included for analysis.Furthermore, we included $10% catch trials of the total in each block (NH: 71 trials; AHL: 53 trials) and a variable fixation duration to prevent anticipatory responses that could bias the test of the RMI (Gondan & Minakata, 2016;and see Section 2.4).In total, (1120 + 71 + 53 =) 1244 trials were performed by each individual.See Figure 2 for a schematic.We stress once more that audiovisual events were always the combination of a spatially and temporally aligned auditory and visual stimulus.The spatial conflict that we expected to influence behaviour is therefore always plug-induced, and not stimulusinduced.
The NH and AHL condition were blocked.Within blocks, trials were randomized and counterbalanced.Four short self-determined breaks were offered after every fifth part of each block (after each 142 trials in NH, and after each 106 in AHL).By pressing the spacebar, the experiment was continued.Between blocks of NH and AHL, a longer break was introduced.The order in which participants took part in the NH and AHL condition was counterbalanced across participants to control for order biases.After participant exclusion, 10 included participants had received an NH/AHL order, and 12 the opposite order of blocks.

| Saccade selection
For each trial, only the first saccade after target onset was used in the analysis.Valid NH trials were defined as trials that (1) did not contain eyeblinks in the crucial time window 100 ms before fixation offset until 500 ms after target onset, (2) contained saccades with latencies between 100 and 600 ms (0 and 600 ms in catch trials), (3) contained saccades with amplitudes of at least 20% of the distance between the fixation and target (0.2 Â 15 = 3 ), (4) contained saccade starting points that fell within 2.8 from the central fixation cross and (5) contained saccade landing points that did not exceed a radius of 11.21 from target location.This radius was chosen based on the size of the unreliable (large) Gaussian blob (32 ) leading participants to make less accurate and precise saccades.
Valid AHL trials were defined the same as valid NH trials, except that the AHL trials should contain (1) saccades with amplitudes of at least 10% (instead of 20%) of the fixation-target distance (0.1 Â 15 = 1.5 ) and ( 2) saccade landing points that did not exceed a radius of 30.72 from the target location.The differences between valid NH and valid AHL trials were based on the expectation that AHL would cause large auditory localization errors in trials with an auditory component causing saccades to land further away from the target location.
F I G U R E 2 An overview of all conditions in this study in the normal hearing (left) and asymmetric hearing loss condition (right).Audiovisual trials consisted of all combinations of unisensory conditions with 40 trials per unisensory combination.
Participants were excluded from further analysis whenever less than 25 out of 40 trials were left in one or more condition(s) after filtering, to ensure sufficient reliability of the measurements.

| Pre-processing
Temporal domain Saccade latency was defined as the time between the onset of the first saccade after the target appeared and the onset time of the target.Besides comparing the saccade latencies in the auditory, visual and audiovisual conditions, the amount of multisensory response enhancement (MRE) was calculated.The amount of MRE indicates the amount of speed up in the multisensory condition relative to the fastest unisensory condition based on their cumulative distribution functions (CDFs).The following steps were performed to obtain the amount of MRE using the RSE-box for Matlab (Otto, 2019): (1) removal of saccade latency outliers using the absolute deviation around the mean as a criterion per condition per participant (Leys et al., 2013; using the outCorrect function), ( 2) equalizing the number of data points in a condition by downsampling all conditions that were required for the comparison (e.g., A left , V low left and AV low left ) to the condition with the lowest number of trials, (3) the MRE was obtained by calculating the area between the AV CDF and the fastest of the unisensory CDFs using the getGain function from the RSE-box and (4) the relative MRE (rMRE) was calculated using Equation (1): RT AV , RT V , and RT A indicate the median RT in each condition.The observation of rMRE does not always indicate the presence of multisensory interactions as rMRE can also occur because of statistical facilitation.When there is no interaction between the senses (i.e., independent processing), responses in the AV condition can still be faster than responses in the fastest unisensory condition because participants can respond to both the onset of a sound and a light in the multisensory condition (i.e., statistical facilitation).The upper limit of statistical facilitation (of independent processing of sensory input) is described by the race model inequality (RMI; see Equation (2); Gondan & Minakata, 2016;Miller, 2016): The equation indicates that the probability (P) of a given response time (t) in the AV condition is less than or equal to the sum of the probability of a given response time in the unisensory auditory and the unisensory visual conditions.If the RMI inequality is violated (RMI violation), the multisensory response times cannot be explained by independent processing of sensory input, which is indicative of multisensory interactions (see Gondan & Minakata, 2016;Miller, 2016, for more on this).The amount of RMI violation was obtained by calculating the area between the AV CDF and the sum of the unisensory CDFs using the RSE-box function getViolation.In sum, rMRE was calculated to provide a somewhat more descriptive measure of MRE (reported in the Supporting Information), whereas RMI was our measure of main interest.

Spatial domain
Saccade accuracy was defined as the distance from the target in degrees of visual angle in the horizontal plane.Precision was calculated as the variance in saccade landing position in the horizontal plane (larger variance indicates lower precision).The relative reliability of auditory and visual sensory estimates can be estimated using the variance of the unimodal localization response distributions (i.e., saccade landing points).If the two distributions (i.e., of A and V) have equal variances, the senses are equally reliable.Statistically speaking, it would then be optimal to equally weigh the localization estimates of both senses to arrive at the multisensory localization estimate (Alais & Burr, 2004;Ernst & Banks, 2002).The multisensory estimate would then be the (weighted) average of the unimodal localization estimates.However, if one unimodal estimate distribution has a smaller variance than the other, it would be statistically optimal to have the localization estimate of the more precise sense preponderate over the second.The combined estimate would then be shifted towards the more reliable sense.Additionally, the variance of the distribution of the multisensory localization estimates is minimized.The visual (w V ) and auditory (w A ) weights were calculated using the following equations: Var V and Var A indicate the variance in saccade landing points in the unisensory visual and auditory conditions, respectively (see Equation 3).The final optimal multisensory mean and variance are given by Equation (5) (AV mean) and Equation (6) (AV variance): with M AV representing the optimal multisensory estimate of the target location, M V and M A representing the observed unisensory mean saccade landing points, and w V and w A denoting the weight given to the visual and auditory system based on the inverse of the variance of the unisensory response distributions.Var AV denotes the optimal variance of the multisensory location estimate, and Var A and Var V represent the observed variance of the unisensory location estimates.Var AV refers to a statistical optimum where the combination of the variances of the visual and auditory components is integrated, resulting in a lower variance for saccades towards audiovisual targets (i.e., more precise; also, see Alais & Burr, 2019).
Optimal cue integration can lead to a reduction of noise (variance) in multisensory relative to unisensory conditions.We subtracted the multisensory variance in each condition from the minimum of the related unisensory variances of each condition to determine the amount of multisensory variance reduction.Positive values indicate that the variance in the multisensory condition is lower than in the component unisensory conditions.
To investigate the extent to which saccade landing points towards audiovisual stimuli are influenced by the reliability of the unisensory estimates, the predicted and observed saccade accuracy and precision of saccade landing points were compared.Comparing the predictions made by the optimal cue integration model with the observed values can show whether humans combine unisensory information in a statistically optimal way in NH and with AHL.

| Assumptions
For each test, outliers were explored using boxplot visualization and distributions were checked using visual inspection and Shapiro-Wilk's test of normality (α = .05).Non-parametric tests were used when the assumption of normality was violated.Whenever the assumption of sphericity was violated, the Greenhouse-Geisser correction was used to correct the degrees of freedom.

| Manipulation check
Testing the effectiveness of our manipulation (plugging the right ear) was twofold.First, we performed the equal loudness task (see Section 2.3) to test whether we induced an asymmetry in hearing.Second, we analysed unisensory performance (latency, accuracy and precision) to assure that auditory but not visual localization was influenced by inserting the earplug.This verification was critical, as impaired auditory localization is a requirement to create the spatial misalignment of visual and auditory information in the audiovisual conditions.Saccade latency towards unisensory stimuli was compared for NH and AHL with repeated-measures ANOVAs with the factors hearing type (NH, AHL), target type (A, V reliable, V unreliable) and location (left, right).Saccade accuracy was tested with a repeated-measures ANOVA with the factors hearing type (NH, AHL), target type (A, V reliable, V unreliable), and location (left, right).For saccade precision, a non-parametric Friedman test with the factors hearing type (NH, AHL), target type (A, V reliable, V unreliable) and target location (left, right) was conducted to investigate differences in saccade landing point variance between conditions.

| Multisensory performance
Temporal domain Saccade latency towards audiovisual stimuli was compared for NH and AHL using a 2 Â 2 Â 2 repeatedmeasures ANOVA with the factors hearing type (NH, AHL), visual reliability (reliable, unreliable) and target location (left, right).Second, the amount of rMRE served as a descriptive measure of MRE.rMRE was tested with a Bayesian Wilcoxon signed-rank test to see whether the rMRE was different from zero, and repeated-measures ANOVA was used to test for differences between conditions.Although rMRE shows whether a multisensory benefit occurred (being faster than the fastest unisensory response), it does not yet show whether or not this benefit occurred due to MSI or rather due to statistical facilitation.Therefore, RMI violation was calculated as a reflection of MSI, and we compared (the amount of) RMI violation between NH and after acute AHL.Here, a Friedman test (non-parametric due to a violation of normality) was used with the factors hearing type (NH, AHL), visual reliability (reliable, unreliable), and target location (left, right), and post hoc non-parametric pairwise comparisons were performed to test differences between conditions.

Spatial domain
Saccade accuracy and precision towards audiovisual stimuli were compared between NH and AHL using a 2 Â 2 Â 2 repeated-measures ANOVA with the factors hearing type (NH, AHL), visual reliability (reliable, unreliable) and target location (left, right).MLE was used to assess cue integration in NH and AHL.Here, we tested the relative weights attributed to either the visual or auditory estimates with a Friedman test with the factors hearing type (NH, AHL), visual reliability (reliable, unreliable) and target location (left, right).We tested the difference between the predicted and observed estimates for saccade mean endpoints and variance to assess whether the model of optimal cue integration could predict multisensory behaviour with NH and after AHL.To this end, a Friedman test with the factors hearing type (NH, AHL), visual reliability (reliable, unreliable) and target location (left, right) was conducted for saccade accuracy.For precision, we first used a Wilcoxon signed-rank test to determine whether there was significant multisensory variance reduction (i.e., a difference from zero).Second, we conducted a Friedman test with the factors hearing type (NH, AHL), visual reliability (reliable, unreliable) and target location (left, right) to understand whether there were differences between conditions in terms of the amount of deviation from optimal.A normality check for saccade landing point data for each participant and each condition in NH and AHL was performed is described in the Supporting Information.

Auditory spatial and audiovisual temporal interaction
When auditory and visual input fall within the spatial binding window, the probability of MSI is higher (Frens et al., 1995;Meredith & Stein, 1986;Spence, 2013).So, when a conflict arises between the senses with AHL, chances are that the visual and auditory unisensory estimates no longer fall within this spatial binding window reducing multisensory benefits.As larger auditory localization error increases the chances of such a spatial mismatch, we analysed whether the auditory localization error was related to audiovisual saccade latency in the audiovisual condition and (the amount of) RMI violation as a measure of MSI.

| Confound of condition order
The paradigm used two physical locations to present sound and light.This might have induced stereotypical oculomotor behaviour towards these locations.We have counterbalanced the blocks to account for potential learning in the NH condition to carry over to AHL localization, and we compared the effects between the different order of conditions to assure that learning was not the case.We compared unisensory and multisensory performance (latency, accuracy and precision) in the AHL condition between participants that completed the NH block first and those that completed the AHL block first and reported the results in the Supporting Information.

| Manipulation check
Extensive descriptions of the results of the equal loudness task and a detailed analysis of unisensory performance with NH and AHL (latency, accuracy, precision) can be found in the Supporting Information.The results confirm that participants had no existing asymmetries in hearing in the NH condition and that the earplug induced a significant amount of AHL throughout the entire experiment.AHL did not affect saccade latencies to unisensory visual targets, but greatly increased the saccade latency for left and right unisensory auditory targets.The effect of the earplug on auditory saccade latencies was unlikely to be explained by changes in the overall sound intensity of the auditory input as demonstrated by a comparison with a reduced intensity sound condition in the NH condition.A saccade accuracy analysis showed that, on average, participants landed closer to the target location with NH than with AHL.Accuracy was lower for auditory than visual targets, and accuracy was lower for unreliable visual targets than for reliable visual targets.Auditory accuracy decreased with AHL but more so for auditory targets that were present on the side of the earplug (right) than targets that were presented on the side of the open (left) ear.Although saccades towards normal intensity sounds (60 dB) landed slightly closer to the actual auditory target location compared with reduced intensity sounds (44 dB), our results indicate that the influence of AHL on auditory saccade accuracy cannot fully be explained by a reduced perceived auditory sound intensity caused by the earplug.For saccade precision, the results indicate that the variance in saccade landing points greatly increased during AHL and was much larger for auditory targets than for visual targets.No effect of sound intensity or target location was found.Overall, these results confirm that our AHL manipulation greatly affected spatial and temporal properties of saccades towards auditory targets compared with NH.See the Supporting Information for detailed statistical results.Also, see Figures 3 and 4 for visualization of the unisensory performance alongside the multisensory performance.
These results show that AHL increases saccade latencies for audiovisual targets, more dramatically so for targets presented on the side of the affected ear (see Figure 3a,b).

Relative multisensory response enhancement
To investigate the presence of multisensory facilitation or inhibition in both visual reliability conditions in the NH and AHL condition, Bayesian Wilcoxon signed-rank tests were used to test whether the rMRE was different from zero.Significant multisensory facilitation was observed in all conditions with NH and AHL (i.e., mean rMRE > 0; all p < .01[Bonferroni corrected], all d > .6),except for the unreliable visual right target AHL condition (i.e., on the side of the earplug; p > .05,Bonferroni corrected).
Further comparisons and visualization of the amount of rMRE between conditions can be found in the Supporting Information.

Race model inequality violation
To test whether the observed rMRE could be explained by independent sensory processing (i.e., statistical facilitation), the RMI violation area was tested against zero using non-parametric Wilcoxon signed-rank tests.Significant RMI violation was observed in all NH and AHL conditions (all p < .01,all d > .6),indicating MSI rather than statistical facilitation.
Given that interactions are not tested in a Friedman test, post hoc non-parametric pairwise comparisons were performed to test for differences between the NH and AHL condition per target location and visual reliability.For unreliable audiovisual targets presented to the left of fixation, the difference between NH and AHL seemed especially apparent (see Figure 3c).However, after Bonferroni correction, this difference was no longer significant (all p > .05,Bonferroni corrected).
Overall, these findings suggest that the amount of RMI violation was not reduced by AHL at a group level.However, there were indications of an asymmetry in the amount of RMI violation between targets presented to the left versus the right side (i.e., the side of the plugged ear) of fixation.If anything, there was an increase in RMI violation for targets that were presented at the side of the open ear during AHL, but only for audiovisual targets with an unreliable visual component.To further understand the impact of AHL on MSI and sensory weighting, the spatial characteristics of saccades (accuracy and precision) towards unisensory and multisensory targets were analysed in the context of optimal cue integration (i.e., MLE and later in the results section).

Saccade accuracy
To investigate whether the observed saccade landing points in the audiovisual condition were in line with an optimal observer model, we compared the predicted and observed audiovisual saccade amplitudes.The predicted amplitude is the weighted average of the unisensory saccade amplitudes (see Equation 3).We analysed whether the observed multisensory behaviour in each condition was in line with the optimal location estimate or not and whether it deviated more from optimal during AHL as compared with NH.To simplify the analysis, we first subtracted the predicted saccade landing points from the observed landing points in the audiovisual conditions to get the amount of deviation from what would be considered statistically optimal.
A comparison of the marginal means of the levels of hearing type with zero indicated no significant difference from optimal in the NH condition (t[39.643]= 2.118, p = .081)but a significant difference from zero (i.e., from optimal) with AHL (t[39.643]= 7.962, p < .001;see Figure 5b).This indicates that in AHL, the model of optimal cue integration did not accurately predict saccade accuracy for audiovisual targets.

Saccade precision
We first analysed whether there was multisensory variance reduction (as expected from optimal cue integration) relative to the unisensory condition with the lowest variance in each of the four multisensory conditions.Using Wilcoxon signed-rank tests, we determined whether there was significant multisensory variance reduction (i.e., a difference from zero).There was some multisensory variance reduction in the NH condition, but only the NH, reliable visual, left target condition survived a strict Bonferroni correction ( p uncorrected < .001;all other p uncorrected < .05).There was no significant multisensory variance reduction in the AHL condition (all p uncorrected > .15),indicating no multisensory precision benefit due to MSI with AHL.
To investigate whether the observed variance in saccade landing points in the multisensory condition was in line with optimal cue integration, the difference between the observed and predicted saccade variance in the audiovisual conditions was compared with zero (see Figure 5c).The results indicated that the variance was not significantly different from optimal in any of the conditions (after correction for multiple comparisons, all p > .05),except for the NH, unreliable visual, right target condition.
To understand whether there were differences between conditions in terms of the amount of deviation from optimal, we conducted a Friedman test with the factors hearing type (NH, AHL), visual reliability (reliable, unreliable) and target location (left, right).There was only a main effect of visual reliability (χ 2 = 13.169,p < .001;other effects: χ 2 < 1.1, p > .3),indicating that the observed variance deviated more from optimal when multisensory targets contained an unreliable rather than a reliable visual component (see Figure 5c).
Overall, saccade landing point variance in the audiovisual condition was well predicted by optimal cue integration.However, the observed saccade variance was best predicted when the visual component was highly reliable.
The analysis above is based on group averages, meaning that predictions for individuals might be balanced out.To gain insight on the differences between observed and predicted outcomes in the individual, we have plotted individual data points for saccade endpoint and variance in NH and AHL across the different conditions (Figure 6).The plots show that there is a deviation from optimal for AHL (i.e., observed and predicted do not align), whereas for NH, the observed values quite nicely fit the prediction.From this, we infer that the optimal cue model is fairly good in predicting behaviour, but that in AHL, the model predicts actual behaviour more poorly.

| Auditory spatial and audiovisual temporal interaction
As a spatial conflict induced by mislocalization of auditory targets was hypothesized to result in impaired multisensory processing, we assessed whether the degree of mislocalization (hence, auditory spatial distortion leading to a multisensory conflict) was related to the multisensory response times.There was a significant correlation between the size of auditory localization errors and the saccade latency for audiovisual targets presented on the side of space of the plugged ear (one-tailed, reliable visual target: r = 0.442, p = .020;unreliable visual target: r = 0.443, p = .020).This indicates that participants who had larger auditory localization errors with AHL also had longer saccade latencies in the audiovisual conditions.
We further plotted the relation between the average unisensory auditory localization error (in the horizontal plane) against RMI violation for each condition (see Figure 7).There seemed to be a relation between the size of the auditory localization error and the amount of RMI violation for targets appearing on the side of the earplug (top and bottom right panels).Therefore, we explored this relationship by calculating correlations.There was a significant negative correlation between auditory localization error and RMI violation for audiovisual targets appearing on the side of the earplug (one-tailed, reliable visual target: r = .391,p = .036,unreliable visual target: r = .396,p = .034).The larger the unisensory auditory localization error, the less RMI violation, in line with the principle of (perceived) spatial alignment.

| DISCUSSION
Although much has been learned about how the senses work together to optimally perceive the world around us, knowledge about the impact of hearing impairments on MSI during spatial perception is currently scarce.With an ever-growing number of cases of hearing loss (WHO, 2020), the goal of the current study was to examine how acute AHL influences unisensory and multisensory spatial localization, and whether and how humans compensate for this disruption of unisensory auditory input.
As expected, acute AHL drastically impacted auditory localization, especially for sounds that were presented on the side of space of the impaired ear.Although auditory information was mislocalized when presented alone, this mislocalization was not observed when the auditory information was presented synchronously with visual information.Put differently, with conductive AHL, participants relied heavily on visual information to accurately localize the audiovisual stimulus.This intact audiovisual localization accuracy came at a cost: a reduced speed of audiovisual localization.This speed cost was related to the degree of auditory mislocalization: larger auditory localization errors (and thus, larger perceived audiovisual spatial conflict) lead to slower-but still accurate-localization.
This finding is reminiscent of earlier studies that experimentally induced a spatial disparity between sound and light by presenting these from different locations (Colonius et al., 2009;Frens et al., 1995;Steenken et al., 2008).Saccadic latencies increased with larger disparities between sound and light.Although these perceptual disparities were experimentally induced, the results fit with our argumentation that the earplug would induce a perceived distance between sound and light that, alike in the earlier studies, comes at the cost of speed when orienting towards a multisensory stimulus.Furthermore, we show that auditory localization impairment also influences auditory reliability (precision), further affecting (over)weighting of visual input in multisensory stimuli to accurately localize these.The use of an earplug to simulate AHL of course limits the generalizability to other types of hearing loss or AHL in the chronic phase.Yet, it offers a solid framework to investigate the impact of a spatial discrepancy induced by acute AHL on MSI.Plugging one ear is more frequently used as an experimental (and reversible) manipulation of AHL.For example, Van Wanrooij and Van Opstal (2007) described the effects of unilateral plugging on sound localization and found similar results with head movements as we found for eye movements, arguing for profound acute effects.Our study adds to this knowledge by not only studying the effect on unisensory localization but also on MSI by using two commonly used frameworks to assess MSI (RMI violation and MLE).
The finding that humans still accurately localize audiovisual events despite compromised unisensory auditory input is not fully in line with the well-known model of optimal cue integration (Alais & Burr, 2004).In this model, the observer uses the reliability of sensory information to optimally integrate auditory and visual inputs.As a result, a lower reliability of auditory information due to AHL the optimal observer should rely more on visual information.However, although our participants performed in line with the optimal observer model in the NH condition, they relied more on visual information than the optimal observer model would predict in the AHL condition.This difference can partly be explained by the distribution of saccade landing points deviating more from normality in AHL than in NH (see Supporting Information).However, the conclusion remains the same: with AHL participants rely more on visual inputs than expected by unisensory reliability and MLE.
One explanation for the unexpectedly stronger reliance on visual information is that optimal cue integration theory does not consider the accuracy of the unisensory estimates.It therefore only provides a statistical comprehension of attributing weight to sensory information based on precision when combining inputs from multiple senses.The work presented here points towards a more adaptive way of combining unisensory input: although optimal cue integration theory might hold for bottom-up sensory weighting, it is not sufficient to describe or predict orienting behaviour in our paradigm in acute AHL.Therefore, we propose that top-down factors can influence the sensory weighting process.As participants are aware that their ear is plugged in the AHL condition, a cognitive mechanism could consider the acute lower reliability of auditory input by increasing the weight on the information that participants know to be as reliable as before AHL: vision.So how could sensory weighting be influenced in a top-down way?Attention to a sensory modality can have a profound influence on how much weight this sensory information receives that influences spatial estimates in the visual and parietal cortex (Ferrari & Noppeney, 2021).This explanation fits the idea that with AHL, observers weigh sensory information based on what is optimal given their sensory circumstances, and not solely based on sensory precision.
Next to deliberately putting more weight on visual information in these 'acute' situations of AHL, this type of sensory weighting may also serve recalibration of the distorted sense in the longer term.Strelnikov and colleagues (Strelnikov et al., 2011) conducted a study of distortions in auditory localization after multiple-day plugging one ear of participants.As expected, acute AHL drastically affects auditory localization accuracy.They then compared different types of auditory localization training to see which type of training improved auditory localization with AHL the most.They observed that audiovisual training yielded the biggest improvement in auditory localization (more than visual training alone) when tested over the course of 5 days.Although the auditory system is able to employ some spontaneous recalibration over time on both the horizontal as the vertical plane without audiovisual training as well-as it may learn by exposure which cues are still intact, and how to reconcile those with the impaired cues over the course of time (Van Wanrooij & Van Opstal, 2005)-studies repeatedly find the beneficial effect of the multisensory component in recalibration.In animals a similar observation was made: multisensory training of deafened ferrets with bilateral Cochlear Implants (CI) enhanced auditory cortex neuronal responsivity and their sensitivity to ILD cues (Isaiah et al., 2014).
Extending these findings for longer-term hearing loss, it has been found that in chronic conductive hearing loss, interaural cues could also be recalibrated with visual guidance to improve auditory localization (see Keating & King, 2013;King, 2009).These findings clearly argue for multisensory stimulation to help recalibration in acute AHL and-in line with our suggestion-make a case for necessity of a visual component to over-rely on to help steer auditory localization in the right direction.Concordantly, Venskytis et al. (2019) found that people overweighed visual information, especially on the impaired side, in acute (plugged for tens of minutes) and severe chronic (>12 years) unilateral hearing loss, implying a compensatory role for vision to restore perceptual asymmetries even after long-term hearing loss.All in all, we suggest that over-relying on visual input may facilitate recalibration of auditory signals in the long run.How this recalibration works exactly over the course of time still remains a question for future research.
In our paradigm, we induced 32-dB hearing loss causing an asymmetry in ILD to examine its impact on audiovisual integration in a controlled environment with little other sensory input than our stimuli.We used high-pass filtered sounds as auditory stimuli because we hypothesized that this type of stimulus would show the greatest impact of AHL on audiovisual integration given that localization of high-pass filtered sounds relies strongly on using ILDs.However, in daily life, sounds are often richer in their frequency composition.This may allow hearingimpaired individuals to use other cues for auditory localization in daily life (e.g., ITD and HTRF).Therefore, one could argue that the impact of AHL on MSI observed here is an overestimation of the effect.However, hearing loss of 32 dB (as simulated in this study) is considered mild in the clinical practice.One could therefore argue that a greater asymmetry in hearing would have an even larger impact on sensory weighting and MSI.
To conclude, our study shows that even though AHL drastically distorts auditory localization, localization of audiovisual events is relatively unaffected in terms of localization accuracy and precision.This suggests that humans find a way to deal with conflicting sensory estimates optimally after AHL by overweighting visual information and maintain spatial accuracy.Whenever sound originates out of sight, however, people with AHL will not be able to localize it accurately, with the risk of misorienting towards a dangerous situation (e.g., in traffic).Therefore, it is important for patients with AHL to actively explore their environment visually.Considering how the senses are used in AHL in daily life can provide insights into why statistically optimal cue integration does not fully explain multisensory orienting behaviour.Currently, the model does not account for cognitive weighting, attention to specific modalities, or sensory accuracy.This argues for an approach in modelling that does not only consider sensory reliability but also encapsulates top-down cognitive factors that can affect sensory weighting.
U R E 1 (a) A schematic bird's-eye view of the setup.In simulated asymmetrical hearing loss, the plug is inserted in the right ear.(b) A schematic depiction of a visual-only trial.
U R E 3 (a) + (b): The average saccade latency for NH (black) and AHL (red) in the different unisensory and multisensory stimulus conditions (left or right presentation) for reliable visual stimuli (a) and unreliable visual stimuli (b).The small letter 'a' (blue) indicates reduced intensity sounds (44 dB(A)).The larger letter 'A' indicates normal intensity sounds (60 dB(A)).'V' indicates the visual stimulus.'AV' refers to the multisensory stimulus.(c) The average RMI violation in the different multisensory conditions (reliable or unreliable visual component, left or right presentation) in NH (black) and AHL (red).Error bars indicate standard errors.U R E 4 (a) Horizontal saccade accuracy in each condition.(b) Horizontal saccade variance in each condition.Error bars indicate standard errors.NH (black) = Normal hearing with a 60-dB stimulus, NH 44dB (blue) = normal hearing with a 44-dB stimulus, AHL (red) = asymmetric hearing loss with a 60-dB stimulus.Subscript R = reliable, subscript U = unreliable visual stimulus.

F
I G U R E 5 (a) The average relative unisensory visual weight for NH and AHL and for reliable (R) and unreliable (U) visual targets for left and right target locations based on unisensory saccade variance.(b) The difference between the observed and the MLE prediction of the average saccade mean endpoints for audiovisual targets for the NH and AHL condition.(c) The difference between the observed and the MLE prediction of the average saccade variance for audiovisual targets for the NH and AHL condition.NH (black) = Normal hearing, AHL (red) = asymmetric hearing loss.Error bars indicate standard errors.
Observed saccade landing positions (a, b) and landing position variance (c, d) for audiovisual targets versus MLEs of these saccade metrics for normal hearing (NH; a, c) and plugged hearing (AHL; b, d), for left (circles) and right (triangles) targets, and reliable (black) and unreliable visual stimuli (red).

F
I G U R E 7 The relation between unisensory auditory localization error and RMI violation in each multisensory condition during NH (black) and AHL (red) for left (a, c) and right (b, d) targets with a reliable (a, b) and unreliable (c, d) visual component.