When Spatial Sounds Affect the Ability to Apprehend Visual Information: A Physiological Approach

Mendonça C, Wang H, Pulkki V. When Spatial Sounds Affect the Ability to Apprehend Visual Information: A Physiological Approach. J Audio Eng Soc. 2023;71(10):679-688. doi:10.17743/jaes.2022.0092

@article{Mendonca2023_22240,
  author = {Mendonça, Catarina and Wang, Heng and Pulkki, Ville},
  title = {{When Spatial Sounds Affect the Ability to Apprehend Visual Information: A Physiological Approach}},
  journal = {Journal of the Audio Engineering Society},
  volume = {71},
  number = {10},
  pages = {679--688},
  year = {2023},
  month = oct,
  publisher = {Audio Engineering Society},
  doi = {10.17743/jaes.2022.0092},
  url = {https://doi.org/10.17743/jaes.2022.0092}
}

TY  - JOUR
TI  - When Spatial Sounds Affect the Ability to Apprehend Visual Information: A Physiological Approach
AU  - Mendonça, Catarina
AU  - Wang, Heng
AU  - Pulkki, Ville
T2  - Journal of the Audio Engineering Society
J2  - J. Audio Eng. Soc.
VL  - 71
IS  - 10
SP  - 679
EP  - 688
PY  - 2023
DA  - 2023/10/06
DO  - 10.17743/jaes.2022.0092
UR  - https://doi.org/10.17743/jaes.2022.0092
PB  - Audio Engineering Society
LA  - en
AB  - The current technological solutions for spatial audio provide realistic auditory impressions but rarely account for multisensory interactions. The intent of this study was to discover if and when spatial sounds could lower the accuracy of visual perception. Sequences of light and sound events were presented, and different sound parameters were tested: spatial and temporal congruency, horizontal and vertical spatial distribution, and source broadness. Participants were asked to report the location of the last visual event, in a left-right discrimination task. During the task, cognitive effort was monitored through pupil size measurements. It was found that both spatial and temporal congruence are important for higher accuracy levels and lower cognitive effort levels. However, spatial congruence was found to not be crucial, if sounds occur within the same spatial region as visual events. Sounds hindered the visual accuracy levels and increased effort when they occurred within a narrower or wider field than that of the visual events, but not too discrepant. These effects were replicated with vertical sound distributions.Broad sounds made the task more effortful and limited negative effects of spatially mismatched audiovisual events. When creating spatial sound for audiovisual reproductions, source distribution and broadness should be intentionally controlled.
ER  -