Real-Time Auralization Pipeline for First-Person Vocal Interaction in Audio-Visual Virtual Environments

Vargas MF, Bates VE, McDonnell R. Real-Time Auralization Pipeline for First-Person Vocal Interaction in Audio-Visual Virtual Environments. In: Express Paper. Audio Engineering Society; 2025.pp. 323:1-. Available from: https://aes.org/publications/elibrary-page/?id=22874

@inproceedings{Vargas2025_22874,
  author = {Vargas, Mauricio Flores and Bates, Vargas Enda and McDonnell, Rachel},
  title = {{Real-Time Auralization Pipeline for First-Person Vocal Interaction in Audio-Visual Virtual Environments}},
  pages = {323:1-},
  year = {2025},
  month = may,
  publisher = {Audio Engineering Society},
  url = {https://aes.org/publications/elibrary-page/?id=22874}
}

TY  - CPAPER
TI  - Real-Time Auralization Pipeline for First-Person Vocal Interaction in Audio-Visual Virtual Environments
AU  - Vargas, Mauricio Flores
AU  - Bates, Vargas Enda
AU  - McDonnell, Rachel
SP  - 1
M1  - Paper 323
PY  - 2025
DA  - 2025/05/12
UR  - https://aes.org/publications/elibrary-page/?id=22874
PB  - Audio Engineering Society
LA  - en
AB  - Multimodal research and applications are becoming more commonplace as Virtual Reality (VR) technology integrates different sensory feedback, enabling the recreation of real spaces in an audio-visual context. Within VR experiences, numerous applications rely on the users voice as a key element of interaction, including music performances and public speaking applications. Self-perception of our voice plays a crucial role in vocal production. When singing or speaking, our voice interacts with the acoustic properties of the environment, shaping the adjustment of vocal parameters in response to the perceived characteristics of the space.;;This technical report presents a real-time auralization pipeline that leverages three-dimensional Spatial Impulse Responses (SIRs) for multimodal research applications in VR requiring first-person vocal interaction. It describes the impulse response creation and rendering workflow, the audio-visual integration, and addresses latency and computational considerations. The system enables users to explore acoustic spaces from various positions and orientations within a predefined area, supporting three and five Degrees of Freedom (3Dof and 5DoF) in audio-visual multimodal perception for both research and creative applications in VR.;;The design of this pipeline arises from the limitations of existing audio tools and spatializers, particularly regarding signal latency, and the lack of SIRs captured from a first-person perspective and in multiple adjacent distributions to enable translational rendering. By addressing these gaps, the system enables real-time auralization of self-generated vocal feedback.
ER  -