From Spatial Recording to Immersive Reproduction—Design & Implementation of a 3DOF Audio-Visual VR System

Kentgens M, Kühl S, Antweiler C, Jax P. From Spatial Recording to Immersive Reproduction—Design & Implementation of a 3DOF Audio-Visual VR System. In: AES Convention 145. Audio Engineering Society; 2018. Paper 10077. Available from: https://aes.org/publications/elibrary-page/?id=19803

@inproceedings{Kentgens2018_19803,
  author = {Kentgens, Maximillian and Kühl, Stefan and Antweiler, Christiane and Jax, Peter},
  title = {{From Spatial Recording to Immersive Reproduction—Design \& Implementation of a 3DOF Audio-Visual VR System}},
  booktitle = {AES Convention 145},
  note = {Paper 10077},
  year = {2018},
  month = oct,
  publisher = {Audio Engineering Society},
  url = {https://aes.org/publications/elibrary-page/?id=19803}
}

TY  - CPAPER
TI  - From Spatial Recording to Immersive Reproduction—Design & Implementation of a 3DOF Audio-Visual VR System
AU  - Kentgens, Maximillian
AU  - Kühl, Stefan
AU  - Antweiler, Christiane
AU  - Jax, Peter
T2  - AES Convention 145
M1  - Paper 10077
PY  - 2018
DA  - 2018/10/06
UR  - https://aes.org/publications/elibrary-page/?id=19803
PB  - Audio Engineering Society
LA  - en
AB  - The complex mutual interaction between human visual perception and hearing demands combined examinations of 360° video and spatial audio systems for Virtual Reality (VR) applications. Therefore, we present a joint audio-visual end-to-end chain from spatial recording to immersive reproduction with full rotational three degrees of freedom (3DOF). The audio-subsystem is based on Higher Order Ambisonics (HOA) obtained by Spherical Microphone Array (SMA) recordings, while the video is captured with a 360° camera rig. A spherical multi-loudspeaker setup for audio in conjunction with a VR head-mounted video display is used to reproduce a scene as close as possible to the original scene with regard to the perceptual modalities of the user. A database of immersive content as a basis for future research in spatial signal processing was set up by recording several rehearsals and concerts of the Aachen Symphony Orchestra. The data was used for a qualitative assessment of the eligibility of the proposed end-to-end system. A discussion shows the potential and limitations of the approach. Therein, we highlight the importance of coherent audio and video to achieve a high degree of immersion with VR recordings.
ER  -