Binaural Reproduction of Microphone Array Recordings With 2D Video in Mixed Reality

Lübeck T, Ben-Hur Z, Lou Alon D, Crukley J. Binaural Reproduction of Microphone Array Recordings With 2D Video in Mixed Reality. J Audio Eng Soc. 2025;73(7/8):461-470. doi:10.17743/jaes.2022.0213

@article{Lubeck2025_22924,
  author = {Lübeck, Tim and Ben-Hur, Zamir and Lou Alon, David and Crukley, Jeffery},
  title = {{Binaural Reproduction of Microphone Array Recordings With 2D Video in Mixed Reality}},
  journal = {Journal of the Audio Engineering Society},
  volume = {73},
  number = {7/8},
  pages = {461--470},
  year = {2025},
  month = jul,
  publisher = {Audio Engineering Society},
  doi = {10.17743/jaes.2022.0213},
  url = {https://doi.org/10.17743/jaes.2022.0213}
}

TY  - JOUR
TI  - Binaural Reproduction of Microphone Array Recordings With 2D Video in Mixed Reality
AU  - Lübeck, Tim
AU  - Ben-Hur, Zamir
AU  - Lou Alon, David
AU  - Crukley, Jeffery
T2  - Journal of the Audio Engineering Society
J2  - J. Audio Eng. Soc.
VL  - 73
IS  - 7/8
SP  - 461
EP  - 470
PY  - 2025
DA  - 2025/07/07
DO  - 10.17743/jaes.2022.0213
UR  - https://doi.org/10.17743/jaes.2022.0213
PB  - Audio Engineering Society
LA  - en
AB  - Head-worn devices equipped with microphone arrays and cameras can be used to capture the experience from a user’s perspective and reproduce it in virtual, mixed, or augmented reality. A concept that has recently introduced is to present the video capture as a 2D video screen augmented into the real-world environment through a mixed reality headset. This study presents such a system for reproducing audio and video capture from glasses arrays as a video “augment” along with binaural audio. Results of an initial listening experiment are presented, evaluating different state-of-the-art methods for binaural rendering. A stereo rendering through virtual loudspeakers attached to the video "augment” is compared with head-locked and world-locked binaural syntheses based on a binaural beamforming approach. The results suggest that listeners rated beamforming-based reproduction higher than stereo rendering. World-locked rendering was not rated significantly better than the head-locked version.
ER  -