Real-time binaural speech reproduction with deep neural networks and dynamic head tracking

Papavasileiou I, Thoidis I, Sevastiadis C, Vryzas N, Papanikolaou G. Real-time binaural speech reproduction with deep neural networks and dynamic head tracking. In: Express Paper. Audio Engineering Society; 2024. Paper 207. Available from: https://aes.org/publications/elibrary-page/?id=22553

@inproceedings{Papavasileiou2024_22553,
  author = {Papavasileiou, Ioannis and Thoidis, Iordanis and Sevastiadis, Christos and Vryzas, Nikolaos and Papanikolaou, George},
  title = {{Real-time binaural speech reproduction with deep neural networks and dynamic head tracking}},
  note = {Paper 207},
  year = {2024},
  month = jun,
  publisher = {Audio Engineering Society},
  url = {https://aes.org/publications/elibrary-page/?id=22553}
}

TY  - CPAPER
TI  - Real-time binaural speech reproduction with deep neural networks and dynamic head tracking
AU  - Papavasileiou, Ioannis
AU  - Thoidis, Iordanis
AU  - Sevastiadis, Christos
AU  - Vryzas, Nikolaos
AU  - Papanikolaou, George
M1  - Paper 207
PY  - 2024
DA  - 2024/06/06
UR  - https://aes.org/publications/elibrary-page/?id=22553
PB  - Audio Engineering Society
LA  - en
AB  - This study examines the efficacy of a deep learning-based binaural speech reproduction system via headphones that integrates real-time head tracking to simulate a stable speech source in a virtual environment. We developed a binaural speech reproduction system that operates in real time using a custom head-tracking device based on the Arduino platform and a pre-trained neural network model for binaural speech synthesis. The system dynamically adapts audio output based on the head orientation of the listener, operating with an overall latency of 70 ms on standard computing hardware. We conducted a subjective speech localization task with 15 young, normal-hearing participants. Initial results showed a 40% accuracy in localizing a static speech source, which significantly improved with dynamic head movements. Participants reported minimal dizziness and distortion, indicating good tolerability and sound quality. Despite some limitations, such as non-personalized settings and perceivable latency, the findings demonstrate the potential of deep learning approaches to enhance realism in virtual auditory environments. Further research is needed to refine these technologies for broader applications in immersive audio settings.
ER  -