Designing Neural Synthesizers for Low-Latency Interaction

Caspe F, Shier J, Sandler M, Saitis C, McPherson A. Designing Neural Synthesizers for Low-Latency Interaction. J Audio Eng Soc. 2025;73(5):240-255. doi:10.17743/jaes.2022.0204

@article{Caspe2025_22820,
  author = {Caspe, Franco and Shier, Jordie and Sandler, Mark and Saitis, Charalampos and McPherson, Andrew},
  title = {{Designing Neural Synthesizers for Low-Latency Interaction}},
  journal = {Journal of the Audio Engineering Society},
  volume = {73},
  number = {5},
  pages = {240--255},
  year = {2025},
  month = may,
  publisher = {Audio Engineering Society},
  doi = {10.17743/jaes.2022.0204},
  url = {https://doi.org/10.17743/jaes.2022.0204}
}

TY  - JOUR
TI  - Designing Neural Synthesizers for Low-Latency Interaction
AU  - Caspe, Franco
AU  - Shier, Jordie
AU  - Sandler, Mark
AU  - Saitis, Charalampos
AU  - McPherson, Andrew
T2  - Journal of the Audio Engineering Society
J2  - J. Audio Eng. Soc.
VL  - 73
IS  - 5
SP  - 240
EP  - 255
PY  - 2025
DA  - 2025/05/01
DO  - 10.17743/jaes.2022.0204
UR  - https://doi.org/10.17743/jaes.2022.0204
PB  - Audio Engineering Society
LA  - en
AB  - Neural audio synthesis (NAS) models offer interactive musical control over high-quality, expressive audio generators. While these models can operate in real time, they often suffer from high latency, making them unsuitable for intimate musical interaction. The impact of architectural choices in deep learning models on audio latency remains largely unexplored in the NAS literature. In this work, the authors investigate the sources of latency and jitter typically found in interactive NAS models. They then apply this analysis to the task of timbre transfer using the RAVE model (Realtime Audio Variational autoEncoder), a convolutional variational autoencoder for audio waveforms introduced by Caillon and Esling in 2021. Finally, an iterative design approach for optimizing latency is presented. This culminates with a model the authors call BRAVE (Bravely Realtime Audio Variational autoEncoder), which is low-latency and exhibits better pitch and loudness replication while showing timbre modification capabilities similar to RAVE. It is implemented in a specialized inference framework for low-latency, real-time inference, and a proof-of-concept audio plugin compatible with audio signals from musical instruments is presented. The authors expect the challenges and guidelines described in this document to support NAS researchers in designing models for low-latency inference from the ground up, enriching the landscape of possibilities for musicians.
ER  -