Phase-Aware Transformations in Variational Autoencoders for Audio Effects

Cámara M, Blanco JL. Phase-Aware Transformations in Variational Autoencoders for Audio Effects. J Audio Eng Soc. 2022;70(9):731-741. doi:10.17743/jaes.2022.0042

@article{Camara2022_21885,
  author = {Cámara, Mateo and Blanco, José Luis},
  title = {{Phase-Aware Transformations in Variational Autoencoders for Audio Effects}},
  journal = {Journal of the Audio Engineering Society},
  volume = {70},
  number = {9},
  pages = {731--741},
  year = {2022},
  month = sep,
  publisher = {Audio Engineering Society},
  doi = {10.17743/jaes.2022.0042},
  url = {https://doi.org/10.17743/jaes.2022.0042}
}

TY  - JOUR
TI  - Phase-Aware Transformations in Variational Autoencoders for Audio Effects
AU  - Cámara, Mateo
AU  - Blanco, José Luis
T2  - Journal of the Audio Engineering Society
J2  - J. Audio Eng. Soc.
VL  - 70
IS  - 9
SP  - 731
EP  - 741
PY  - 2022
DA  - 2022/09/06
DO  - 10.17743/jaes.2022.0042
UR  - https://doi.org/10.17743/jaes.2022.0042
PB  - Audio Engineering Society
LA  - en
AB  - This paper analyzes the impact of signal phase handling in one of the most popular architectures for the generative synthesis of audio effects: variational autoencoders (VAEs). Until quite recently, autoencoders based on the Fast Fourier Transform routinely avoided the phase of the signal. They store the phase information and retrieve it at the output or rely on signal phase regenerators such as Griffin--Lim. We evaluate different VAE networks capable of generating a latent space with intrinsic information from signal amplitude and phase. The Modulated Complex Lapped Transform (MCLT) has been evaluated as an alternative to the Short-Time Fourier Transform (STFT). A novel database on beats has been designed for testing the architectures. Results were objectively assessed (reconstruction errors and objective metrics approximating opinion scores) with autoencoders on STFT and MCLT representations, using Griffin--Lim phase regeneration, multichannel networks, as well as the Complex VAE. The autoencoders successfully learned to represent the phase information and handle it in a holistic approach. State-of-the-art quality standards were reached for audio effects. The autoencoders show a remarkable ability to generalize and deliver new sounds, while overall quality depends on the reconstruction of phase and amplitude.
ER  -