A Variational Y-Autoencoder for Disentangling Gesture and Material of Interaction Sounds

Schwär S, Müller M, Schlecht SJ. A Variational Y-Autoencoder for Disentangling Gesture and Material of Interaction Sounds. In: AES Conference: AES 2022 International Audio for Virtual and Augmented Reality Conference. Audio Engineering Society; 2022. Paper 23. Available from: https://aes.org/publications/elibrary-page/?id=21853

@inproceedings{Schwar2022_21853,
  author = {Schwär, Simon and Müller, Meinard and Schlecht, Sebastian J.},
  title = {{A Variational Y-Autoencoder for Disentangling Gesture and Material of Interaction Sounds}},
  booktitle = {AES Conference: AES 2022 International Audio for Virtual and Augmented Reality Conference},
  note = {Paper 23},
  year = {2022},
  month = aug,
  publisher = {Audio Engineering Society},
  url = {https://aes.org/publications/elibrary-page/?id=21853}
}

TY  - CPAPER
TI  - A Variational Y-Autoencoder for Disentangling Gesture and Material of Interaction Sounds
AU  - Schwär, Simon
AU  - Müller, Meinard
AU  - Schlecht, Sebastian J.
T2  - AES Conference: AES 2022 International Audio for Virtual and Augmented Reality Conference
M1  - Paper 23
PY  - 2022
DA  - 2022/08/06
UR  - https://aes.org/publications/elibrary-page/?id=21853
PB  - Audio Engineering Society
LA  - en
AB  - Appropriate sound effects are an important aspect of immersive virtual experiences. Particularly in mixed reality scenarios it may be desirable to change the acoustic properties of a naturally occurring interaction sound (e.g., the sound of a metal spoon scraping a wooden bowl) to a sound matching the characteristics of the corresponding interaction in the virtual environment (e.g., using wooden tools in a porcelain bowl). In this paper, we adapt the concept of a Y-Autoencoder (YAE) to the domain of sound e?ect analysis and synthesis. The YAE model makes it possible to disentangle the gesture and material properties of sound e?ects with a weakly supervised training strategy where only an identifier label for the material in each training example is given. We show that such a model makes it possible to resynthesize sound e?ects after exchanging the material label of an encoded example and obtain perceptually meaningful synthesis results with relatively low computational e?ort. By introducing a variational regularization for the encoded gesture, as well as an adversarial loss, we can further use the model to generate new and varying sound e?ects with the material characteristics of the training data, while the analyzed audio signal can originate from interactions with unknown materials.
ER  -