Leveraging Neural Representations for Audio Manipulation

Hawley SH, Steinmetz CJ. Leveraging Neural Representations for Audio Manipulation. In: AES Convention 154. Audio Engineering Society; 2023. Paper 96. Available from: https://aes.org/publications/elibrary-page/?id=22121

@inproceedings{Hawley2023_22121,
  author = {Hawley, Scott H. and Steinmetz, Christian J.},
  title = {{Leveraging Neural Representations for Audio Manipulation}},
  booktitle = {AES Convention 154},
  note = {Paper 96},
  year = {2023},
  month = may,
  publisher = {Audio Engineering Society},
  url = {https://aes.org/publications/elibrary-page/?id=22121}
}

TY  - CPAPER
TI  - Leveraging Neural Representations for Audio Manipulation
AU  - Hawley, Scott H.
AU  - Steinmetz, Christian J.
T2  - AES Convention 154
M1  - Paper 96
PY  - 2023
DA  - 2023/05/06
UR  - https://aes.org/publications/elibrary-page/?id=22121
PB  - Audio Engineering Society
LA  - en
AB  - We investigate applying audio manipulations using pretrained neural network-based autoencoders as an alternative to traditional signal processing methods, since the former may provide greater semantic or perceptual organization. To establish the potential of this approach, we first establish if representations from these models encode information about manipulations. We carry out experiments and produce visualizations using representations from two different pretrained autoencoders. Our findings indicate that, while some information about audio manipulations is encoded, this information is both limited and encoded in a non-trivial way. This is supported by our attempts to visualize these representations, which demonstrated that trajectories of representations for common manipulations are typically nonlinear and content dependent, even for linear signal manipulations. As a result, it is not yet clear how these pretrained autoencoders can be used to manipulate audio signals, however, our results indicate this may be due to the lack of disentanglement with respect to common audio manipulations.
ER  -