Style Transfer of Audio Effects with Differentiable Signal Processing

Steinmetz CJ, Bryan NJ, Reiss JD. Style Transfer of Audio Effects with Differentiable Signal Processing. J Audio Eng Soc. 2022;70(9):708-721. doi:10.17743/jaes.2022.0025

@article{Steinmetz2022_21883,
  author = {Steinmetz, Christian J. and Bryan, Nicholas J. and Reiss, Joshua D.},
  title = {{Style Transfer of Audio Effects with Differentiable Signal Processing}},
  journal = {Journal of the Audio Engineering Society},
  volume = {70},
  number = {9},
  pages = {708--721},
  year = {2022},
  month = sep,
  publisher = {Audio Engineering Society},
  doi = {10.17743/jaes.2022.0025},
  url = {https://doi.org/10.17743/jaes.2022.0025}
}

TY  - JOUR
TI  - Style Transfer of Audio Effects with Differentiable Signal Processing
AU  - Steinmetz, Christian J.
AU  - Bryan, Nicholas J.
AU  - Reiss, Joshua D.
T2  - Journal of the Audio Engineering Society
J2  - J. Audio Eng. Soc.
VL  - 70
IS  - 9
SP  - 708
EP  - 721
PY  - 2022
DA  - 2022/09/06
DO  - 10.17743/jaes.2022.0025
UR  - https://doi.org/10.17743/jaes.2022.0025
PB  - Audio Engineering Society
LA  - en
AB  - This work presents a framework to impose the audio effects and production style from one recording to another by example with the goal of simplifying the audio production process. A deep neural network was trained to analyze an input recording and a style reference recording and predict the control parameters of audio effects used to render the output. In contrast to past work, this approach integrates audio effects as differentiable operators, enabling backpropagation through audio effects and end-to-end optimization with an audio-domain loss. Pairing this framework with a self-supervised training strategy enables automatic control of audio effects without the use of any labeled or paired training data. A survey of existing and new approaches for differentiable signal processing is presented, demonstrating how each can be integrated into the proposed framework along with a discussion of their trade-offs. The proposed approach is evaluated on both speech and music tasks, demonstrating generalization both to unseen recordings and even sample rates different than those during training. Convincing production style transfer results are demonstrated with the ability to transform input recordings to produced recordings, yielding audio effect control parameters that enable interpretability and user interaction.
ER  -