DDSP-Piano: A Neural Sound Synthesizer Informed by Instrument Knowledge

Renault L, Mignot R, Roebel A. DDSP-Piano: A Neural Sound Synthesizer Informed by Instrument Knowledge. J Audio Eng Soc. 2023;71(9):552-565. doi:10.17743/jaes.2022.0102

@article{Renault2023_22231,
  author = {Renault, Lenny and Mignot, Rémi and Roebel, Axel},
  title = {{DDSP-Piano: A Neural Sound Synthesizer Informed by Instrument Knowledge}},
  journal = {Journal of the Audio Engineering Society},
  volume = {71},
  number = {9},
  pages = {552--565},
  year = {2023},
  month = sep,
  publisher = {Audio Engineering Society},
  doi = {10.17743/jaes.2022.0102},
  url = {https://doi.org/10.17743/jaes.2022.0102}
}

TY  - JOUR
TI  - DDSP-Piano: A Neural Sound Synthesizer Informed by Instrument Knowledge
AU  - Renault, Lenny
AU  - Mignot, Rémi
AU  - Roebel, Axel
T2  - Journal of the Audio Engineering Society
J2  - J. Audio Eng. Soc.
VL  - 71
IS  - 9
SP  - 552
EP  - 565
PY  - 2023
DA  - 2023/09/06
DO  - 10.17743/jaes.2022.0102
UR  - https://doi.org/10.17743/jaes.2022.0102
PB  - Audio Engineering Society
LA  - en
AB  - Instrument sound synthesis using deep neural networks has received numerous improvements over the last couple of years. Among them, the Differentiable Digital Signal Processing (DDSP) framework has modernized the spectral modeling paradigm by including signal-based synthesizers and effects into fully differentiable architectures. The present work extends the applications of DDSP to the task of polyphonic sound synthesis, with the proposal of a differentiable piano synthesizer conditioned on MIDI inputs. The model architecture is motivated by high-level acoustic modeling knowledge of the instrument, which, along with the sound structure priors inherent to the DDSP components, makes for a lightweight, interpretable, and realistic-sounding piano model. A subjective listening test has revealed that the proposed approach achieves better sound quality than a state-of-the-art neural-based piano synthesizer, but physical-modeling-based models still hold the best quality. Leveraging its interpretability and modularity, a qualitative analysis of the model behavior was also conducted: it highlights where additional modeling knowledge and optimization procedures could be inserted in order to improve the synthesis quality and the manipulation of sound properties. Eventually, the proposed differentiable synthesizer can be further used with other deep learning models for alternative musical tasks handling polyphonic audio and symbolic data.
ER  -