Mesostructures: Beyond Spectrogram Loss in Differentiable Time--Frequency Analysis

Vahidi C, Han H, Wang C, Lagrange M, Fazekas G, Lostanlen V. Mesostructures: Beyond Spectrogram Loss in Differentiable Time--Frequency Analysis. J Audio Eng Soc. 2023;71(9):577-585. doi:10.17743/jaes.2022.0103

@article{Vahidi2023_22233,
  author = {Vahidi, Cyrus and Han, Han and Wang, Changhong and Lagrange, Mathieu and Fazekas, György and Lostanlen, Vincent},
  title = {{Mesostructures: Beyond Spectrogram Loss in Differentiable Time--Frequency Analysis}},
  journal = {Journal of the Audio Engineering Society},
  volume = {71},
  number = {9},
  pages = {577--585},
  year = {2023},
  month = sep,
  publisher = {Audio Engineering Society},
  doi = {10.17743/jaes.2022.0103},
  url = {https://doi.org/10.17743/jaes.2022.0103}
}

TY  - JOUR
TI  - Mesostructures: Beyond Spectrogram Loss in Differentiable Time--Frequency Analysis
AU  - Vahidi, Cyrus
AU  - Han, Han
AU  - Wang, Changhong
AU  - Lagrange, Mathieu
AU  - Fazekas, György
AU  - Lostanlen, Vincent
T2  - Journal of the Audio Engineering Society
J2  - J. Audio Eng. Soc.
VL  - 71
IS  - 9
SP  - 577
EP  - 585
PY  - 2023
DA  - 2023/09/06
DO  - 10.17743/jaes.2022.0103
UR  - https://doi.org/10.17743/jaes.2022.0103
PB  - Audio Engineering Society
LA  - en
AB  - Computer musicians refer to mesostructures as the intermediate levels of articulation between the microstructure of waveshapes and the macrostructure of musical forms. Examples of mesostructures include melody, arpeggios, syncopation, polyphonic grouping, and textural contrast. Despite their central role in musical expression, they have received limited attention in recent applications of deep learning to the analysis and synthesis of musical audio. Currently, autoencoders and neural audio synthesizers are only trained and evaluated at the scale of microstructure, i.e., local amplitude variations up to 100 ms or so. In this paper, the authors formulate and address the problem of mesostructural audio modeling via a composition of a differentiable arpeggiator and time-frequency scattering. The authors empirically demonstrate that time--frequency scattering serves as a differentiable model of similarity between synthesis parameters that govern mesostructure. By exposing the sensitivity of short-time spectral distances to time alignment, the authors motivate the need for a time-invariant and multiscale differentiable time--frequency model of similarity at the level of both local spectra and spectrotemporal modulations.
ER  -