Multi-Input Architecture and Disentangled Representation Learning for Multi-Dimensional Modeling of Music Similarity

Ribecky S, Abeßer J, Lukashevich H. Multi-Input Architecture and Disentangled Representation Learning for Multi-Dimensional Modeling of Music Similarity. In: AES Convention 152. Audio Engineering Society; 2022. Paper 10568. Available from: https://aes.org/publications/elibrary-page/?id=21681

@inproceedings{Ribecky2022_21681,
  author = {Ribecky, Sebastian and Abeßer, Jakob and Lukashevich, Hanna},
  title = {{Multi-Input Architecture and Disentangled Representation Learning for Multi-Dimensional Modeling of Music Similarity}},
  booktitle = {AES Convention 152},
  note = {Paper 10568},
  year = {2022},
  month = may,
  publisher = {Audio Engineering Society},
  url = {https://aes.org/publications/elibrary-page/?id=21681}
}

TY  - CPAPER
TI  - Multi-Input Architecture and Disentangled Representation Learning for Multi-Dimensional Modeling of Music Similarity
AU  - Ribecky, Sebastian
AU  - Abeßer, Jakob
AU  - Lukashevich, Hanna
T2  - AES Convention 152
M1  - Paper 10568
PY  - 2022
DA  - 2022/05/06
UR  - https://aes.org/publications/elibrary-page/?id=21681
PB  - Audio Engineering Society
LA  - en
AB  - In the context of music information retrieval, similarity-based approaches are useful for a variety of tasks that benefit from a query-by-example approach. Music however, naturally decomposes into a set of semantically meaningful factors of variation. Current representation learning strategies pursue the disentanglement of such factors from deep representations, and result in highly interpretable models. This allows to model the perception of music similarity, which is highly subjective and multi-dimensional. While the focus of prior work is on metadata driven similarity, we suggest to directly model the human notion of multi-dimensional music similarity. To achieve this, we propose a multi-input deep neural network architecture, which simultaneously processes mel-spectrogram, CENSchromagram and tempogram representations in order to extract informative features for different disentangled musical dimensions: genre, mood, instrument, era, tempo, and key. We evaluated the proposed music similarity approach using a triplet prediction task and found that the proposed multi-input architecture outperforms a state of the art method. Furthermore, we present a novel multi-dimensional analysis to evaluate the influence of each disentangled dimension on the perception of music similarity.
ER  -