A Hierarchical Sonification Framework Based on Convolutional Neural Network Modeling of Musical Genre

Geng S, Ren G, Ogihara M. A Hierarchical Sonification Framework Based on Convolutional Neural Network Modeling of Musical Genre. In: AES Convention 141. Audio Engineering Society; 2016. Paper 9628. Available from: https://aes.org/publications/elibrary-page/?id=18432

@inproceedings{Geng2016_18432,
  author = {Geng, Shijia and Ren, Gang and Ogihara, Mitsunori},
  title = {{A Hierarchical Sonification Framework Based on Convolutional Neural Network Modeling of Musical Genre}},
  booktitle = {AES Convention 141},
  note = {Paper 9628},
  year = {2016},
  month = sep,
  publisher = {Audio Engineering Society},
  url = {https://aes.org/publications/elibrary-page/?id=18432}
}

TY  - CPAPER
TI  - A Hierarchical Sonification Framework Based on Convolutional Neural Network Modeling of Musical Genre
AU  - Geng, Shijia
AU  - Ren, Gang
AU  - Ogihara, Mitsunori
T2  - AES Convention 141
M1  - Paper 9628
PY  - 2016
DA  - 2016/09/06
UR  - https://aes.org/publications/elibrary-page/?id=18432
PB  - Audio Engineering Society
LA  - en
AB  - Convolutional neural networks have satisfactory discriminative performances for various music-related tasks. However, the models are implemented as “black boxes” and thus their processed representations are non-transparent for manual interactions. In this paper, a hierarchical sonification framework with a musical genre modeling module and a sample-level sonification module has been implemented for aural interaction. The modeling module trains a convolutional neural network from musical signal segments with genre labels. Then the sonification module performs sample-level modification according to each convolutional layer, where lower sonification levels produce auralized pulses and higher sonification levels produce audio signals similar to the input musical signal. The usage of the proposed sonification framework is demonstrated using a musical stylistic morphing example.
ER  -