When XR Meets AI: Integrating Interactive Machine Learning with an XR Musical Instrument

Barthet M, Graf M. When XR Meets AI: Integrating Interactive Machine Learning with an XR Musical Instrument. In: conference Paper. Audio Engineering Society; 2024. Paper 4. Available from: https://aes.org/publications/elibrary-page/?id=22436

@inproceedings{Barthet2024_22436,
  author = {Barthet, Mathieu and Graf, Max},
  title = {{When XR Meets AI: Integrating Interactive Machine Learning with an XR Musical Instrument}},
  note = {Paper 4},
  year = {2024},
  month = may,
  publisher = {Audio Engineering Society},
  url = {https://aes.org/publications/elibrary-page/?id=22436}
}

TY  - CPAPER
TI  - When XR Meets AI: Integrating Interactive Machine Learning with an XR Musical Instrument
AU  - Barthet, Mathieu
AU  - Graf, Max
M1  - Paper 4
PY  - 2024
DA  - 2024/05/28
UR  - https://aes.org/publications/elibrary-page/?id=22436
PB  - Audio Engineering Society
LA  - en
AB  - This paper explores the integration of artificial intelligence (AI) with extended reality (XR) through the development of Netz, an XR musical instrument (XRMI) designed to enhance expressive control using deep learning techniques. Netz implements algorithms to map physical gestures to musical controls, offering customisable control schemes that enhance gesture interpretation accuracy and elevate the overall musical experience. The instrument was developed through a participatory design process involving a professional keyboard player and music producer. The process spanned three phases with corresponding design sessions: exploration, making, and performance & refinement. Initial challenges with traditional computational approaches to hand-pose classification were overcome by incorporating an interactive machine learning (IML) system, enabling personalised gesture control. A set of musical performance tasks encompassing melodies and chord progressions were used to assess the instrument’s playability and expressivity in collaboration with our musician partner. Thematic analysis of reflective interviews revealed that the IML system enhanced musical interaction, suggesting AI’s potential to improve XR musical performance. Future work will involve a wider range of musicians to assess the generalisability of our findings.
ER  -