JoyCam: Blending Facial Recognition with Neural Activity measurement for Real-time Estimation of Listener Emotion

Williams D. JoyCam: Blending Facial Recognition with Neural Activity measurement for Real-time Estimation of Listener Emotion. In: Convention Paper. Audio Engineering Society; 2026. Paper 10267. Available from: https://aes.org/publications/elibrary-page/?id=23181

@inproceedings{Williams2026_23181,
  author = {Williams, Duncan},
  title = {{JoyCam: Blending Facial Recognition with Neural Activity measurement for Real-time Estimation of Listener Emotion}},
  note = {Paper 10267},
  year = {2026},
  month = may,
  publisher = {Audio Engineering Society},
  url = {https://aes.org/publications/elibrary-page/?id=23181}
}

TY  - CPAPER
TI  - JoyCam: Blending Facial Recognition with Neural Activity measurement for Real-time Estimation of Listener Emotion
AU  - Williams, Duncan
M1  - Paper 10267
PY  - 2026
DA  - 2026/05/28
UR  - https://aes.org/publications/elibrary-page/?id=23181
PB  - Audio Engineering Society
LA  - en
AB  - The ability to objectively measure listener emotion is a critical frontier for adaptive audio systems, healthcare, and personalized music therapy. While music is a powerful driver of affect, traditional self-reporting is often intrusive or inaccessible for users in wellbeing settings who may struggle to articulate their mood. This paper introduces JoyCam, a multimodal system that estimates subtle moments of joyful engagement by blending lightweight brain-wave monitoring (wearable EEG) with facial-expression sensing. By capturing physiological reactions that occur below the threshold of conscious awareness, the system creates a more stable emotional profile than single-modality methods. In our system, Facial joy is estimated via MediaPipe landmark analysis, focusing on normalized mouth-width deviations. Simultaneously, neurological engagement is tracked through Frontal Alpha Asymmetry (FAA) using an OpenBCI Cyton system. To address the sensitivity of EEG to movement, a dynamic artefact index down-weights neural signals during high-frequency interference. The system was tested in a pilot study with five participants. Preliminary results indicate that baseline-corrected physiological scores align closely with self-reported music impact and valence ratings across joyful and sad conditions. These findings suggest that JoyCam offers a robust framework for responsive musical companions that can adjust playlists or production parameters based on a listeners real-time physiological state
ER  -