Confidently Wrong: Evaluating AudioSet-Trained Models Under Real-World Deployment

Garcia-Ballesteros M, Naranjo-Alcazar J, Grau-Haro J, Ribes-Serrano R, Zuccarello P. Confidently Wrong: Evaluating AudioSet-Trained Models Under Real-World Deployment. In: Convention Paper. Audio Engineering Society; 2026. Paper 10274. Available from: https://aes.org/publications/elibrary-page/?id=23215

@inproceedings{GarciaBallesteros2026_23215,
  author = {Garcia-Ballesteros, Marta and Naranjo-Alcazar, Javier and Grau-Haro, Jordi and Ribes-Serrano, Ruben and Zuccarello, Pedro},
  title = {{Confidently Wrong: Evaluating AudioSet-Trained Models Under Real-World Deployment}},
  note = {Paper 10274},
  year = {2026},
  month = may,
  publisher = {Audio Engineering Society},
  url = {https://aes.org/publications/elibrary-page/?id=23215}
}

TY  - CPAPER
TI  - Confidently Wrong: Evaluating AudioSet-Trained Models Under Real-World Deployment
AU  - Garcia-Ballesteros, Marta
AU  - Naranjo-Alcazar, Javier
AU  - Grau-Haro, Jordi
AU  - Ribes-Serrano, Ruben
AU  - Zuccarello, Pedro
M1  - Paper 10274
PY  - 2026
DA  - 2026/05/28
UR  - https://aes.org/publications/elibrary-page/?id=23215
PB  - Audio Engineering Society
LA  - en
AB  - Audio event-classification models trained on AudioSet are widely adopted and form a central component of the state of the art in machine listening, yet their behavior when deployed in complex, open acoustic environments remains largely unexplored. In this study, we evaluate several AudioSet-pretrained architecturesparticularly models from the PANNs family, including MobileNetV2, Wavegram LogMel CNN14, and the transformer-based PaSST modelwhen applied to a real operational scenario at the commercial Port of Valencia, Spain. We observed a recurring but model-dependent behavior: the models frequently assigned relatively high probability to the class Music for non-musical industrial and transportation sounds. These events included train wheels squealing, motorcycle acceleration, emergency sirens, and reversing beepssound categories that are common in port logistics environments but acoustically different from music. By analyzing the probability distributions output by the models, we demonstrate that this erroneous Music activation is not limited to isolated cases but is observed across multiple architectures, with varying intensity depending on the model and sound category. Our findings highlight limitations in the robustness and domain generalization of AudioSet-derived models and emphasize the need for targeted adaptation techniques when deploying them in real industrial settings.
ER  -