Conference Paper
Are Traditional Evaluation Methods Still Reliable for Neural Audio Codecs?
AES Show 2026 Nashville · Paper 37 · October 2026
Abstract
Neural audio codecs (NACs) represent a fundamentally different approach to audio compression compared to established signal processing-based codecs. Due to their relative novelty, there is currently no common consensus
on best practices in their evaluation. For application in commercial services, it is of special interest to guarantee that audio codec quality does not drop below a certain guaranteed level independently of the input signal type. Therefore, audio codecs are traditionally evaluated on sets of test items that contain both generic and critical items to achieve a balanced and comprehensive system evaluation. This raises the question of whether the same approach and test items remain sufficient for NACs. This paper investigates this question by conducting a side-by-side performance evaluation of two classic audio codecs (CACs) and three well-known NACs with subjective and objective methods. We conduct a MUSHRA listening test with both generic and critical items and a P.808 DCR test with a larger, less critical test set. We state that both test methods lead to similar quality rankings. Experiments further show interesting results in that strong transients, a typical CAC worst-case signal, are usually not challenging for several of the evaluated NACs while highly tonal signals remain critical items. NACs can exhibit types of artifacts that have not been observed before with CACs, such as hallucination effects. When re-evaluating the reliability of
various state-of-the-art perceptual quality measures, this behavior was found to make quality prediction difficult for most investigated measurement schemes.
