Toward a Statistically Well-Grounded Evaluation of Listening Tests—Avoiding Pitfalls, Misuse, and Misconceptions

Nagel F, Sporer T, Sedlmeier P. Toward a Statistically Well-Grounded Evaluation of Listening Tests—Avoiding Pitfalls, Misuse, and Misconceptions. In: AES Convention 128. Audio Engineering Society; 2010. Paper 8146. Available from: https://aes.org/publications/elibrary-page/?id=15442

@inproceedings{Nagel2010_15442,
  author = {Nagel, Frederik and Sporer, Thomas and Sedlmeier, Peter},
  title = {{Toward a Statistically Well-Grounded Evaluation of Listening Tests—Avoiding Pitfalls, Misuse, and Misconceptions}},
  booktitle = {AES Convention 128},
  note = {Paper 8146},
  year = {2010},
  month = may,
  publisher = {Audio Engineering Society},
  url = {https://aes.org/publications/elibrary-page/?id=15442}
}

TY  - CPAPER
TI  - Toward a Statistically Well-Grounded Evaluation of Listening Tests—Avoiding Pitfalls, Misuse, and Misconceptions
AU  - Nagel, Frederik
AU  - Sporer, Thomas
AU  - Sedlmeier, Peter
T2  - AES Convention 128
M1  - Paper 8146
PY  - 2010
DA  - 2010/05/06
UR  - https://aes.org/publications/elibrary-page/?id=15442
PB  - Audio Engineering Society
LA  - en
AB  - Many recent publications in audio research present subjective evaluations of audio quality based on the Recommendation ITU-R BS.1534-1 (MUSHRA, MUltiple Stimuli with Hidden Reference and Anchor). This is a very welcome trend because it enables researchers to assess the implications of their developments. The evaluation of listening tests, however, sometimes sufers from an incomplete understanding of the underlying statistics. The present paper aims at identifying the causes for the pitfalls and misconceptions in MUSHRA evaluations. It exemplifes the impact of falsely used or even misused statistics. Subsequently, schemes for evaluating the listeners' judgments that are well-grounded on statistical considerations comprising an understanding of the concepts of statistical power and efect size are proposed.
ER  -