Optimized DSP Implementation of Non-Linear Quantization

Subramaniam G, Rao R. Optimized DSP Implementation of Non-Linear Quantization. In: AES Convention 110. Audio Engineering Society; 2001. Paper 5363. Available from: https://aes.org/publications/elibrary-page/?id=9951

@inproceedings{Subramaniam2001_9951,
  author = {Subramaniam, Girish and Rao, Raghunath},
  title = {{Optimized DSP Implementation of Non-Linear Quantization}},
  booktitle = {AES Convention 110},
  note = {Paper 5363},
  year = {2001},
  month = may,
  publisher = {Audio Engineering Society},
  url = {https://aes.org/publications/elibrary-page/?id=9951}
}

TY  - CPAPER
TI  - Optimized DSP Implementation of Non-Linear Quantization
AU  - Subramaniam, Girish
AU  - Rao, Raghunath
T2  - AES Convention 110
M1  - Paper 5363
PY  - 2001
DA  - 2001/05/06
UR  - https://aes.org/publications/elibrary-page/?id=9951
PB  - Audio Engineering Society
LA  - en
AB  - Non-linear quantization of the type INT(x^(M/N) + constant) is commonly used in audio compression techniques, particularly MPEG-1 and MPEG-2 layer III (MP3) and MPEG Advanced Audio Coding (AAC). Finding a suitable DSP implementation is a problem since lookup table methods are prohibitive due to excessive storage requirements, conventional series approximation methods do not give sufficient precision, and not all processors have log/exp assist functions. This paper describes a method which utilizes the property of geometric periodicity of the x^(M/N) function to first normalize the problem to a small range of input x. Subsequently one can choose to perform the x^(M/N) in this limited range based on lookup, interpolation, or series expansion, and finally re-normalize the output to obtain the overall answer. Using a hybrid scheme based on lookup and interpolation very good overall precision is achieved. Compared to direct application of any of the above techniques, there is very little additional computational burden, and the improvement in precision is very significant. Mathematically, this method is shown to be a special case of log-exp based computation where the log is quantized.
ER  -