Joint Neural Translation and Classification of Videos for Audio Processing

Bharitkar S, Cajica A. Joint Neural Translation and Classification of Videos for Audio Processing. In: Convention Paper. Audio Engineering Society; 2026. Paper 10261. Available from: https://aes.org/publications/elibrary-page/?id=23208

@inproceedings{Bharitkar2026_23208,
  author = {Bharitkar, Sunil and Cajica, Alejandro},
  title = {{Joint Neural Translation and Classification of Videos for Audio Processing}},
  note = {Paper 10261},
  year = {2026},
  month = may,
  publisher = {Audio Engineering Society},
  url = {https://aes.org/publications/elibrary-page/?id=23208}
}

TY  - CPAPER
TI  - Joint Neural Translation and Classification of Videos for Audio Processing
AU  - Bharitkar, Sunil
AU  - Cajica, Alejandro
M1  - Paper 10261
PY  - 2026
DA  - 2026/05/28
UR  - https://aes.org/publications/elibrary-page/?id=23208
PB  - Audio Engineering Society
LA  - en
AB  - A low-parameter-count machine-learning model for classifying streaming video can enable content-aware audio/video processing on consumer edge devices with latency, computational, and battery constraints. In this paper, we propose a low-compute classification technique that uses only text metadata from the streaming file header, enabling near-instantaneous inference without decoding and analyzing audio or video signals as is traditionally done. In particular, to support multilingual platforms such as YouTube, we first apply neural machine translation as a pre-processing step for the text metadata and optimize a lightweight neural classifier for a three-class audio-centric classification taxonomy (movie, music, dialog/other). Experiments on a mixed-language YouTube dataset achieve 90% classification accuracy on a test set using a combined translation and a classification model (with only 22K parameters), demonstrating a globally-scalable approach for robust classification on the edge.
ER  -