Multi-Stage Real-Time Music Source Separation Using Random and Aligned Mixing with individualized GAN Loss

Chen Y, Feng J, Ma G. Multi-Stage Real-Time Music Source Separation Using Random and Aligned Mixing with individualized GAN Loss. In: 2026 AES 6th International Automotive Audio Conference. Audio Engineering Society; 2026. Paper 20. Available from: https://aes.org/publications/elibrary-page/?id=23413

@inproceedings{Chen2026_23413,
  author = {Chen, Yangzhen and Feng, Jianyuan and Ma, Guilin},
  title = {{Multi-Stage Real-Time Music Source Separation Using Random and Aligned Mixing with individualized GAN Loss}},
  booktitle = {2026 AES 6th International Automotive Audio Conference},
  note = {Paper 20},
  year = {2026},
  month = jul,
  publisher = {Audio Engineering Society},
  url = {https://aes.org/publications/elibrary-page/?id=23413}
}

TY  - CPAPER
TI  - Multi-Stage Real-Time Music Source Separation Using Random and Aligned Mixing with individualized GAN Loss
AU  - Chen, Yangzhen
AU  - Feng, Jianyuan
AU  - Ma, Guilin
T2  - 2026 AES 6th International Automotive Audio Conference
M1  - Paper 20
PY  - 2026
DA  - 2026/07/29
UR  - https://aes.org/publications/elibrary-page/?id=23413
PB  - Audio Engineering Society
LA  - en
AB  - As intelligent cockpits develop, music source separation (MSS) is increasingly used in automotive audio to address complex sound mixtures failing to meet users diverse needs. In karaoke, it separates vocals and accompaniment for humming and enables independent male/female vocal volume adjustment for duets. For in-vehicle audio up-mixing, extracted stems reconfigure stereo mixes into cockpit-optimized multichannel layouts. For real-time rendering, it is required to enhance specific tracks to adapt to cabin noise and user preferences. However, audio sources other than vocal, bass and drums lack research for real-time automotive applications. This paper proposes targeted optimizations for data augmentation and model structure: for target tracks (guitar, piano, male/female lead/backing vocals), a parallel single-track model is used for piano/guitar separation, and a two-stage model for male/female voice separation (first separating general vocals, then splitting into lead and backing vocals). A "Random Mixing" and "Aligned Mixing" combined method adapts to harmonic overlap in real songs. In terms of loss function, besides time-domain L1-loss and Multi-scale STFT loss, a GAN-based training procedure with individualized discriminators for each instrument stem improves audio quality and separation accuracy. Training uses a dataset from MedleyDB, MoiseDB and 3,000 private songs. To enhance the real-time causal models time-dimension receptive field, this paper utilized a modified version of SCNet with dilated convolutions and a source based band split setting which is critical for separating instruments with reverberant components. As a result, the real-time model achieves 64 ms latency with only 4.36M parameters and Signal-to-Distortion Ratio (SDR) of 7.25 dB (piano), 6.95 dB (guitar), 6.88 dB (male vocals), and 6.82 dB (female vocals) .
ER  -