Improved Singing Voice Conversion with Frame-Level Content and Melody-Informed Speaker Embeddings Using Cross-Attention

Yeh JW, Liu EM, Liu YW. Improved Singing Voice Conversion with Frame-Level Content and Melody-Informed Speaker Embeddings Using Cross-Attention. In: 2025 AES International Conference on Artificial Intelligence and Machine Learning for Audio. Audio Engineering Society; 2025. Paper 23. Available from: https://aes.org/publications/elibrary-page/?id=23012

@inproceedings{Yeh2025_23012,
  author = {Yeh, Jih-Wei and Liu, Elaine M. and Liu, Yi-Wen},
  title = {{Improved Singing Voice Conversion with Frame-Level Content and Melody-Informed Speaker Embeddings Using Cross-Attention}},
  booktitle = {2025 AES International Conference on Artificial Intelligence and Machine Learning for Audio},
  note = {Paper 23},
  year = {2025},
  month = sep,
  publisher = {Audio Engineering Society},
  url = {https://aes.org/publications/elibrary-page/?id=23012}
}

TY  - CPAPER
TI  - Improved Singing Voice Conversion with Frame-Level Content and Melody-Informed Speaker Embeddings Using Cross-Attention
AU  - Yeh, Jih-Wei
AU  - Liu, Elaine M.
AU  - Liu, Yi-Wen
T2  - 2025 AES International Conference on Artificial Intelligence and Machine Learning for Audio
M1  - Paper 23
PY  - 2025
DA  - 2025/09/02
UR  - https://aes.org/publications/elibrary-page/?id=23012
PB  - Audio Engineering Society
LA  - en
AB  - Singing voice conversion (SVC) converts audio sung by a source voice into the timbre of a target voice without changing the linguistic content or melody of the sung utterance. The popular SVC paradigm is to extract content features from the source and condition on timbre features extracted from the target to generate converted audio. In recent years, great strides have been made in SVC with advancements in content representations. However, what constitutes an accurate representation of timbre is still an emerging topic. Currently, most SVC systems use a single speaker embedding to represent one singer, but this disregards the large variations in timbre characteristics even for the same person when singing in different techniques and vocal ranges. In this paper, we propose a cross-attention mechanism to generate frame-wise speaker timbre embeddings informed by linguistic content and melody; we aim to generate timbre features that more accurately reflect how the voice changes with different vocal ranges and enunciations at different times. Objective metrics and subjective listening tests of our converted singing audio show improvements in enunciation clarity and voice similarity, while expressiveness is a more elusive challenge.
ER  -