M5-TTSA Mini-Scale Multilingual and Multi-Dialect Text-to-Speech Language Model with Multi-Token Prediction
ASLP @ NPU & NJU · Pimei Technology · Moonstep AI · WeNet
Affiliations & author notes
1 Audio, Speech and Language Processing Group (ASLP@NPU), School of Computer Science, Northwestern Polytechnical University
2 Shenzhen Pimei Technology Co., Ltd. 3 Moonstep AI 4 WeNet Open Source Community
5 Audio, Speech and Language Processing Group (ASLP@NJU), School of Intelligence Science and Technology, Nanjing University
Abstract
We present M5-TTS, a compact text-to-speech system supporting multilingual and Chinese multi-dialect synthesis, zero-shot voice cloning, speaker-specific multilingual generation, and CPU streaming. Its 50M-parameter language model, excluding the speech codec, combines a temporal backbone with a local multi-token prediction module over 16 codebooks. GRPO post-training normalizes speaker-similarity and pause rewards separately within each candidate group before weighting and summation, then adds CER-guided local process feedback. ASR edit counts yield group-relative character advantages; negative feedback propagates backward within a clause before projection onto codec frames. All 16 codec tokens within a frame share the resulting advantage for policy updates.
M5-TTS achieves 1.37% CER on Seed-ZH and 2.91% WER on Seed-EN, corresponding to relative error reductions of 62.2% and 35.3% over MOSS-TTS-Nano with half the language-model parameters. It also achieves lower content error rates in five of six evaluated MiniMax languages, while difficult Chinese inputs remain a limitation. On four Intel Xeon Gold 6530 cores, it achieves a real-time factor of 0.653 and a warm first-chunk latency of 225 ms. VoiceJudge, our internal model for assessing speech naturalness and expressiveness, prefers M5-TTS over MOSS-TTS-Nano in 84.89% and 86.50% of naturalness comparisons under the two A/B presentation orders. We will release inference code, together with voice-cloning and speaker-specific checkpoints, to support reproducible research on compact speech generation for on-device voice interaction.
Speech tokenizer
A low-frame-rate, causal representation for streaming speech. M5-TTS adopts Qwen-TTS-Tokenizer-12Hz, encoding each frame into one semantic token and 15 complementary acoustic tokens.
Semantic + acoustic
One semantic codebook and 15 acoustic codebooks preserve complementary speech information.
Low frame rate
12.5 frames per second keeps the temporal sequence compact for efficient generation.
Causal reconstruction
No look-ahead in the encoder or decoder, enabling incremental waveform synthesis.
Model architecture
A temporal backbone Transformer models speech across frames, while a lightweight multi-token prediction (MTP) module refines the acoustic representation within each frame. Pre-training uses accent and language IDs; speaker ID is added during speaker-specific fine-tuning.
One frame, sixteen tokens. The backbone retains its attention cache across frames; the MTP cache resets for each new frame. This separates temporal modeling from within-frame acoustic refinement without expanding all 16 codebooks into separate backbone time steps.
Chinese examples
Explore Chinese speech through everyday delivery, emotion, and prosody. Choose a collection, then a listening category to compare M5-TTS and MOSS-TTS-Nano with the same reference prompt and target text.
Everyday Chinese sentences and natural spoken phrasing.
Curated listening examples, not an unbiased benchmark. Target punctuation and reference transcripts are preserved.
Multilingual examples
Compare M5-TTS with MOSS-TTS-Nano across English, German, French, Japanese, and Korean, using the same reference voice and target text.
Dialect examples
Explore Sichuanese and Cantonese speech synthesis with M5-TTS. Each example includes a reference voice prompt, the target text, and the generated speech, with reference transcripts shown where available. Baseline audio is not provided for these dialect samples.