Easy Turn: Integrating Acoustic and Linguistic Modalities for Robust Turn-Taking in Full-Duplex Spoken Dialogue Systems

Abstract

Full-duplex interaction is crucial for natural human–machine communication, yet remains challenging as it requires robust turn-taking detection to decide when the system should speak, listen, or remain silent. Existing solutions either rely on dedicated turn-taking models, most of which are not open-sourced. The few available ones are limited by their large parameter size or by supporting only a single modality, such as acoustic or linguistic. Alternatively, some approaches finetune LLM backbones to enable full-duplex capability, but this requires large amounts of full-duplex data, which remain scarce in open-source form. To address these issues, we propose Easy Turn—an open-source, modular turn-taking detection model that integrates acoustic and linguistic bimodal information to predict four dialogue turn states: complete, incomplete, backchannel, and wait, accompanied by the release of Easy Turn trainset, a 1,145-hour speech dataset designed for training turn-taking detection models. Compared to existing open-source models like TEN Turn Detection and Smart Turn V2, our model achieves state-of-the-art turn-taking detection accuracy on our open-source Easy Turn testset. The data and model will be made publicly available on GitHub.

Demo: Interacting with OSUM-Echat with Easy-Turn runing

Easy Turn

To accelerate research in full-duplex spoken dialogue field, we introduce Easy Turn, an open-source and modular turn-taking detection model. The model accepts user's speech as input and outputs both the corresponding ASR transcription and the dialogue turn state, effectively integrating acoustic and linguistic information. The Easy Turn comprises three key components: an audio encoder, an audio adaptor, and an LLM. Its design draws inspiration from Qwen-Audio, employing Whisper as the audio encoder and Qwen2.5 as the LLM. To better integrate acoustic and linguistic modalities, we adopt an ASR+Turn-Detection paradigm, where the LLM generates ASR transcriptions and fuses them with acoustic features to sequentially predict dialogue turn state labels (complete, incomplete, backchannel or wait).

Easy Turn Trainset

The Easy Turn trainset is a large-scale speech dataset designed for training turn-taking detection models, comprising both real and synthetic data. It comprises four dialogue turn states (complete, incomplete, backchannel, wait), totaling approximately 1,145 hours. Detailed statistical data are shown in the table below. The detailed data processing pipeline is shown in the following figure. For more details, please refer to the original paper.

Data Samples

EXPERIMENTS

Main Results

We evaluate Easy Turn against two open-source turn-taking detection models, TEN Turn Detection and Smart Turn V2, using the Easy Turn testset. All experiments are conducted on a single NVIDIA RTX 4090 GPU. Notably, since TEN Turn Detection lacks direct speech support, we use Paraformer as the ASR model to transcribe speech into text and take the text as its input. The table below reports the results: ACC_cp, ACC_incp, ACC_bc and ACC_wait denote the turn-taking detection accuracy for complete, incomplete, backchannel, and wait states (higher is better). Params, Latency, and Memory represent total model size, average inference time, and GPU usage, where lower values indicate greater efficiency. The symbol “–” indicates that the corresponding model does not support detection of a particular state.

Ablation Study

We conduct ablation experiments on the Easy Turn to evaluate the contribution of individual modalities within its architecture and to assess the impact of the ASR + Turn-Detection paradigm on performance. The primary metric is ACC_avg, computed as the average detection accuracy across four dialogue turn states (complete, incomplete, backchannel, and wait). Easy Turn only-state uses the same architecture as Easy Turn but omits the ASR + Turn-Detection paradigm, directly predicting dialogue turn state labels without first generating ASR transcriptions. Finetuned Whisper + Linear refers to fine-tuning Whisper-Medium audio encoder with an additional linear classifier on our Easy Turn trainset, taking only speech as input and directly predicting dialogue turn state labels, representing the acoustic-only modality. Finetuned Qwen2.5-0.5B-Instruct is fine-tuned on text transcriptions from our Easy Turn trainset, taking only text as input and outputting dialogue turn state labels, representing the linguistic-only modality. Detailed results are shown in the table below.