📖 Table of Contents
Abstract
Aesthetics serve as an implicit and important criterion in song generation tasks that reflect human perception beyond objective metrics. However, evaluating the aesthetics of generated songs remains a fundamental challenge, as the appreciation of music is highly subjective. Existing evaluation metrics, such as embedding-based distances, are limited in reflecting the subjective and perceptual aspects that define musical appeal. To address this issue, we introduce SongEval, the first open-source, large-scale benchmark dataset for evaluating the aesthetics of full-length songs. SongEval includes over 2,399 songs in full length, summing up to more than 140 hours, with aesthetic ratings from 16 professional annotators with musical backgrounds. Each song is evaluated across five key dimensions: overall coherence, memorability, naturalness of vocal breathing and phrasing, clarity of song structure, and overall musicality. The dataset covers both English and Chinese songs, spanning nine mainstream genres. Moreover, to assess the effectiveness of song aesthetic evaluation, we conduct experiments using SongEval to predict aesthetic scores and demonstrate better performance than existing objective evaluation metrics in predicting human-perceived musical quality.
SongEval
Figure 1. Aesthetic evaluation dimensions and structural components of a song.
Figure 2. The data collection pipeline of SongEval.
Audio Samples
🎵 Generated Songs with Higher Aesthetic Scores
English Song 1
English Song 2
English Song 3
Chinese Song 1
Chinese Song 2
Chinese Song 3
🎵 Generated Songs with Lower Aesthetic Scores over Each Dimension
Overall Coherence
Memorability
Naturalness of Vocal Breathing and Phrasing
Clarity of Song Structure
Overall Musicality
Supplementary Material
A. Limitations and Future Work
Dimension Interdependence. One key limitation lies in the potential correlation and overlap among the five defined aesthetic dimensions. For instance, dimensions such as overall coherence and structural clarity, or musicality and memorability, may exhibit partial interdependence in practice. This is partly due to the holistic nature of song perception, where multiple musical aspects often influence a listener’s judgment simultaneously. Despite our efforts to provide clear annotation guidelines and expert training to distinguish these dimensions, subjective perception inherently involves cognitive and emotional entanglement, making absolute separation between factors challenging. We view this limitation as reflecting a psychologically grounded characteristic of how listeners experience songs, rather than as a flaw unique to the dataset.
Score Distribution Skew and Limited High-End Discriminability. Another limitation is that the score distribution in SongEval is skewed toward higher ratings, with a substantial proportion of annotations concentrated in the 4-5 range across most dimensions. This partly reflects the practical observation that current song generation systems often produce outputs without severe defects, rather than annotation bias alone. Nevertheless, such imbalance limits the benchmark’s ability to distinguish strong samples from truly top-tier ones. This issue is also consistent with our observations from the ICASSP 2026 Automatic Song Aesthetics Evaluation (ASAE) Challenge, where participating systems generally achieved strong rank-correlation performance but remained limited in top-tier discrimination. Future versions of the dataset may therefore benefit from broader quality coverage and clearer score anchoring at the upper end of the scale.
Inter-Annotator Agreement. Although each song in SongEval was independently rated by four expert annotators and multiple quality control measures were applied, the main paper did not report formal inter-annotator agreement statistics. We provide them here for transparency: Krippendorff’s alpha values are 0.767, 0.703, 0.670, 0.746, and 0.769 for Coherence, Memorability, Naturalness, Clarity, and Musicality, respectively. These results indicate moderate-to-substantial agreement for a subjective aesthetic evaluation task, while also suggesting that some dimensions remain more perceptually ambiguous than others. In particular, Naturalness yields the lowest agreement, which is consistent with findings from the ICASSP 2026 ASAE Challenge, where Naturalness also proved to be the most difficult dimension to predict reliably.
B. Characteristic details of generated songs
To ensure consistency across samples from different music generation systems, we standardized and documented the audio format for all generated songs. The table below summarizes the sampling rate and channel configuration used by each system. YUE produces mono-channel audio; to maintain uniformity for downstream processing and model training, we duplicated the mono channel to simulate a stereo channel signal.
| Property | Suno | Udio | Mureka | YUE | DiffRhythm |
|---|---|---|---|---|---|
| Sampling Rate | 48000 | 48000 | 44100 | 44100 | 44100 |
| Channel | 2 | 2 | 2 | 1 | 2 |
C. Duration distribution of generated songs
Figure 3. Duration distribution across different languages and generation models.
As shown in Figure 3, the generated songs exhibit distinct duration distributions depending on both the language and the generation model used. A noticeable phenomenon is that the songs generated by DiffRhythm have a fixed duration of exactly 285 seconds, which leads to a sharp concentration of songs around the four-minute mark in the overall distribution. This characteristic is an artifact of the DiffRhythm model's specific generation pipeline and constraint settings.
D. Filter process of generated songs
E. Definitions of each point on the 1-5 scale
We define a framework that focuses on five key dimensions, each assessed on a 1-5 point scale, where higher scores indicate greater proficiency and adherence to established musical principles. These dimensions aim to provide a detailed and objective measure of a piece's overall quality, from its structural integrity to its emotional impact and memorability. The scoring criteria for each dimension are defined below, offering clear benchmarks for evaluators.
- Coherence: This dimension assesses the interconnectedness and emotional unity across various musical sections (e.g., introduction, verse, chorus, bridge, outro) within a piece. It evaluates the natural flow and dynamic integration between these elements.
- Memorability: This dimension evaluates whether the piece contains elements that are easily remembered or recognized after a single listening, such as distinctive melodic hooks, rhythmic patterns, or unique instrumental motifs.
- Naturalness: This dimension evaluates the appropriateness of breath control, articulation, and the alignment of vocal delivery with the lyrical meaning, rhythmic structure, and melodic flow of the song. Inappropriate vocal delivery can significantly disrupt the listening experience.
- Clarity: This dimension assesses the clarity and logical arrangement of the song's structural sections (e.g., intro, verse, chorus, bridge, outro). It also considers whether the arrangement follows traditional song structures or presents a clear and effective innovative design.
- Musicality: This dimension assesses the overall sensory comfort and pleasantness of the musical experience, considering the mix, balance, timbre, arrangement, vocal performance, and the synergy between different musical elements (e.g., human voice and instrumental accompaniment).
F. Details of annotation process
To ensure reliable and standardized subjective annotation, we employed a web-based annotation platform that integrates both audio playback and visual spectrogram display. Annotators were asked to listen to the full song before assigning ratings across five aesthetic dimensions using intuitive sliders, each ranging from 1 (very poor) to 5 (excellent). The interface was designed for clarity and efficiency, facilitating streamlined submission and navigation between songs.
Figure 4. Screenshot of subjective annotation interface used for evaluating musical aesthetics.
To guarantee high-quality and unbiased annotations, we collaborated with an independent third-party team specializing in audio annotation. This team was responsible for managing the annotation workflow, verifying annotator qualifications, and monitoring consistency throughout the process. Annotators were selected based on their musical background or relevant auditory experience, and were given detailed training on the five aesthetic criteria.
Each annotator was compensated at a rate of $5 USD per song, calibrated to reflect the average song duration (2–6 minutes) and required attention. In total, annotations were collected for 2,399 songs, with the complete annotation process managed and quality-controlled by the third-party team. On the generation side, to build a musically diverse and high-quality dataset, we accessed three commercial song generation systems—Udio, Suno, and Mureka—through official APIs or premium memberships. These services required monthly subscriptions or credit-based payments, averaging $30 USD per system. The total cost for song generation and access rights amounted to approximately $48,000 USD, including necessary premium plans for exporting full-length tracks.