Challenge and Scenario Description
Driven by the rapid advancement of Large Language Models (LLMs) and Multimodal LLMs, AI-powered smart glasses are emerging as a next-generation platform for human-computer interaction. Equipped with microphone arrays and cameras, smart glasses naturally capture the wearer’s egocentric (first-person) perspective, enabling hands-free multimodal communication throughout daily life.
However, deploying robust speech-centric interaction systems on smart glasses introduces distinct challenges compared with traditional stationary devices such as smart speakers or handheld devices such as smartphones. Smart glasses operate in highly dynamic acoustic environments, including environmental noise, user-generated motion noise, and speech from surrounding people.
To address these challenges, the SmartGlasses Challenge introduces a new benchmark for evaluating Automatic Speech Recognition (ASR) and Spoken Language Understanding (SLU) in real-world egocentric interaction scenarios, including human–machine dialogue, dyadic conversation, and multi-party meetings.
Stay updated with the latest announcements from the organizers
Here are the latest updates and news from the ESmartGlasses Challenge organizers:
Details about the Tracks to be completed
Scenario: Daily human–AI glasses dialogue recorded on-device, covering both single-turn commands and multi-turn conversational interactions grounded in audio and visual context.
ASR Task: Evaluate speech recognition robustness under diverse real-world environments including noisy streets, cycling, and whisper speech. The evaluation metric is Word Error Rate (WER).
SLU Task: Evaluate the system’s ability to interpret the user’s intent from spoken commands. In multi-turn interactions, the system must leverage dialogue history to resolve contextual dependencies.
Scenario: Face-to-face two-person conversations in everyday settings, involving overlapping speech, background interference, topic shifts, and complex semantic structures.
TSA-ASR Task: Evaluate speaker-attributed transcription with time alignment in overlapping speech scenarios. The metric is time-constrained minimum permutation WER (tcpWER).
SLU Task: Evaluate the system’s ability to capture factual details, track logical flow, and understand relationships between speakers within dyadic dialogues.
Scenario: Multi-speaker meetings with varying numbers of participants, frequent turn-taking, long conversational contexts, and domain-specific vocabulary.
TSA-ASR Task: Evaluate multi-speaker speech recognition with speaker diarization and temporal alignment in highly overlapped environments. The metric is tcpWER.
SLU Task: Evaluate the system’s ability to understand complex meeting discussions, extract key information, and summarize speaker-wise viewpoints from long-form conversations.
Explore the datasets and evaluation metrics for the challenge
The SmartGlasses dataset comprises about 200 hours of 4-channel audio-visual recordings, entirely recorded in real-life scenarios using commercial-grade AI glasses. To ensure the comprehensiveness and fairness of the evaluation, the dataset covers diverse acoustic environments, varying speech volumes, rich user profiles, and introduces highly challenging multispeaker conversation scenarios.
| Track | Scene | Data Modality | Audio Channels | Total Duration | Tasks |
|---|---|---|---|---|---|
| Track 1 | Human–Machine Command Dialogue | Audio & Video | 4 | TBD | ASR & SLU |
| Track 2 | Dyadic Dialogue Understanding | Audio | 4 | TBD | TSA-ASR & SLU |
| Track 3 | Multi-Party Meeting Understanding | Audio | 4 | TBD | TSA-ASR & SLU |
Understand how your results will be evaluated
ASR Evaluation: Track 1 adopts the standard Word Error Rate (WER). Tracks 2 and 3 involve multi-speaker scenarios and adopt Time-Stamped Speaker-Attributed ASR (TSA-ASR). The evaluation metric is time-constrained minimum permutation WER (tcpWER).
SLU Evaluation: All tracks evaluate semantic understanding using objective Question Answering (QA) accuracy based on multiple-choice questions constructed from the dialogue context.
Follow the steps below to complete your registration for SmartGlasses
Teams wishing to participate in the challenge should register via the provided Registration Form (TBD) . Please submit the following details for each participant:
After successful registration, teams will be provided with access to the 200-hour SmartGlasses dataset, including training and validation sets.
If you have any questions, please contact the organizers.
The tentative timeline for running the challenge
The tentative timeline for running the challenge is as follows (TBD):
Rankings will be available once the challenge results are released.
The SmartGlasses dataset is available for academic research. The following conditions require your compliance:
All rights not expressly granted to you are reserved by the organizers of this challenge.
