CBT-Audio: Evaluating Audio Language Models for Patient-Side Distress Intensity Estimation in CBT Session Recordings
Qixuan Hu, Shuchang Ye, Xumou Zhang, Anastasia Serafimovska, Anastasia Suraev, Amit Saha, Ping-hsiu Lin, Sydney Su
Abstract
Cognitive behavioural therapy is widely used to help patients understand and manage psychological distress. It is often delivered through spoken conversation, where therapists attend not only to what patients say, but also to how they say it, because these cues can help therapists decide how to respond and adapt treatment. Progress in building AI systems for CBT remains largely limited to text, partly because most available datasets are text based and shareable spoken CBT data are scarce under ethical and privacy constraints. This creates a blind spot because text based models and evaluations cannot capture the mismatch between the transcript and the patient's voice, even though therapists often rely on this mismatch to understand patient distress. We introduce CBT-Audio, a dataset for evaluating patient distress estimation from spoken CBT sessions with audio language models. CBT-Audio contains 1,802 patient turns from 96 publicly available CBT recordings, with turn-level distress labels validated on an experts-annotated subset. We evaluate 10 open source audio language models under three input conditions, where models receive only patient audio, only the transcript, or both audio and transcript. Our results show that audio can provide useful information beyond text, especially when combined with transcripts. Adding audio to transcript input improves distress estimation over using the transcript alone in 8 of 10 model families, with significant gains in 4, and case studies show the clearest benefit when verbal content and vocal delivery diverge. CBT-Audio makes spoken patient behaviour measurable for AI evaluation in CBT-related tasks and supports future work on audio language models for mental health interaction.
AI Impact Assessments
(1 model)Scientific Impact Assessment: CBT-Audio
1. Core Contribution
CBT-Audio addresses a genuine gap at the intersection of AI for mental health and audio language model evaluation. The paper's central argument is well-motivated: CBT is fundamentally a spoken interaction, yet nearly all AI-CBT research operates on text alone, missing paralinguistic cues (tone, hesitation, pace, tremor) that therapists routinely use to assess patient distress. The paper contributes: (1) a dataset of 1,802 patient turns from 96 publicly available CBT recordings with turn-level distress labels on a 1–5 scale; (2) a systematic evaluation of 10 open-source ALMs under three input conditions (audio-only, transcript-only, audio+transcript); and (3) evidence that audio provides complementary information to transcripts, with audio+transcript improving over transcript-only in 8 of 10 model families.
The key finding—that audio doesn't reliably beat text alone but consistently helps *when combined* with text—is nuanced and practically important. It reframes the value of audio not as a replacement for text but as a complementary signal, particularly when verbal content and vocal delivery diverge.
2. Methodological Rigor
Strengths in design: The controlled three-condition evaluation (AO, TO, AT) on identical patient turns is well-designed, enabling clean modality comparisons. The use of paired Wilcoxon signed-rank tests with bootstrap confidence intervals is appropriate for this ordinal prediction task.
Labeling pipeline concerns: The reference labels are generated via GPT-audio-1.5 using a semantic similarity rating (SSR) approach rather than direct human annotation. While the SSR method (generating descriptions, embedding them, matching to anchors) is creative and avoids single-model scale bias, it means the ground truth is fundamentally model-generated. The expert validation on 194 clips (81.4% within ±1 agreement) provides some reassurance, but this is a lenient criterion on a 5-point scale—chance agreement within ±1 would already be substantial. The inter-rater reliability among human experts is only moderate (Krippendorff's α = 0.439), which both validates that distress rating is subjective and raises questions about the ceiling for any evaluation system.
Data limitations: The recordings are educational role-plays and case-study walkthroughs, not real therapy sessions. The authors acknowledge this but it significantly limits ecological validity. Actors and trainees may display more stereotypical or exaggerated emotional patterns than actual patients, potentially inflating the apparent utility of audio cues. The English-only constraint further limits generalizability.
Label distribution: The comparison between SSR and direct numeric prompting (Figure 5) shows SSR produces a broader distribution, but one could argue this is an artifact rather than a virtue—the highly concentrated distribution from direct prompting might reflect genuine base rates in educational recordings where extreme distress is rare.
3. Potential Impact
Immediate utility: CBT-Audio fills a practical need for benchmarking ALMs on clinical-adjacent speech tasks. The mental health AI community has been constrained by text-only datasets, and even an imperfect audio benchmark creates new evaluation possibilities.
Broader implications: The finding that audio+transcript outperforms transcript-only supports the development of multimodal therapy support tools. This could influence how future therapy AI systems are designed—arguing for audio processing capabilities even when transcripts are available.
Clinical translation: The paper is careful not to overclaim clinical applicability, which is appropriate. However, the distance from educational role-plays to real clinical settings is substantial, and the work should be viewed as a proof-of-concept for evaluation methodology rather than evidence for clinical deployment.
Dataset contribution: The release of metadata (URLs, timestamps, labels, code) without redistributing audio is a pragmatic approach to reproducibility under copyright constraints, though it introduces fragility—YouTube videos can be removed.
4. Timeliness & Relevance
The paper is well-timed. ALMs have proliferated rapidly (the 10 evaluated models span 2023–2026), and there is active interest in applying these models to mental health applications. The paper correctly identifies that existing speech emotion recognition benchmarks (IEMOCAP, MELD, RAVDESS) use general-domain categorical emotions rather than clinical constructs like distress intensity in therapeutic contexts. The focus on open-source models is also timely given privacy concerns around sending sensitive clinical audio to commercial APIs.
5. Strengths & Limitations
Key strengths:
Notable limitations:
Missing comparisons: The paper doesn't compare against traditional speech emotion recognition systems or acoustic feature extractors (e.g., OpenSMILE + classifier), which would help contextualize whether ALMs offer advantages over simpler approaches.
Summary
CBT-Audio makes a meaningful contribution by introducing an evaluation framework for audio language models in a clinically-motivated task. The controlled experimental design is a strength, and the core finding about audio's complementary value is both believable and useful. However, the reliance on model-generated labels, educational role-play data, and the absence of traditional baselines temper the impact. This is best viewed as foundational work that opens a new evaluation direction rather than definitive evidence about ALM capabilities in clinical settings.
Generated May 19, 2026
Comparison History (22)
Paper 2 addresses a critical and highly active bottleneck in AI research: the reproducible, scalable, and faithful benchmarking of LLM-powered GUI agents. By providing a fully synthetic, backend-free framework with automated rewards, it enables rapid iteration for agent development, a universally applicable problem in tech. While Paper 1 offers a valuable dataset for a high-impact societal domain (mental health), Paper 2 provides a foundational tool for the broader and rapidly expanding field of autonomous agents, likely resulting in significantly higher adoption and citation rates.
Paper 2 is likely higher impact due to a more broadly applicable and novel inference-time framework (guided stochastic exploration) for improving recursive reasoning models without retraining, plus new label-free diagnostics that generalize across tasks. This can influence multiple areas (reasoning architectures, inference algorithms, reliability/uncertainty, evaluation) and is timely given interest in test-time compute and scalable reasoning. Paper 1 provides a valuable dataset and evaluation for a high-importance domain, but its impact is narrower (CBT distress estimation) and constrained by data size/availability and domain-specific adoption hurdles.
Paper 1 addresses a fundamental limitation in neural combinatorial optimization by unifying symmetric and asymmetric routing problems, demonstrating results across 110 VRP variants. Its methodological contribution (coordinate-free embedding, weight-decomposed decoding) has broad applicability across operations research and AI. Paper 2 introduces a valuable but niche evaluation dataset for audio-based distress estimation in CBT, with impact primarily limited to mental health NLP. Paper 1's broader technical contribution, methodological novelty, and wider applicability across optimization and AI give it higher potential impact.
Paper 1 offers a novel, general formalization of trust calibration for agentic tool use as preference learning, connecting it to Preferential Bayesian Optimization with a clear uncertainty-driven querying strategy. This has broad, timely applicability across autonomous agents, human-in-the-loop governance, safety, and deployment policy (allow/block/ask). Paper 2 provides a valuable dataset and evaluation for audio LMs in CBT, with concrete application relevance, but its impact is narrower to mental-health NLP and constrained by dataset size and privacy-driven limits on generalization. Overall, Paper 1 is likely to influence more methods and domains.
Paper 1 is likely to have higher scientific impact due to its methodological novelty and broad relevance: it extends machine unlearning to realistic multi-task shared-backbone settings, identifies concrete interference mechanisms, and proposes a generally applicable optimization framework (task-aware projection + instance-level orthogonalization) with clear quantitative gains. This advances a timely area (unlearning/privacy/compliance) with potential deployment implications across many multi-task domains. Paper 2 provides a valuable evaluation dataset and benchmark for audio-based distress estimation in CBT, but its impact is narrower (mental-health audio evaluation) and more incremental on methodology.
Paper 2 (CBT-Audio) addresses a critical gap in mental health AI by introducing a novel, ethically-grounded dataset for evaluating audio language models in CBT. It has broader real-world applicability in clinical mental health settings, fills an important data scarcity problem, and provides a reusable benchmark. Paper 1, while intellectually interesting in probing MLLM spatial reasoning and Theory of Mind, addresses a more niche problem with a complex framework whose practical applicability is less immediate. Paper 2's contribution of a shareable spoken CBT dataset and systematic evaluation across 10 models is more likely to catalyze follow-up research across NLP, clinical AI, and mental health communities.
CBT-Audio addresses a significant gap at the intersection of AI, mental health, and audio language models. It introduces a novel dataset for an underexplored modality (audio) in CBT evaluation, with clear clinical relevance. The finding that audio provides complementary information to text—especially when verbal content and vocal delivery diverge—has meaningful implications for therapeutic AI systems. WebGameBench, while well-constructed, addresses a narrower problem in coding agent evaluation with less cross-disciplinary impact. CBT-Audio's contributions to mental health AI, multimodal evaluation, and clinical applications give it broader and more lasting scientific impact.
Paper 2 likely has higher scientific impact due to broader relevance and timeliness: it introduces a rigorous, decision-grade benchmark for deep research agents used in enterprise settings, with multi-layer evaluation (deterministic verifiers + SME rubrics) and adversarial “cognitive traps,” enabling more reliable measurement across models and deployments. Its methodology and scoring framework are generalizable across domains and can shape evaluation standards for agentic systems. Paper 1 is novel and valuable for mental-health audio modeling, but its impact is narrower due to dataset size, domain specificity, and data availability constraints.
Skim addresses critical bottlenecks—cost and latency—in the rapidly growing field of AI web agents. By introducing speculative execution to bypass heavyweight components without accuracy loss, it offers broad, immediate utility for deploying practical AI agents across numerous domains. While Paper 2 provides a valuable dataset for mental health AI, Paper 1's system-level optimization has wider cross-domain applicability and directly accelerates the broader adoption of autonomous agents.
CBT-Audio addresses a significant gap at the intersection of AI and mental health by introducing a novel dataset and evaluation framework for audio-based distress estimation in therapy. It tackles real ethical/privacy challenges, enables multimodal (audio+text) evaluation of language models in clinical settings, and has broad applicability across mental health AI research. Paper 2, while technically sophisticated, addresses a narrower DevOps/microservices problem with an incremental multi-agent framework. CBT-Audio's unique dataset contribution, clinical relevance, and potential to catalyze research in audio language models for mental health give it broader and more lasting impact.