A multimodal and naturalistic approach to assessment in aphasia
OA Version
Citation
Abstract
Traditional aphasia assessment relies on static, decontextualized stimuli that fail to capture the dynamic, multimodal nature of real-world communication. This dissertation presents three complementary studies employing naturalistic movie-viewing paradigms to examine how aphasia affects the integrated systems of language, emotion, and visual attention.Study 1 examined real-time emotional reactivity in 57 persons with aphasia (PWA) and 43 healthy controls (HC) during movie viewing. PWA demonstrated reduced typicality and temporal complexity of emotional valence ratings and altered discrete emotion judgments. Principal component analysis revealed a primary dimension integrating linguistic accuracy and emotional reactivity that distinguished groups (p<.001) and correlated with both aphasia severity (r=0.644, p<.001) and depression scores (r=-0.305, p=.041). Machine learning classification achieved 86% accuracy, with both language and emotion features contributing meaningfully. These findings support psychological constructionist theories proposing that language provides essential scaffolding for emotional experience.
Study 2 investigated visual attention deployment in 19 PWA, 18 individuals with mild cognitive impairment (MCI), and 49 HC using eye-tracking during naturalistic viewing across four audiovisual conditions (minimal, visual-only, auditory-only, multimodal). PWA exhibited markedly heterogeneous, condition-invariant viewing strategies reflected in consistently low within-group gaze synchronization (ISC range: 0.030-0.046), contrasting sharply with HC who showed flexible condition-dependent modulation (ISC range: 0.039-0.172). MCI demonstrated intermediate synchrony (ISC range: 0.062-0.154) without significant modulation. Critically, all groups showed, to some extent, preserved automatic spatial orienting, indicating that aphasia disrupts controlled temporal coordination while potentially sparing bottom-up attention mechanisms.
Study 3 developed and preliminarily validated an automated multimodal main concept analysis pipeline using large language models (Gemini 2.5 Pro) to generate evidence-grounded narrative inventories from video stimuli. In 54 PWA and 51 HC, main concept distance to centroid, which quantifies the semantic deviation from a target narrative structure, showed the largest discriminative effect (Cohen's d=1.46) and strongest correlation with aphasia severity (r=-0.78, p<.001), substantially exceeding traditional discourse metrics. LASSO classification achieved 89.5% accuracy (AUC=0.917, sensitivity=85.2%, specificity=94.1%), demonstrating that automated macrolinguistic analysis of naturalistic stimuli provides scalable, clinically valid assessment.
These convergent findings advance our theoretical understanding by revealing that communication impairments in aphasia encompass disrupted multimodal integration. Clinically, naturalistic paradigms offer superior ecological validity compared to traditional batteries, enabling assessment of functional communication abilities while maintaining psychometric rigor. Finally, the automated computational approaches developed here address longstanding barriers to discourse assessment, potentially transforming clinical practice by enabling scalable, patient-centered evaluation using culturally relevant, emotionally engaging materials.
Description
2026
License
Attribution 4.0 International