12.05.2026 Asli Özyürek
Adaptive and Evolving Multimodal Language in Humans and Machines
Most traditional theories, as well as descriptive and empirical studies of human language, have primarily focused on its spoken and textual forms. This emphasis has led to ongoing debates about the role of visual modalities in language—such as co-speech gestures and sign languages used in deaf communities—and whether these should be considered precursors to language evolution.
In this talk, I introduce the Multimodal Language Framework (MLF), which conceptualizes multimodality as a fundamental design feature of human language. MLF proposes that linguistic theory must incorporate modality-specific properties of both manual and non-manual bodily communicative movements—such as iconicity, simultaneity, and visible indexicality—and integrate these into structural accounts of spoken and sign languages alike- to account for its adaptive and flexible nature.I will present evidence for this from cross-linguistic studies of multimodal language in both adults and children, along with findings from neural, cognitive, and processing research involving both typical and atypical populations
In this talk, I introduce the Multimodal Language Framework (MLF), which conceptualizes multimodality as a fundamental design feature of human language. MLF proposes that linguistic theory must incorporate modality-specific properties of both manual and non-manual bodily communicative movements—such as iconicity, simultaneity, and visible indexicality—and integrate these into structural accounts of spoken and sign languages alike- to account for its adaptive and flexible nature.I will present evidence for this from cross-linguistic studies of multimodal language in both adults and children, along with findings from neural, cognitive, and processing research involving both typical and atypical populations
I will argue then that within the MLF, it becomes more appropriate to ask how and why language evolved as a multimodal system, rather than assuming a unidirectional shift from visual to spoken-textual forms. Finally, I will touch upon recent developments in artificial generative systems, highlighting their transition from text-based large language models (LLMs) to multimodal models (VLLMs). I suggest that, if artificial systems are to effectively mimic human language and interact naturally with human users, they must also adopt a fundamentally multimodal architecture, enabling them to be adaptive and flexible systems akin to natural human languages.