Multimodal-first or pantomime-first? Communicating events through pantomime with and without vocalization

Research output: Contribution to journalArticle

Standard

Multimodal-first or pantomime-first? Communicating events through pantomime with and without vocalization. / Zlatev, Jordan; Wacewicz, Sławomir; Zywiczynski, Przemyslaw; van de Weijer, Joost.

In: Interaction Studies, Vol. 18, No. 3, 2017, p. 465-488.

Research output: Contribution to journalArticle

Harvard

APA

CBE

MLA

Vancouver

Author

RIS

TY - JOUR

T1 - Multimodal-first or pantomime-first?

T2 - Communicating events through pantomime with and without vocalization

AU - Zlatev, Jordan

AU - Wacewicz, Sławomir

AU - Zywiczynski, Przemyslaw

AU - van de Weijer, Joost

PY - 2017

Y1 - 2017

N2 - A persistent controversy in language evolution research has been whether language emerged in the gestural-visual or in the vocal-auditory modality. A "dialectic" solution to this age-old debate has now been gaining ground: language was fully multimodal from the start, and remains so to this day. In this paper, we show this solution to be too simplistic and outline a more specific theoretical proposal, which we designate as pantomime-first. To decide between the multimodal-first and pantomime-first alternatives, we review several lines of interdisciplinary evidence and complement it with a cognitive-semiotic experiment. In the study, the participants saw – and then matched to hand-drawn images – recordings of short transitive events enacted by 4 actors in two conditions: visual (only body movement), and multimodal (body movement accompanied by nonlinguistic vocalization). Significantly, the matching accuracy was greater in the visual than the multimodal condition, though a follow-up experiment revealed that the emotional pro les of the events enacted in the multimodal condition could be reliably detected from the sound alone. We see these results as supporting the proposed pantomime-first scenario.

AB - A persistent controversy in language evolution research has been whether language emerged in the gestural-visual or in the vocal-auditory modality. A "dialectic" solution to this age-old debate has now been gaining ground: language was fully multimodal from the start, and remains so to this day. In this paper, we show this solution to be too simplistic and outline a more specific theoretical proposal, which we designate as pantomime-first. To decide between the multimodal-first and pantomime-first alternatives, we review several lines of interdisciplinary evidence and complement it with a cognitive-semiotic experiment. In the study, the participants saw – and then matched to hand-drawn images – recordings of short transitive events enacted by 4 actors in two conditions: visual (only body movement), and multimodal (body movement accompanied by nonlinguistic vocalization). Significantly, the matching accuracy was greater in the visual than the multimodal condition, though a follow-up experiment revealed that the emotional pro les of the events enacted in the multimodal condition could be reliably detected from the sound alone. We see these results as supporting the proposed pantomime-first scenario.

KW - bodily mimesis

KW - pantomime

KW - multimodal origins

KW - multimodality

KW - language origins

KW - language evolution

KW - gesture

M3 - Article

VL - 18

SP - 465

EP - 488

JO - Interaction Studies: Social Behaviour and Communication in Biological and Artificial Systems

JF - Interaction Studies: Social Behaviour and Communication in Biological and Artificial Systems

SN - 1572-0373

IS - 3

ER -