Narratives and the Impact of Timing of Gestures and speech on an Addressee

Project: Research

Layman's description

The aims of this project are twofold. On one level the project aim is to shed light on the effects of the temporal co-ordination of speech and gestures on an addressee who watches and listens to somebody that tells a story. One another level the aim is to contribute to a more general research platform/infrastructure, for addressing various issues on interaction and communication.

The aims of this project are twofold. On one level the project aim is to shed light on the effects of the temporal co-ordination of speech and gestures on an addressee who watches and listens to somebody that tells a story. One another level the aim is to contribute to a more general research platform/infrastructure, for addressing various issues on interaction and communication.
Level one: It is well established that there is a tight semantic-pragmatic and temporal link between speech and gesture – but it remains unknown how this fine-grained temporal synchronization influences the listener/observer (addressee), and how important it is that the alignment of speech and gestures is precise.
Eg: At what interval of displacement does temporal misalignment between speech and gesture affect comprehension of a narrative, and what are those effects? Does such temporal misalignment cause semantic integration difficulties in processing as detectable in electrophysiological measurements?At what time interval of displacement does temporal misalignment affect emotional experiences? Are there individual and group differences in sensitivity for temporal misalignment?
To address these questions, the project uses virtual human techniques - developed to enable fine-grained manipulation of parameters. Virtual humans are developed on the basis of motion capture data of natural speech and gesture. This allows us to displace gesture relative to speech in controlled manners.
StatusActive
Effective start/end date2011/01/012019/12/31

Participants