Narratives and the Impact of Timing of Gestures and speech on an Addressee
The aims of this project are twofold. On one level the project aim is to shed light on the effects of the temporal co-ordination of speech and gestures on an addressee who watches and listens to somebody that tells a story. One another level the aim is to contribute to a more general research platform/infrastructure, for addressing various issues on interaction and communication.
Level one: It is well established that there is a tight semantic-pragmatic and temporal link between speech and gesture – but it remains unknown how this fine-grained temporal synchronization influences the listener/observer (addressee), and how important it is that the alignment of speech and gestures is precise.
Eg: At what interval of displacement does temporal misalignment between speech and gesture affect comprehension of a narrative, and what are those effects? Does such temporal misalignment cause semantic integration difficulties in processing as detectable in electrophysiological measurements?At what time interval of displacement does temporal misalignment affect emotional experiences? Are there individual and group differences in sensitivity for temporal misalignment?
To address these questions, the project uses virtual human techniques - developed to enable fine-grained manipulation of parameters. Virtual humans are developed on the basis of motion capture data of natural speech and gesture. This allows us to displace gesture relative to speech in controlled manners.
|Effective start/end date||2011/01/01 → 2019/12/31|
Eva Sjöstrand, Ingela Byström, Magnus Lindgren, Johan Mårtensson, Annika Andersson, Birgitta Sahlén, Kristina Hansson, Marianne Gullberg, Roger Johansson, Mikael Johansson, Andreas Falck, Mikael Roll, Freddy Ståhlberg, Merle Horne, Frida Blomberg, Victoria Johansson, Sven Strömqvist, Jens Nirme, Agneta Gulz, Joost van de Weijer, Carita Paradis, Betty Tärning, Ines Bramao, Susan Sayehli, Simone Löhndorf, Caroline Willners, Johan Blomberg, Susanne Schötz, Magnus Haake, Jonas Brännström, Emily Grenner, Peter Gärdenfors, Jana Holsanova, Åsa Wengelin, Magnus Johnsson, Stefan Winberg, Christian Balkenius, Zahra Gharaee & Rasmus Bååth
Swedish Research Council
2008/01/01 → 2018/12/31