Compensation for a large gesture-speech asynchrony in instructional videos

Andrey Anikin; Jens Nirme; Sarah Alomari; Joakim Bonnevier; Magnus Haake

Compensation for a large gesture-speech asynchrony in instructional videos

Andrey Anikin, Jens Nirme, Sarah Alomari, Joakim Bonnevier, Magnus Haake

Cognitive Science

Research output: Chapter in Book/Report/Conference proceeding › Paper in conference proceeding › peer-review

Abstract

We investigated the pragmatic effects of gesture-speech lag by asking participants to reconstruct formations of geometric shapes based on instructional films in four conditions: sync, video or audio lag (±1,500 ms), audio only. All three video groups rated the task as less difficult compared to the audio-only group and performed better. The scores were slightly lower when sound preceded gestures (video lag), but not when gestures preceded sound (audio lag). Participants thus compensated for delays of 1.5 seconds in either direction, apparently without making a conscious effort. This greatly exceeds the previously reported time window for automatic multimodal integration.

Original language	English
Title of host publication	Gesture and Speech in Interaction - 4th edition (GESPIN 4)
Editors	Gaëlle Ferré, Mark Tutton
Pages	19-23
Publication status	Published - 2015
Event	Gesture and Speech in Interaction (GESPIN 4) - Nantes, France Duration: 2015 Sept 2 → 2015 Sept 4

Conference

Conference	Gesture and Speech in Interaction (GESPIN 4)
Country/Territory	France
City	Nantes
Period	2015/09/02 → 2015/09/04

Subject classification (UKÄ)

Psychology

Free keywords

gesture-speech synchronization
multimodal integration
temporal synchronization
comprehension

Access to Document

Gesture and Speech in Interaction (GESPIN 4)
Nirme, J. (Presenter)
2015 Sept 2
Activity: Participating in or organising an event › Participation in conference

Cite this

@inproceedings{048e9d684124414ba92498fd2a516077,

title = "Compensation for a large gesture-speech asynchrony in instructional videos",

abstract = "We investigated the pragmatic effects of gesture-speech lag by asking participants to reconstruct formations of geometric shapes based on instructional films in four conditions: sync, video or audio lag (±1,500 ms), audio only. All three video groups rated the task as less difficult compared to the audio-only group and performed better. The scores were slightly lower when sound preceded gestures (video lag), but not when gestures preceded sound (audio lag). Participants thus compensated for delays of 1.5 seconds in either direction, apparently without making a conscious effort. This greatly exceeds the previously reported time window for automatic multimodal integration.",

keywords = "gesture-speech synchronization, multimodal integration, temporal synchronization, comprehension",

author = "Andrey Anikin and Jens Nirme and Sarah Alomari and Joakim Bonnevier and Magnus Haake",

year = "2015",

language = "English",

pages = "19--23",

editor = "Ga{\"e}lle Ferr{\'e} and Mark Tutton",

booktitle = "Gesture and Speech in Interaction - 4th edition (GESPIN 4)",

note = "Gesture and Speech in Interaction (GESPIN 4) ; Conference date: 02-09-2015 Through 04-09-2015",

}

TY - GEN

T1 - Compensation for a large gesture-speech asynchrony in instructional videos

AU - Anikin, Andrey

AU - Nirme, Jens

AU - Alomari, Sarah

AU - Bonnevier, Joakim

AU - Haake, Magnus

PY - 2015

Y1 - 2015

N2 - We investigated the pragmatic effects of gesture-speech lag by asking participants to reconstruct formations of geometric shapes based on instructional films in four conditions: sync, video or audio lag (±1,500 ms), audio only. All three video groups rated the task as less difficult compared to the audio-only group and performed better. The scores were slightly lower when sound preceded gestures (video lag), but not when gestures preceded sound (audio lag). Participants thus compensated for delays of 1.5 seconds in either direction, apparently without making a conscious effort. This greatly exceeds the previously reported time window for automatic multimodal integration.

AB - We investigated the pragmatic effects of gesture-speech lag by asking participants to reconstruct formations of geometric shapes based on instructional films in four conditions: sync, video or audio lag (±1,500 ms), audio only. All three video groups rated the task as less difficult compared to the audio-only group and performed better. The scores were slightly lower when sound preceded gestures (video lag), but not when gestures preceded sound (audio lag). Participants thus compensated for delays of 1.5 seconds in either direction, apparently without making a conscious effort. This greatly exceeds the previously reported time window for automatic multimodal integration.

KW - gesture-speech synchronization

KW - multimodal integration

KW - temporal synchronization

KW - comprehension

M3 - Paper in conference proceeding

SP - 19

EP - 23

BT - Gesture and Speech in Interaction - 4th edition (GESPIN 4)

A2 - Ferré, Gaëlle

A2 - Tutton, Mark

T2 - Gesture and Speech in Interaction (GESPIN 4)

Y2 - 2 September 2015 through 4 September 2015

ER -

Compensation for a large gesture-speech asynchrony in instructional videos

Abstract

Conference

Subject classification (UKÄ)

Free keywords

Access to Document

Fingerprint

Activities

Gesture and Speech in Interaction (GESPIN 4)

Cite this