The role of manual gestures in second language comprehension: A simultaneous interpreting experiment

Manual gestures and speech form a single integrated system during nativelanguage comprehension. However, it remains unclear whether this hold forsecond language (L2) comprehension, more specifically for simultaneousinterpreting (SI), which involves comprehension in one language and simultaneousproduction in another. In a combined mismatch and priming paradigm,we presented Swedish speakers fluent in L2 English with multimodal stimuli inwhich speech was congruent or incongruent with a gesture. A picture prime wasdisplayed before the stimuli. Participants had to decide whether the video wasrelated to the prime, focusing either on the auditory or the visual information.Participants performed the task either during passive viewing or during SI intotheir L1 Swedish (order counterbalanced). Incongruent stimuli yielded longerreaction times than congruent stimuli, during both viewing and interpreting.Visual and audio targets were processed equally easily in both activities. However,in both activities incongruent speech was more disruptive for gesture processingthan incongruent gesture was for speech processing. Thus, the data only partlysupports the expected mutual and obligatory interaction of gesture and speechin L2 comprehension. Interestingly, there were no differences between activitiessuggesting that the language comprehension component in SI shares featureswith other (L2) comprehension tasks.