Multimodal språkbehandling i L2-sammanhang

My research focuses on the modulation of speech processing in the L2 by visual information from the speaker's mouth and lips. Research on how this effect is modulated by different linguistic and individual factors within and across different levels of processing is scarce, especially concerning a systematic treatment of variables such as visual salience, in combination with the distance of L1 and L2 sound inventories, and individual listener variables.

Thus far, my work has focused on operationalizing the notion of visual salience. Fine-grained statements on this variable would allow for vowel contrasts to be grouped into high- and low-salient on continuous scales. This was investigated with a computer vision approach to extract visual parameters of vowels in Swedish from the addressee’s perspective. The measured parameters will be used as reference material to define visual salience in the forthcoming perception experiments.

The planned experiments are centered around vowel perception in Swedish learners of German and German learners of Swedish. These groups offer a particularly interesting scenario since German learners of Swedish encounter an additional vowel phoneme, whereas Swedish learners of German must adjust to a vowel system with one less vowel category compared to their L1. The experiments target three different levels of speech processing, namely discrimination, classification, and identification of vowels in three conditions: audio-only, visual-only, and audiovisual. More specifically, the experiments probe what auditory and visual features L2 listeners use for discriminating, classifying, and identifying vowels in the auditory and visual modality; whether audiovisual speech input facilitates their processing of L2 vowels; and if so, which linguistic and individual listener-related factors modulate this benefit. This will be addressed by examining the effect of acoustic and visual salience within the L2 vowel system and the L2 vowels’ distance to the L1 vowels. Moreover, cognitive measurements of general working memory, phonological and visuospatial short-term memory will be included, and potential effects of proficiency-related variables will be examined.