Multi-modal levels of prominence: How verbal and visual signals interact in the coding of fine distinctions in information structure

Project: Research

Layman's description

This project investigates how eye-brow and head movements (“visual prosody”) interact with speech melody (“verbal prosody”) in order to highlight words in discourse. How do these two modes of human communication together produce different degrees of highlighting (prominence)? Can these be used to mark different types of information?

The melody of speech, or intonation, plays a crucial role in spoken, face-to-face communication. By altering the speech melody, speakers can highlight certain words in a discourse and draw attention to important or new information. Facial gestures, such as eyebrow and head movements, can often have the same function as intonation such as nodding on an important word to make it more prominent. In this project, we investigate how verbal and visual signals work together in conveying different degrees of highlighting. We also study how these signals can be used to code nuances of information structure. For example, in what ways do speakers highlight words differently depending on whether they can be inferred from the context or not, such as talking about a judge in a court (predictable in the context) vs. a judge in a supermarket (not so predictable)?

We are studying two types of ecologically valid speech data – news broadcasts from Swedish television and Swedish spontaneous dialogues – by analysing audio, video, and motion capture data. For the latter, reflectors were attached to selected points on the participants' bodies, who were filmed with special infrared cameras.

In addition, we will study how listeners perceive different combinations of verbal and visual cues. In these perception experiments, we will make use of computer animated agents (“talking heads”) which can be implemented in various speech technology applications, such as dialog systems and virtual tutors.
StatusActive
Effective start/end date2013/07/012017/12/31

Participants

Related projects

Johan Frid & Marianne Gullberg

2014/01/012018/12/31

Project: NetworkNational collaboration

View all (1)

Related research output

Johan Frid, Gilbert Ambrazaitis, Malin Svensson Lundmark & House, D. 2017 Sep 25 Proceedings of the 4th European and 7th Nordic Symposium on Multimodal Communication (MMSYM 2016). Paggio, P. & Navarretta, C. (eds.). Linköping University Electronic Press, Linköpings universitet, p. 4-9 6 p. 002. (Linköping Electronic Conference Proceedings)

Research output: Chapter in Book/Report/Conference proceedingPaper in conference proceeding

Gilbert Ambrazaitis & House, D. 2017 Proceedings of The 14th International Conference on Auditory-Visual Speech Processing (AVSP2017). Ouni, S., Davis, C., Jesse, A. & Beskow, J. (eds.). KTH, 6 p.

Research output: Chapter in Book/Report/Conference proceedingPaper in conference proceeding

View all (7)