Multimodality and timing: A Study in Audio Description

The goal of this project is to systematically investigate the process of simultaneous verbal description of visual scenes in the framework of cognitive science theories of visual perception and attention, multimodality, timing, embodied cognition and mental imagery.

To perceive and experience visual scenes, images, gestures, facial expressions, events and movements on stage during film and theatre performances is something that the sighted audience takes for granted. Visually impaired audience can, however, miss important information that is only visible but not articulated through language or sound.

The task of the interpreter is to evoke vivid mental images by simultaneously describing visual scenes verbally for people without sight. That means that the interpreter under time pressure has to select relevant non-verbal information from the visual scene, linearize it, articulate it linguistically in an efficient manner to evoke such images and to time this piece of information so that it matches with what at the moment happens linguistically, ie what is said in the dialogues and what is heard in the sounds. This timing is crucial for the integration of the different modalities, for meaning–making and for achieving of the intended communicative effect.