Binding and unbinding the auditory and visual streams in the McGurk effect

Olha Nahorna (1) , Frédéric Berthommier (1) , Jean-Luc Schwartz (1)

1 GIPSA-PCMD - GIPSA - Perception, Contrôle, Multimodalité et Dynamiques de la parole

Subjects presented with coherent auditory and visual streams generally fuse them into a single per- cept. This results in enhanced intelligibility in noise, or in visual modification of the auditory per- cept in the McGurk effect. It is classically considered that processing is done independently in the auditory and visual systems before interaction occurs at a certain representational stage, resulting in an integrated percept. However, some behavioral and neurophysiological data suggest the existence of a two-stage process. A first stage would involve binding together the appropriate pieces of audio and video information before fusion per se in a second stage. Then it should be possible to design experiments leading to unbinding . It is shown here that if a given McGurk stimulus is preceded by an incoherent audiovisual context, the amount of McGurk effect is largely reduced. Various kinds of incoherent contexts (acoustic syllables dubbed on video sentences or phonetic or temporal modi- fications of the acoustic content of a regular sequence of audiovisual syllables) can significantly reduce the McGurk effect even when they are short (less than 4s). The data are interpreted in the framework of a two-stage "binding and fusion" model for audiovisual speech perception.

Mots clés: binding; speech perception; McGurk effect; audiovisual

Domaines: Psychologie

Olha Nahorna, Frédéric Berthommier, Jean-Luc Schwartz. Binding and unbinding the auditory and visual streams in the McGurk effect. Journal of the Acoustical Society of America, 2012, 132 (2), pp.1061-1077. ⟨10.1121/1.4728187⟩. ⟨hal-00968408⟩