Audio-visual speech scene analysis: Characterization of the dynamics of unbinding and rebinding the McGurk effect

Olha Nahorna 1 Frédéric Berthommier 1 Jean-Luc Schwartz 1
1 GIPSA-PCMD - PCMD
GIPSA-DPC - Département Parole et Cognition
Abstract : While audiovisual interactions in speech perception have long been considered as automatic, recentdata suggest that this is not the case. In a previous study, Nahorna et al. [(2012). J. Acoust. Soc.Am. 132, 1061–1077] showed that the McGurk effect is reduced by a previous incoherentaudiovisual context. This was interpreted as showing the existence of an audiovisual binding stagecontrolling the fusion process. Incoherence would produce unbinding and decrease the weight ofthe visual input in fusion. The present paper explores the audiovisual binding system to characterizeits dynamics. A first experiment assesses the dynamics of unbinding, and shows that it is rapid: Anincoherent context less than 0.5 s long (typically one syllable) suffices to produce a maximalreduction in the McGurk effect. A second experiment tests the rebinding process, by presenting ashort period of either coherent material or silence after the incoherent unbinding context.Coherence provides rebinding, with a recovery of the McGurk effect, while silence providesno rebinding and hence freezes the unbinding process. These experiments are interpreted in theframework of an audiovisual speech scene analysis process assessing the perceptual organization ofan audiovisual speech input before decision takes place at a higher processing stage.
Complete list of metadatas

Cited literature [46 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-01213897
Contributor : Jean-Luc Schwartz <>
Submitted on : Friday, October 9, 2015 - 5:03:03 PM
Last modification on : Friday, October 12, 2018 - 1:16:22 AM
Long-term archiving on : Sunday, January 10, 2016 - 10:23:53 AM

File

Nahorna_JASA_Binding2_second_r...
Files produced by the author(s)

Identifiers

Collections

Citation

Olha Nahorna, Frédéric Berthommier, Jean-Luc Schwartz. Audio-visual speech scene analysis: Characterization of the dynamics of unbinding and rebinding the McGurk effect. Journal of the Acoustical Society of America, Acoustical Society of America, 2015, 137 (1), pp.362-377. ⟨10.1121/1.4904536⟩. ⟨hal-01213897⟩

Share

Metrics

Record views

2172

Files downloads

252