Synchronous reading: learning French orthography by audiovisual training

Gérard Bailly (1) , William-Seamus Barbour (1)

1 GIPSA-MAGIC - GIPSA - Machines parlantes, Gestes oro-faciaux, Interaction Face-à-face, Communication augmentée

We assess here the potential benefit of a karaoke-style reading system for learning sound-to-letter mapping in irregular languages. We have developed a framework that eases the development of interactive systems exploiting the alignment of text with audio at various levels (letters, phones, syllables, words, chunks, etc). Synchronous reading consists of using time-aligned text with speech at the phone level to displace a cursor - here a virtual finger - on the text in synchrony with its verbalization. We demonstrate here that this bimodal reading implicitly facilitates the learning of the correspondence between sounds and letters in French for native and foreign subjects. Native subjects are shown to benefit more strongly from synchronous reading.

Domaines: Traitement du signal et de l'image [eess.SP]; Traitement du signal et de l'image [eess.SP]

Gérard Bailly, William-Seamus Barbour. Synchronous reading: learning French orthography by audiovisual training. Interspeech 2011 - 12th Annual Conference of the International Speech Communication Association, Aug 2011, Florence, Italy. pp.1153-1156. ⟨hal-00618780⟩