Influence of sound on visual gaze when looking at videos
Résumé
We propose an analysis of the effect of sound on visual gaze when looking at videos. An audio-visual experiment was designed where participants were asked to look at short video excerpts, without any specific task. To test the effect of sound, fifteen participants completed the experiment in an audio-visual condition and fifteen participants in a visual condition. Sixty video excerpts (a total of about 10900 frames) lasting 5-8 seconds were selected from films that were interesting for both visual and audio. During the experiment, human eye position was tracked by an Eyetracker Eyelink II (SR Research). (i) We classified the sound in three classes: on-screen speech, non-speech and non-sound. We observed with statistical methods that the effect of sound is different depending on the class of sound. (ii) Then a comparison of the experimental data and a visual saliency model was carried out. The results showed that adding sound to video decreased the predictability of the visual saliency model without a sound pathway. (iii) Finally, we created a ''sound localization pathway'' by locating the coordinates of the sound source on each image manually. In this case, the prediction was better for participants in the audio-visual condition.