Evaluating the Acquisition of Semantic Knowledge from Cross-situational Learning in Artificial Neural Networks

Mitja Nikolaus (1, 2, 3, 4) , Abdellah Fourtassi (1, 3, 4)

1 ILCB - Institute of Language, Communication and the Brain
2 LPL - Laboratoire Parole et Langage
3 LIS - Laboratoire d'Informatique et des Systèmes (LIS) (Marseille, Toulon)
4 TALEP - Traitement Automatique du Langage Ecrit et Parlé

When learning their native language, children acquire the meanings of words and sentences from highly ambiguous input without much explicit supervision. One possible learning mechanism is cross-situational learning, which has been successfully tested in laboratory experiments with children. Here we use Artificial Neural Networks to test if this mechanism scales up to more natural language and visual scenes using a large dataset of crowd-sourced images with corresponding descriptions. We evaluate learning using a series of tasks inspired by methods commonly used in laboratory studies of language acquisition. We show that the model acquires rich semantic knowledge both at the word-and sentence-level, mirroring the patterns and trajectory of learning in early childhood. Our work highlights the usefulness of low-level cooccurrence statistics across modalities in facilitating the early acquisition of higher-level semantic knowledge.

Domaines: Sciences cognitives; Psychologie; Informatique et langage [cs.CL]

Mitja Nikolaus, Abdellah Fourtassi. Evaluating the Acquisition of Semantic Knowledge from Cross-situational Learning in Artificial Neural Networks. Workshop on Cognitive Modeling and Computational Linguistics, Jun 2021, Online, Mexico. pp.200-210, ⟨10.18653/v1/2021.cmcl-1.24⟩. ⟨hal-03371662⟩