Skip to Main content Skip to Navigation
New interface
Conference papers

Deep Learning Uncertainty in Machine Teaching

Abstract : Machine Learning models can output confident but incorrect predictions. To address this, ML researchers use various techniques to reliably estimate ML uncertainty, usually performed on controlled benchmarks once the model has been trained. We explore how the two types of uncertainty-aleatoric and epistemic-can help non-expert users understand the strengths and weaknesses of a classifier in an interactive setting. We are interested not only in their use of uncertainty to teach and understand the classifier, but also in their perception of the difference between aleatoric and epistemic uncertainty. We conducted an experiment where non-experts train a classifier to recognize card images, and are tested on their ability to predict classifier outcomes. Participants who used either larger or more varied training sets significantly improved their understanding of uncertainty, both epistemic or aleatoric. However, participants who relied on the uncertainty measure to guide their choice of training data did not significantly improve classifier training, nor were they better able to guess the classifier outcome. We identified three specific situations where participants successfully identified the difference between aleatoric and epistemic uncertainty: placing a new card in the exact same position as a training card; placing different cards next to each other; and placing a non-card, such as their hand, next to or on top of a card. We discuss our methodology for estimating uncertainty for Interactive Machine Learning systems and question the need for two-level uncertainty in Machine Teaching.
Complete list of metadata

https://hal.archives-ouvertes.fr/hal-03579448
Contributor : Téo Sanchez Connect in order to contact the contributor
Submitted on : Friday, February 18, 2022 - 10:00:52 AM
Last modification on : Wednesday, November 16, 2022 - 9:37:35 AM
Long-term archiving on: : Thursday, May 19, 2022 - 6:28:36 PM

File

iui_2022_author_version.pdf
Files produced by the author(s)

Identifiers

Citation

Téo Sanchez, Baptiste Caramiaux, Pierre Thiel, Wendy E. Mackay. Deep Learning Uncertainty in Machine Teaching. IUI 2022 - 27th Annual Conference on Intelligent User Interfaces, Mar 2022, Helsinki / Virtual, Finland. ⟨10.1145/3490099.3511117⟩. ⟨hal-03579448⟩

Share

Metrics

Record views

342

Files downloads

149