A Belief-Based Sequential Fusion Approach for Fusing Manual and Non-Manual Signs

Oya Aran (1) , Thomas Burger (2) , Alice Caplier (3) , Lale Akarun (1)

1 Department of Computer Engineering [Bogazici]
2 FT R&D - France Télécom Recherche & Développement
3 GIPSA-GPIG - GIPSA - Géométrie, Perception, Images, Geste

Most of the research on sign language recognition concentrates on recognizing only manual signs (hand gestures and shapes), discarding a very important component: the non-manual signs (facial expressions and head/shoulder motion). We address the recognition of signs with both manual and nonmanual components using a sequential belief-based fusion technique. The manual components, which carry information of primary importance, are utilized in the first stage. The second stage, which makes use of nonmanual components, is only employed if there is hesitation in the decision of the first stage. We employ belief formalism both to model the hesitation and to determine the sign clusters within which the discrimination takes place in the second stage. We have implemented this technique in a sign tutor application. Our results on the eNTERFACE'06 ASL database show an improvment over the baseline system which uses parallel or feature fusion of manual and non-manual features via HMM recognizers and achieves an accuracy of 81.6%.

Domaines: Vision par ordinateur et reconnaissance de formes [cs.CV]; Interface homme-machine [cs.HC]

Oya Aran, Thomas Burger, Alice Caplier, Lale Akarun. A Belief-Based Sequential Fusion Approach for Fusing Manual and Non-Manual Signs. Pattern Recognition, 2009, 42 (5), pp.812-822. ⟨10.1016/j.patcog.2008.09.010⟩. ⟨hal-00327779⟩