Parametrization of an image understanding quality metric with a subjective evaluation

Baptiste Hemery (1) , Hélène Laurent (2) , Bruno Emile (2) , Christophe Rosenberger (1)

1 Equipe SAFE - Laboratoire GREYC - UMR6072
2 Pôle IRAuS ; Axe Images et Vision

Image understanding has many real industrial applications (video-monitoring, image retrieval . . . ). Given an image and an associated ground truth, it is possible to quantify the quality of understanding results provided by different algorithms or parameters. To this end, it is necessary to take into account many factors for each object in the image: localization and recognition errors and under or over-detection of objects. In order to define an evaluation metric for quantifying the quality of an image understanding result, we have to set, as for example, the weights of each kind of error in the global score. For a correct parameters setting of an evaluation metric we defined previously, we conducted a subjective evaluation of image understanding results involving many experts in image processing. We present in this paper the developed method and analyze the obtained results to weight the various errors in an appropriate way. We show the benefit of this kind of study to define the correct parameters of the metric in order to have a judgment as reliable the one provided by experts. Experimental results on many images from the PASCAL VOC Challenge show the good behavior of this metric compared to the human judgment.

Domaines: Vision par ordinateur et reconnaissance de formes [cs.CV]; Traitement des images [eess.IV]; Traitement du signal et de l'image [eess.SP]; Traitement du signal et de l'image [eess.SP]

Baptiste Hemery, Hélène Laurent, Bruno Emile, Christophe Rosenberger. Parametrization of an image understanding quality metric with a subjective evaluation. Pattern Recognition Letters, 2012, pp.PATREC 5572. ⟨10.1016/j.patrec.2012.11.016⟩. ⟨hal-00771732⟩