Tradeoffs in Subjective Testing Methods for Image and Video Quality
Résumé
The merit of an objective quality estimator for either still images or video is gauged by its ability to accurately estimate the perceived quality scores of a collection of stimuli. Encounters with radically different distortion types that arise in novel media representations require that researchers collect perceived quality scores representative of these new distortions to confidently evaluate a candidate objective quality estimator. Two common methods used to collect perceived quality scores are absolute categorical rating (ACR)1 and subjective assessment for video quality (SAMVIQ).2,3 The choice of a particular test method affects the accuracy and reliability of the data collected. An awareness of the potential benefits and/or costs attributed to the ACR and SAMVIQ test methods can guide researchers to choose the more suitable method for a particular application. This paper investigates the tradeoffs of these two subjective testing methods using three different subjective databases that have scores corresponding to each method. The subjective databases contain either still-images or video sequences. This paper has the following organization: Section 2 summarizes the two test methods compared in this paper, ACR and SAMVIQ. Section 3 summarizes the content of the three subjective databases used to evaluate the two test methods. An analysis of the ACR and SAMVIQ test methods is presented in Section 4. Section 5 concludes this paper.