Higher-order Occurrence Pooling for Bags-of-Words: Visual Concept Detection

Piotr Koniusz (1) , Fei Yan (2) , Philippe-Henri Gosselin (3) , Krystian Mikolajczyk (4)

1 ANU - Australian Centre for Robotic Vision, Australian National University, Canberra
2 Key Laboratory of Functional Gene and Regulation Technologies under Chongqing Municipal Education Commission (Chongqing, China)
3 ETIS - UMR 8051 - Equipes Traitement de l'Information et Systèmes
4 CVSSP - Centre for Vision, Speech and Signal Processing

In object recognition, the Bag-of-Words model assumes: i) extraction of local descriptors from images, ii) embedding the descriptors by a coder to a given visual vocabulary space which results in mid-level features, iii) extracting statistics from mid-level features with a pooling operator that aggregates occurrences of visual words in images into signatures, which we refer to as First-order Occurrence Pooling. This paper investigates higher-order pooling that aggregates over co-occurrences of visual words. We derive Bag-of-Words with Higher-order Occurrence Pooling based on linearisation of Minor Polynomial Kernel, and extend this model to work with various pooling operators. This approach is then effectively used for fusion of various descriptor types. Moreover, we introduce Higher-order Occurrence Pooling performed directly on local image descriptors as well as a novel pooling operator that reduces the correlation in the image signatures. Finally, First-, Second-, and Third-order Occurrence Pooling are evaluated given various coders and pooling operators on several widely used benchmarks. The proposed methods are compared to other approaches such as Fisher Vector Encoding and demonstrate improved results.

Keywords: Bag-of-Words; Mid-level features; First-order; Second-order; Co-occurrence; Pooling Operator; Sparse Coding

Domains: Machine Learning [stat.ML]; Computer Vision and Pattern Recognition [cs.CV]

Piotr Koniusz, Fei Yan, Philippe-Henri Gosselin, Krystian Mikolajczyk. Higher-order Occurrence Pooling for Bags-of-Words: Visual Concept Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, ⟨10.1109/TPAMI.2016.2545667⟩. ⟨hal-01356149⟩