HAL will be down for maintenance from Friday, June 10 at 4pm through Monday, June 13 at 9am. More information
Skip to Main content Skip to Navigation
Journal articles

Prototyping a Web-Scale Multimedia Retrieval Service Using Spark

Abstract : The world has experienced phenomenal growth in data production and storage in recent years, much of which has taken the form of media files. At the same time, computing power has become abundant with multi-core machines, grids, and clouds. Yet it remains a challenge to harness the available power and move toward gracefully searching and retrieving from web-scale media collections. Several researchers have experimented with using automatically distributed computing frameworks, notably Hadoop and Spark, for processing multimedia material, but mostly using small collections on small computing clusters. In this article, we describe a prototype of a (near) web-scale throughput-oriented MM retrieval service using the Spark framework running on the AWS cloud service. We present retrieval results using up to 43 billion SIFT feature vectors from the public YFCC 100M collection, making this the largest high-dimensional feature vector collection reported in the literature. We also present a publicly available demonstration retrieval system, running on our own servers, where the implementation of the Spark pipelines can be observed in practice using standard image benchmarks, and downloaded for research purposes. Finally, we describe a method to evaluate retrieval quality of the ever-growing high-dimensional index of the prototype, without actually indexing a web-scale media collection.
Document type :
Journal articles
Complete list of metadata

Contributor : Laurent Amsaleg Connect in order to contact the contributor
Submitted on : Friday, August 3, 2018 - 11:04:03 AM
Last modification on : Friday, April 8, 2022 - 4:08:03 PM

Links full text



Laurent Amsaleg, Gylfi Thór Gudmundsson, Björn Þór Jónsson, Michael Franklin. Prototyping a Web-Scale Multimedia Retrieval Service Using Spark. ACM Transactions on Multimedia Computing, Communications and Applications, Association for Computing Machinery, 2018, 14 (3s), pp.1 - 24. ⟨10.1145/3209662⟩. ⟨hal-01853379⟩



Record views