Prototyping a Web-Scale Multimedia Retrieval Service Using Spark

Abstract : The world has experienced phenomenal growth in data production and storage in recent years, much of which has taken the form of media files. At the same time, computing power has become abundant with multi-core machines, grids, and clouds. Yet it remains a challenge to harness the available power and move toward gracefully searching and retrieving from web-scale media collections. Several researchers have experimented with using automatically distributed computing frameworks, notably Hadoop and Spark, for processing multimedia material, but mostly using small collections on small computing clusters. In this article, we describe a prototype of a (near) web-scale throughput-oriented MM retrieval service using the Spark framework running on the AWS cloud service. We present retrieval results using up to 43 billion SIFT feature vectors from the public YFCC 100M collection, making this the largest high-dimensional feature vector collection reported in the literature. We also present a publicly available demonstration retrieval system, running on our own servers, where the implementation of the Spark pipelines can be observed in practice using standard image benchmarks, and downloaded for research purposes. Finally, we describe a method to evaluate retrieval quality of the ever-growing high-dimensional index of the prototype, without actually indexing a web-scale media collection.
Type de document :
Article dans une revue
ACM Transactions on Multimedia Computing, Communications and Applications, Association for Computing Machinery, 2018, 14 (3s), pp.1 - 24. 〈10.1145/3209662〉
Liste complète des métadonnées

https://hal.archives-ouvertes.fr/hal-01853379
Contributeur : Laurent Amsaleg <>
Soumis le : vendredi 3 août 2018 - 11:04:03
Dernière modification le : mercredi 17 octobre 2018 - 16:02:45

Lien texte intégral

Identifiants

Citation

Laurent Amsaleg, Gylfi Thór Gudmundsson, Björn Þór Jónsson, Michael Franklin. Prototyping a Web-Scale Multimedia Retrieval Service Using Spark. ACM Transactions on Multimedia Computing, Communications and Applications, Association for Computing Machinery, 2018, 14 (3s), pp.1 - 24. 〈10.1145/3209662〉. 〈hal-01853379〉

Partager

Métriques

Consultations de la notice

108