TýrFS: Increasing Small Files Access Performance with Dynamic Metadata Replication

Pierre Matri (1, 2) , María Pérez (3, 1) , Alexandru Costan (4, 5) , Gabriel Antoniu (5)

1 UPM - Universidad Politécnica de Madrid
2 OEG - Ontology Engineering Group [Madrid]
3 DATSI - Departamento de Arquitectura y Tecnología de Sistemas Informáticos [Madrid]
4 INSA Rennes - Institut National des Sciences Appliquées - Rennes
5 KerData - Scalable Storage for Clouds and Beyond

Small files are known to pose major performance challenges for file systems. Yet, such workloads are increasingly common in a number of Big Data Analytics workflows or large-scale HPC simulations. These challenges are mainly caused by the common architecture of most state-of-the-art file systems needing one or multiple metadata requests before being able to read from a file. Small input file size causes the overhead of this metadata management to gain relative importance as the size of each file decreases. In this paper we propose a set of techniques leveraging consistent hashing and dynamic metadata replication to significantly reduce this metadata overhead. We implement such techniques inside a new file system named TýrFS, built as a thin layer above the Týr object store. We prove that TýrFS increases small file access performance up to one order of magnitude compared to other state-of-the-art file systems, while only causing a minimal impact on file write throughput.

Domaines: Informatique [cs]; Calcul parallèle, distribué et partagé [cs.DC]

Pierre Matri, María Pérez, Alexandru Costan, Gabriel Antoniu. TýrFS: Increasing Small Files Access Performance with Dynamic Metadata Replication. CCGRID 2018 - 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, May 2018, Washington, United States. pp.452-461, ⟨10.1109/CCGRID.2018.00072⟩. ⟨hal-01892691⟩