Skip to Main content Skip to Navigation
New interface
Conference papers

Streaming Hypergraph Partitioning Algorithms on Limited Memory Environments

Abstract : Many well-known, real-world problems involve dynamic, interrelated data items. Hypergraphs are powerful combinatorial structures that are frequently used to model such data. Many of today's data-centric have streaming data; new items arrive continuously, and the data grow over time. With paradigms such as Internet of Things and Edge Computing, such applications become more natural and more practical. In this work, we assume a streaming model where the data items and their relations are modeled as a hypergraph, which is generated at the edge. This hypergraph is then partitioned, and the parts are sent to remote nodes via an algorithm running on a memory-restricted device, such as a single board computer. Such a partitioning is usually performed by taking a connectivity metric into account to minimize the communication cost of later analyses that will be performed in a distributed fashion. Although there are many offline tools that can partition static hypergraphs effectively, algorithms for the streaming settings are rare. We analyze a well-known algorithm from the literature and significantly improve its run time by altering its inner data structure. On a medium-scale hypergraph, the new algorithm reduces the run time from 17800 seconds to 10 seconds. We then propose sketch-and hash-based algorithms, as well as ones that can leverage extra memory to store a small portion of the data to enable the refinement of partitioning when possible. We experimentally analyze the performance of these algorithms and report their run times, connectivity metric scores, and memory uses on a high-end server and four different single-board computer architectures.
Document type :
Conference papers
Complete list of metadata

https://hal.archives-ouvertes.fr/hal-03182122
Contributor : Bora Uçar Connect in order to contact the contributor
Submitted on : Friday, March 26, 2021 - 10:40:23 AM
Last modification on : Tuesday, October 25, 2022 - 4:23:02 PM
Long-term archiving on: : Sunday, June 27, 2021 - 6:25:54 PM

File

hpcs-streaming.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-03182122, version 1
  • ARXIV : 2103.05394

Collections

Citation

Fatih Taşyaran, Berkay Demireller, Kamer Kaya, Bora Uçar. Streaming Hypergraph Partitioning Algorithms on Limited Memory Environments. HPCS 2020 - International Conference on High Performance Computing & Simulation, Mar 2021, Virtual online, Spain. pp.1-8. ⟨hal-03182122⟩

Share

Metrics

Record views

43

Files downloads

66