Skip to Main content Skip to Navigation
Journal articles

JITA4DS: Disaggregated execution of Data Science Pipelines between the Edge and the Data Centre

Abstract : This paper targets the execution of data science (DS) pipelines supported by data processing, transmission and sharing across several resources executing greedy processes. Current data science pipelines environments provide various infrastructure services with computing resources such as general-purpose processors (GPP), Graphics Processing Units (GPUs), Field Programmable Gate Arrays (FPGAs) and Tensor Processing Unit (TPU) coupled with platform and software services to design, run and maintain DS pipelines. These one-fits-all solutions impose the complete externalization of data pipeline tasks. However, some tasks can be executed in the edge, and the backend can provide just in time resources to ensure ad-hoc and elastic execution environments. This paper introduces an innovative composable "Just in Time Architecture" for configuring DCs for Data Science Pipelines (JITA-4DS) and associated resource management techniques. JITA-4DS is a cross-layer management system that is aware of both the application characteristics and the underlying infrastructures to break the barriers between applications, middleware/operating system, and hardware layers. Vertical integration of these layers is needed for building a customizable Virtual Data Center (VDC) to meet the dynamically changing data science pipelines' requirements such as performance, availability, and energy consumption. Accordingly, the paper shows an experimental simulation devoted to run data science workloads and determine the best strategies for scheduling the allocation of resources implemented by JITA-4DS.
Document type :
Journal articles
Complete list of metadata

https://hal.archives-ouvertes.fr/hal-03315931
Contributor : Genoveva Vargas-Solar Connect in order to contact the contributor
Submitted on : Friday, August 6, 2021 - 10:02:47 AM
Last modification on : Friday, April 1, 2022 - 3:55:30 AM

Links full text

Identifiers

Citation

Genoveva Vargas-Solar, Ali Akoglu, Md Sahil Hassan. JITA4DS: Disaggregated execution of Data Science Pipelines between the Edge and the Data Centre. Journal of Web Engineering, Rinton Press, 2021, ⟨10.13052/jwe1540-9589.2111⟩. ⟨hal-03315931⟩

Share

Metrics

Record views

19