short-paper

SLAM: scalable locality-aware middleware for I/O in scientific analysis and visualization

Authors:

Junyao ZhangAuthors Info & Claims

HPDC '14: Proceedings of the 23rd international symposium on High-performance parallel and distributed computing

Pages 257 - 260

https://doi.org/10.1145/2600212.2600709

Published: 23 June 2014 Publication History

Get Access

Abstract

Whereas traditional scientific applications are computationally intensive, recent applications require more data-intensive analysis and visualization. As the computational power and size of compute clusters continue to increase, the I/O read rates and associated network cost for these data-intensive applications create a serious performance bottleneck when faced with the massive data sets of today's "big data" era.

In this paper, we present "Scalable Locality-Aware Middleware" (SLAM) for scientific data analysis applications. SLAM leverages a distributed file system (DFS) to provide scalable data access for scientific applications. To reduce data movement and enforce data process locality, a data-centric scheduler (DC-scheduler) is proposed to enable scientific applications to read data locally from a DFS. We prototype our proposed SLAM system along with the Hadoop distributed file system (HDFS) on two well-known scientific applications. We find in our experiments that SLAM can greatly reduce I/O cost and double the overall performance, as compared to existing approaches.

References

[1]

J. Ahrens, B. Geveci, and C. Law. Paraview: An end-user tool for large data visualization. The Visualization Handbook, 717:731, 2005.

Google Scholar

[2]

J. C. Bennett, H. Abbasi, P.-T. Bremer, R. Grout, A. Gyulassy, T. Jin, S. Klasky, H. Kolla, M. Parashar, V. Pascucci, P. Pebay, D. Thompson, H. Yu, F. Zhang, and J. Chen. Combining in-situ and in-transit processing to enable extreme-scale scientific analysis. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, SC '12, pages 49:1--49:9, Los Alamitos, CA, USA, 2012. IEEE Computer Society Press.

Digital Library

Google Scholar

[3]

J. Dean and S. Ghemawat. Mapreduce: simplified data processing on large clusters. Communications of the ACM, 51(1):107--113, 2008.

Digital Library

Google Scholar

[4]

G. Gibson, G. Grider, A. Jacobson, and W. Lloyd. Probe: A thousand-node experimental cluster for computer systems research. volume 38, June 2013.

Google Scholar

[5]

H. Lin, X. Ma, W. Feng, and N. F. Samatova. Coordinating computation and i/o in massively parallel sequence search. IEEE Trans. Parallel Distrib. Syst., 22(4):529--543, Apr. 2011.

Digital Library

Google Scholar

[6]

C. Mitchell, J. Ahrens, and J. Wang. Visio: Enabling interactive visualization of ultra-scale, time series data via high-bandwidth distributed i/o systems. In IPDPS, 2011 IEEE International, pages 68--79, May.

Digital Library

Google Scholar

[7]

S. Sehrish, G. Mackey, J. Wang, and J. Bent. Mrap: A novel mapreduce-based framework to support hpc analytics applications with access patterns. In Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, HPDC '10, pages 107--118, New York, NY, USA, 2010. ACM.

Digital Library

Google Scholar

Cited By

View all

Wang JHan DYin JZhou XJiang C(2020)ODDS: Optimizing Data-Locality Access for Scientific Data AnalysisIEEE Transactions on Cloud Computing10.1109/TCC.2017.27544848:1(220-231)Online publication date: 1-Jan-2020
https://doi.org/10.1109/TCC.2017.2754484
Zhang JAji AChu MWang HFeng WKaeli DPericàs M(2018)Taming irregular applications via advanced dynamic parallelism on GPUsProceedings of the 15th ACM International Conference on Computing Frontiers10.1145/3203217.3203243(146-154)Online publication date: 8-May-2018
https://dl.acm.org/doi/10.1145/3203217.3203243
Huang DHan DWang JYin JChen XZhang XZhou JYe M(2018)Achieving Load Balance for Parallel Data Access on Distributed File SystemsIEEE Transactions on Computers10.1109/TC.2017.274922967:3(388-402)Online publication date: 1-Mar-2018
https://doi.org/10.1109/TC.2017.2749229
Show More Cited By

Index Terms

SLAM: scalable locality-aware middleware for I/O in scientific analysis and visualization
1. Information systems
  1. Data management systems
    1. Data structures
      1. Data access methods
  2. Information storage systems
    1. Record storage systems
      1. Record storage alternatives
2. Software and its engineering
  1. Software organization and properties
    1. Contextual software domains
      1. Operating systems
        File systems management
        Memory management
        Distributed memory
    2. Software system structures
      1. Distributed systems organizing principles

Recommendations

A Performance Analysis of MapReduce Task with Large Number of Files Dataset in Big Data Using Hadoop
CSNT '14: Proceedings of the 2014 Fourth International Conference on Communication Systems and Network Technologies

Big Data is a huge amount of data that cannot be managed by the traditional data management system. Hadoop is a technological answer to Big Data. Hadoop Distributed File System (HDFS) and MapReduce programming model is used for storage and retrieval of ...
Managing Variant Calling Files the Big Data Way: Using HDFS and Apache Parquet
BDCAT '17: Proceedings of the Fourth IEEE/ACM International Conference on Big Data Computing, Applications and Technologies

Big Data has been seen as a remedy for the efficient management of the ever-increasing genomic data. In this paper, we investigate the use of Apache Spark to store and process Variant Calling Files (VCF) on a Hadoop cluster. We demonstrate Tomatula, a ...
Optimization strategy of Hadoop small file storage for big data in healthcare

As the era of "big data" comes, the data processing platform like Hadoop was born at the right moment. But its carrier for storage, Hadoop distributed file system (HDFS) has the great weakness in storage of the numerous small files. The storage of ...

Comments

Information & Contributors

Information

Published In

HPDC '14: Proceedings of the 23rd international symposium on High-performance parallel and distributed computing

June 2014

334 pages

ISBN:9781450327497

DOI:10.1145/2600212

General Chairs:
Beth Plale
Indiana University, USA
,
Matei Ripeanu
University of British Columbia, CA
,
Program Chairs:
Franck Cappello
Argonne National Lab and INRIA, USA
,
Dongyan Xu
Purdue University, USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 June 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

HPDC'14

Sponsor:

SIGARCH

HPDC'14: The 23rd International Symposium on High-Performance Parallel and Distributed Computing

June 23 - 27, 2014

BC, Vancouver, Canada

Acceptance Rates

HPDC '14 Paper Acceptance Rate 21 of 130 submissions, 16%;

Overall Acceptance Rate 166 of 966 submissions, 17%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
210
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 14 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Wang JHan DYin JZhou XJiang C(2020)ODDS: Optimizing Data-Locality Access for Scientific Data AnalysisIEEE Transactions on Cloud Computing10.1109/TCC.2017.27544848:1(220-231)Online publication date: 1-Jan-2020
https://doi.org/10.1109/TCC.2017.2754484
Zhang JAji AChu MWang HFeng WKaeli DPericàs M(2018)Taming irregular applications via advanced dynamic parallelism on GPUsProceedings of the 15th ACM International Conference on Computing Frontiers10.1145/3203217.3203243(146-154)Online publication date: 8-May-2018
https://dl.acm.org/doi/10.1145/3203217.3203243
Huang DHan DWang JYin JChen XZhang XZhou JYe M(2018)Achieving Load Balance for Parallel Data Access on Distributed File SystemsIEEE Transactions on Computers10.1109/TC.2017.274922967:3(388-402)Online publication date: 1-Mar-2018
https://doi.org/10.1109/TC.2017.2749229
Wang JZhang XYin JWang RWu HHan D(2018)Speed Up Big Data Analytics by Unveiling the Storage Distribution of Sub-DatasetsIEEE Transactions on Big Data10.1109/TBDATA.2016.26327444:2(231-244)Online publication date: 1-Jun-2018
https://doi.org/10.1109/TBDATA.2016.2632744
Wang JYin JZhou JZhang XWang R(2016)DataNet: A Data Distribution-Aware Method for Sub-Dataset Analysis on Distributed File Systems2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS)10.1109/IPDPS.2016.33(504-513)Online publication date: May-2016
https://doi.org/10.1109/IPDPS.2016.33
Yin JWang JZhou JLukasiewicz THuang DZhang J(2015)OpassProceedings of the 2015 IEEE International Parallel and Distributed Processing Symposium10.1109/IPDPS.2015.55(623-632)Online publication date: 25-May-2015
https://dl.acm.org/doi/10.1109/IPDPS.2015.55
Yin JWang JBalaji PXu C(2015)Optimize parallel data access in big data processingProceedings of the 15th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing10.1109/CCGrid.2015.168(721-724)Online publication date: 4-May-2015
https://dl.acm.org/doi/10.1109/CCGrid.2015.168

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

A Performance Analysis of MapReduce Task with Large Number of Files Dataset in Big Data Using Hadoop

Managing Variant Calling Files the Big Data Way: Using HDFS and Apache Parquet

Optimization strategy of Hadoop small file storage for big data in healthcare