Computer Science > Distributed, Parallel, and Cluster Computing
[Submitted on 18 May 2014 (this version), latest version 22 Jul 2014 (v3)]
Title:On The Reproducibility and Scalability of Extreme Scale Applications
View PDFAbstract:For dedicated small scale computing environments, having replicated code and data are sufficient to reproduce prior results if the processing platform is also compatible with both the code and the data. Experiences indicate that these dependencies often quickly deteriorate due to rapid technology advances. The current main-stream explicit parallel programming paradigm not only makes the parallel programs closely coupled with the processing platform at the time of programming but also builds growing instabilities into the parallel programs making reproducibility impossible in larger scales even if the processing platform only changes in size. This impossibility is also linked to our inability to quantify parallel application scalability.
As technology developments will inevitably change the future data processing methods and environments, the explicit parallel paradigms and virtual circuit based APIs should be considered counter-productive to scientific research and big data processing applications, especially in light of reproducibility of large scale cloud computer applications.
This position paper details why and how this long standing problem could be overcome. Preliminary computational results are reported in support of the proposed position.
Submission history
From: Justin Shi [view email][v1] Sun, 18 May 2014 06:29:02 UTC (401 KB)
[v2] Mon, 21 Jul 2014 17:58:52 UTC (478 KB)
[v3] Tue, 22 Jul 2014 03:32:48 UTC (478 KB)
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.