Research Article

Scripting distributed scientific workflows using Weaver

  1. Peter Bui*,
  2. Li Yu,
  3. Andrew Thrasher,
  4. Rory Carmichael,
  5. Irena Lanc,
  6. Patrick Donnelly,
  7. Douglas Thain

Article first published online: 9 NOV 2011

DOI: 10.1002/cpe.1871

Issue

Cover image for Vol. 24 Issue 9

Concurrency and Computation: Practice and Experience

Early View (Online Version of Record published before inclusion in an issue)

Additional Information

How to Cite

Bui, P., Yu, L., Thrasher, A., Carmichael, R., Lanc, I., Donnelly, P. and Thain, D. (2011), Scripting distributed scientific workflows using Weaver. Concurrency Computat.: Pract. Exper.. doi: 10.1002/cpe.1871

Author Information

  1. Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, IN, USA

*Peter Bui, Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, IN, USA.
E-mail: pbui@cse.nd.edu

Publication History

  1. Article first published online: 9 NOV 2011
  2. Manuscript Accepted: 29 AUG 2011
  3. Manuscript Revised: 24 JUN 2011
  4. Manuscript Received: 8 FEB 2011

SEARCH

SEARCH BY CITATION

ARTICLE TOOLS

View Full Article (HTML) Get PDF (1367K)

Keywords:

  • scripting;
  • workflow;
  • distributed systems;
  • weaver;
  • makeflow;
  • python

SUMMARY

Weaver is a high-level distributed computing framework that enables researchers to construct scalable scientific data-processing workflows. Instead of developing a new workflow language, we introduce a domain-specific language built on top of Python called Weaver, which takes advantage of users' familiarity with the programming language, minimizes barriers to adoption, and allows for integration with a rich ecosystem of existing software. In this paper, we provide an overview of Weaver's programming model, which allows users to organize and specify scientific workflows by using a collection of datasets, functions, and abstractions. We also explain how these workflow specifications are compiled into a directed acyclic graph that is used by the Makeflow workflow manager to dispatch work to a variety of distributed execution platforms. To demonstrate the power and benefits of using the framework in constructing scientific research applications, the paper examines four distinct real-world applications scripted using Weaver and analyzes the performance, scalability, and impact of the distributed generated scientific workflows. Copyright © 2011 John Wiley & Sons, Ltd.

View Full Article (HTML) Get PDF (1367K)