Research Article

Global-scale distributed I/O with ParaMEDIC

  1. P. Balaji1,*,
  2. W. Feng2,
  3. H. Lin2,
  4. J. Archuleta2,
  5. S. Matsuoka3,
  6. A. Warren4,
  7. J. Setubal4,
  8. Ewing Lusk1,
  9. R. Thakur1,
  10. I. Foster1,5,
  11. D. S. Katz1,5,6,
  12. S. Jha6,
  13. K. Shinpaugh2,
  14. S. Coghlan1,
  15. D. Reed7

Article first published online: 23 APR 2010

DOI: 10.1002/cpe.1590

Issue

Concurrency and Computation: Practice and Experience

Concurrency and Computation: Practice and Experience

Special Issue: International Supercomputing Conference

Volume 22, Issue 16, pages 2266–2281, November 2010

Additional Information

How to Cite

Balaji, P., Feng, W., Lin, H., Archuleta, J., Matsuoka, S., Warren, A., Setubal, J., Lusk, E., Thakur, R., Foster, I., Katz, D. S., Jha, S., Shinpaugh, K., Coghlan, S. and Reed, D. (2010), Global-scale distributed I/O with ParaMEDIC. Concurrency Computat.: Pract. Exper., 22: 2266–2281. doi: 10.1002/cpe.1590

Author Information

  1. 1

    Mathematics and Computer Science Division, Argonne National Laboratory, 9700 S. Cass Avenue, Argonne, IL 60439, U.S.A.

  2. 2

    Department of Computer Science, Virginia Tech, Blacksburg, VA, U.S.A.

  3. 3

    Department of Mathematical and Computing Science, Tokyo Institute of Technology, Tokyo, Japan

  4. 4

    Virginia Bioinformatics Institute, Virginia Tech, Blacksburg, VA, U.S.A.

  5. 5

    Computation Institute, University of Chicago, Chicago, IL, U.S.A.

  6. 6

    Center for Computation and Technology, Louisiana State University, Baton Rogue, LA, U.S.A.

  7. 7

    Scalable Computing and Multicore Division, Extreme Computing Group, Microsoft Research, Redmond, WA, U.S.A.

*Mathematics and Computer Science Division, Argonne National Laboratory, 9700 S. Cass Avenue, Argonne, IL 60439, U.S.A.

Publication History

  1. Issue published online: 23 APR 2010
  2. Article first published online: 23 APR 2010
  3. Manuscript Accepted: 24 FEB 2010
  4. Manuscript Revised: 21 FEB 2010
  5. Manuscript Received: 11 FEB 2010

SEARCH

SEARCH BY CITATION

ARTICLE TOOLS

View Full Article (HTML) Get PDF (234K)

Keywords:

  • distributed I/O;
  • Bioinformatics;
  • BLAST;
  • grid computing;
  • cluster computing

Abstract

Achieving high performance for distributed I/O on a wide-area network continues to be an elusive holy grail. Despite enhancements in network hardware as well as software stacks, achieving high-performance remains a challenge. In this paper, our worldwide team took a completely new and non-traditional approach to distributed I/O, called ParaMEDIC: Parallel Metadata Environment for Distributed I/O and Computing, by utilizing application-specific transformation of data to orders of magnitude smaller metadata before performing the actual I/O. Specifically, this paper details our experiences in deploying a large-scale system to facilitate the discovery of missing genes and constructing a genome similarity tree by encapsulating the mpiBLAST sequence-search algorithm into ParaMEDIC. The overall project involved nine computational sites spread across the U.S. and generated more than a petabyte of data that was ‘teleported’ to a large-scale facility in Tokyo for storage. Copyright © 2010 John Wiley & Sons, Ltd.

View Full Article (HTML) Get PDF (234K)