Skip to content
Free planNo credit card requiredStart writing
AI for Research home
AI for Research home
Dashboard
Sign inGet started

Search 250 million research papers

Read the abstract, see who wrote it and who cites it, and get the citation in APA, MLA, Chicago or Harvard.

About 10,000 papers for “parallel I/O”

Improving Parallel I/O Performance Using Interval I/O

Logan, Jeremy · DigitalCommons (California Polytechnic State University) · 2010

Today's most advanced scientific applications run on large clusters consisting of hundreds of thousands of processing cores, access state of the art parallel file systems that allow files to be distributed across hundreds of storage targets, and utilize advanced interconnections systems that allow for theoretical I/O bandwidth of hundreds of gigabytes per second. Despite these advanced technologies, these applications often fail to obtain a reasonable proportion of available I/O bandwidth. The reasons for the poor performance of application I/O include the noncontiguous I/O access patterns used for scientific computing, contention due to false sharing, and the somewhat finicky nature of parallel file system performance. We argue that a more fundamental cause of this problem is the legacy view of a file as a linear sequence of bytes. To address these issues, we introduce a novel approach for parallel I/O called Interval I/O. Interval I/O is an innovative approach that uses application access patterns to partition a file into a series of intervals, which are used as the fundamental unit for subsequent I/O operations. Use of this approach provides superior performance for the noncontiguous access patterns which are frequently used by scientific applications. In addition, the approach reduces false contention and the unnecessary serialization it causes. Interval I/O also significantly increases the performance of atomic mode operations. Finally, the Interval I/O approach includes a technique for supporting parallel I/O for cooperating applications. We provide a prototype implementation of our Interval I/O system and use it to demonstrate performance improvements of as much as 1000% compared to ROMIO when using Interval I/O with several common benchmarks.

Cited by 0Open access

Parallel I/O Systems and Interfaces for Parallel Computers

Dror G. Feitelson, Peter F. Corbett, Yarsun Hsu et al. · 1995

Introduction Continued improvements in processor performance have exposed I/O subsystems as a significant bottleneck, which prevents applications from achieving full system utilization [33, 54]. This problem is exacerbated in massively parallel processors (MPPs), where multiple processors are used together. As a result, I/O subsystems have become the focus of much research, leading to the design of parallel I/O hardware and matching system software. The requirement driving the work on I/O subsystems is the desire to achieve a balanced system [8]. The degree to which a system is balanced is typically expressed by the F=b ratio, which is defined as the ratio of the rate of executing floating point operations (F ) to the rate of performing I/O, in bits per second (b). A widely accepted rule of thumb, attributed to Amdahl, calls for F=b 1. While this was originally expressed in instructions rather than floating po

Cited by 0

Sun MPI I/O: Efficient I/O for Parallel Applications

Len Wisniewski, Brad Smisloff, Nils Nieuwejaar et al. · 2007

Many parallel applications require high-performance I/O to avoid negating some or all of the benefit derived from parallelizing its computation. When these applications are run on a loosely-coupled cluster of SMPs, the limitations of existing hardware and software present even more hurdles to performing high-performance I/O. In this paper, we describe our full implementation of the I/O portion of the MPI-2 specification. In particular, we discuss the limitations inherent in performing high-performance I/O on a cluster of SMPs and demonstrate the benefits of using a cluster-based filesystem over a traditional node-based filesystem. 1 Introduction The MPI I/O interface enables efficient use of an underlying parallel file system by allowing parallel applications to describe complex I/O requests at a high level.[1, 2] For example, parallel applications in weather/climate modeling, seismic processing, computational fluid dynamics, and data mining can use such an interface to bett...

Cited by 0

ZioLib: A parallel I/O library

Woo-Sun Yang, Chris Ding · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · 2003

In a distributed memory parallel environment, many applications rely on a serial I/O strategy, where the global array is gathered on a single MPI process and then written out to a file. I/O performance with this approach is largely limited by single process I/O bandwidth. Even when parallel I/O is used, satisfactory parallel scaling is not always observed. It is because in many applications fields are not necessarily in a most favorable parallel decomposition for I/O. The best I/O rates are obtained when a field is decomposed with respect to the array's last dimension (referred to here as Z). Another situation often encountered in many applications is that a field in CPU resident memory is in one index order but must be stored in a disk file in another order. Changing index orders can complicate a parallel I/O implementation and slow down I/O. ZioLib facilitates an efficient parallel I/O for arrays in such situations. In case of a write, ZioLib remaps a distributed field into a Z-decomposition on a subset of processes (which will be called the I/O staging processes) and from there writes to a disk file in parallel. In this Z-decomposition, the data layout of the remapped array on the staging processes memory is the same as on disk, thus only block data transfer occurs during parallel I/O, achieving maximum efficiency. In case of a read the steps are reversed to build the required distributed arrays on the computational processes.

Cited by 2Open access

Evaluating Parallel I/O Energy Efficiency

Rong Ge · 2010

Energy efficiency and parallel I/O performance are two primary constraints in high performance computing (HPC). Large scale computing systems consume vast amounts of energy and require scalable I/O performance to meet the needs of a broad spectrum of data intensive scientific applications. However, little research has provided an in-depth understanding of energy efficiency in parallel I/O subsystems. In this paper, we experimentally investigate the efficiency impact of a wide range of variables including the application's access patterns, parallel file system deployment, and processor frequency scaling. Our studies lead to several key findings on parallel I/O energy efficiency. First, the application's I/O access pattern significantly affects system energy efficiency. By aggregating file accesses and choosing appropriate data sizes, we can both improve performance and reduce energy use. Second, parallel file systems, such as PVFS, not only increase the I/O scalability but also are more energy efficient. Matching the application's I/O buffer size to the file system strip size will lead to much higher efficiency than poorly chosen I/O buffer sizes. Third, DVFS can further reduce energy use of parallel I/O operations with little performance impact. These findings are useful to guide the development of energy efficient HPC systems and software.

Cited by 7

A Transparent Parallel I/O Environment

Darren Erik Vengroff · 1994

We describe TPIE, a Transparent Parallel I/O Environment. TPIE is a system designed to bridge the gap between current theoretical knowledge about the construction of I/O-optimal algorithms on parallel disk systems and the design and implementation of parallel I/O systems. We discuss the design of TPIE and its interface, the structure of a typical implementation, applications of the system, our prototype, and future research directions. The initial goal of our work is a prototype system to demonstrate: 1) that optimal algorithms can be made to run efficiently on parallel I/O devices; and 2) that high level hardware independent interfaces to the I/O paradigms required to implement such algorithms can be provided to application programmers. The TPIE interface is designed to be portable across a variety of parallel hardware platforms; thus code that runs efficiently on one machine will run efficiently on others. Longer term goals for TPIE include extending the prototype in ways t...

Cited by 0

A brief parallel I/O tutorial.

H. Ward · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · 2010

This document provides common best practices for the efficient utilization of parallel file systems for analysts and application developers. A multi-program, parallel supercomputer is able to provide effective compute power by aggregating a host of lower-power processors using a network. The idea, in general, is that one either constructs the application to distribute parts to the different nodes and processors available and then collects the result (a parallel application), or one launches a large number of small jobs, each doing similar work on different subsets (a campaign). The I/O system on these machines is usually implemented as a tightly-coupled, parallel application itself. It is providing the concept of a 'file' to the host applications. The 'file' is an addressable store of bytes and that address space is global in nature. In essence, it is providing a global address space. Beyond the simple reality that the I/O system is normally composed of a small, less capable, collection of hardware, that concept of a global address space will cause problems if not very carefully utilized. How much of a problem and the ways in which those problems manifest will be different, but that it is problem prone has been well established. Worse, the file system is a shared resource on the machine - a system service. What an application does when it uses the file system impacts all users. It is not the case that some portion of the available resource is reserved. Instead, the I/O system responds to requests by scheduling and queuing based on instantaneous demand. Using the system well contributes to the overall throughput on the machine. From a solely self-centered perspective, using it well reduces the time that the application or campaign is subject to impact by others. The developer's goal should be to accomplish I/O in a way that minimizes interaction with the I/O system, maximizes the amount of data moved per call, and provides the I/O system the most information about the I/O transfer per request.

Cited by 2Open access

Parallel I/O Applications and Environments

2009

Due to the growth of Internet-driven applications, issues such as storage capacity and access speed have become critical in the design of today's computer systems Book fills the need for a readily-accessible single reference source on the subject of high-performance, large scale storage and delivery systems Contains the latest information and future directions of disk arrays and parallel I/O A Wiley-IEEE Press Publication

Cited by 0

Parallel I/O aware query optimization

Pedram Ghodsnia, Ivan T. Bowman, Anisoara Nica · 2014

New trends in storage industry suggest that in the near future a majority of the hard disk drive-based storage subsystems will be replaced by solid state drives (SSDs). Database management systems can substantially benefit from the superior I/O performance of SSDs. Although the impact of using SSD in query processing has been studied in the past, exploiting the I/O parallelism of SSDs in query processing and optimization has not received enough attention. In this paper, at first, we show why the query optimizer needs to be aware of the benefit of the I/O parallelism in solid state drives. We characterize the benefit of exploiting I/O parallelism in database scan operators in SAP SQL Anywhere and propose a novel general I/O cost model that considers the impact of device I/O queue depth in I/O cost estimation. We show that using this model, the best plans found by the optimizer would be much closer to optimal. The proposed model is implemented in SAP SQL Anywhere. This model, dynamically defined by a calibration process, summarizes the behavior of the I/O subsystem, without having any prior knowledge about the type and the number of devices which are used in the storage subsystem.

Cited by 15

User-Level Parallel File I/O

Ernst L. Leiss, Yili Wang · CLEI electronic journal · 1998


 
 
 
 Parallel disk I/O subsystems are becoming more important in today’s large-scale parallel machines. Parallel disk systems provide a significant boost in I/O performance reducing the gap between processor and disk speeds. We describe a Unix-like file I/O user interface, implemented in a parallel file I/O subsystem on an MIMD machine, the nCUBE 2. Based on message passing, we develop parallel disk read/write algorithms to achieve higher parallelism when more disk drives are used. We use the closed queuing network model to analyze the effect of some tunable system parameters for our parallel file system. We then performed simulation experiments in order to obtain more realistic performance data, by comparing the original vendor-supplied file system with ours. The results indicate that the speedup in I/O performance is almost equal to the number of disk drives we use. Thus, our user-level parallel file I/O approach will provide scalable I/O performance.
 
 
 

Cited by 1Open access

HPF with Parallel I/O Extensions

Rajesh Bordawekar, Alok Choudhary · 2007

This paper proposes some directives for parallel I/O that can be used in conjunction with other HPF directives. These directives have proven to be useful in the initial implementation of runtime and compiler support for parallel I/O in our HPF compiler [9]. 2 Parallel I/O Directives

Cited by 0

File concepts for parallel I/O

T. W. Crockett · 1989

The subject of I/O has often been neglected in the design of parallel computer systems, although for many problems I/O rates will limit the speedup attainable. The I/O problem is addressed here by considering the role of files in parallel systems. The notion of parallel files is introduced. Parallel files provide for concurrent access by multiple processes, and utilize parallelism in the I/O system to improve performance. Parallel files can also be used conventionally by sequential programs. A set of standard parallel file organizations is proposed, based on common data partitioning techniques. Implementation strategies for the proposed organizations are suggested, using multiple storage devices. Problem areas are also identified and discussed.

Cited by 61Open access

Usability of Parallel I/O Templates

Ian Parsons, Ron Unrau, Jonathan Schaeffer et al. · 2007

This paper presents an alternative high-level approach to parallelizing file I/O. Each parallel file descriptor is annotated with a high-level specification, or template, of the expected parallel behaviour. The annotations are external to and independent of the source code. At run-time, all I/O using a template file descriptor adheres to the semantics of the selected template. By separating the parallel I/O behaviour from the code, a user can quickly change I/O semantics without rewriting code. Templates can be combined hierarchically to allow the simple construction of complex access patterns. Two sample parallel programs using these templates are compared against versions implemented in an existing parallel I/O system (PIOUS). The sample programs show that the use of parallel I/O templates are beneficial from both the performance and software engineering points of view. 1 Introduction The development of parallel applications has focused on computational parallelism. However, the co...

Cited by 0

Parallel I/o At Hpc Facilities

2014

Modern high performance computing facilities deliver computational and data resources to an ever broadening set of scientific and engineering domains. What was once a cottage industry just two decades ago, HPC is now integral to studies ranging from biology to materials science and everything in between. These domains are using simulation at ever increasing scales driven by increasingly higher resolution models with more accurate representation of physical processes. This confluence has resulted in the build-out of extreme-scale HPC facilities with compute platforms composed of hundreds of thousands of compute cores, high performance networking infrastructures, and parallel I/O environments capable of scaling to terabytes/second of I/O bandwidth while providing tens of petabytes of capacity.

Cited by 0