Skip to content
Free planNo credit card requiredStart writing
AI for Research home
AI for Research home
Dashboard
Sign inGet started

Search 250 million research papers

Read the abstract, see who wrote it and who cites it, and get the citation in APA, MLA, Chicago or Harvard.

About 10,000 papers for “parallel I/O”

M02---Parallel I/O in practice

Robert Ross, Rajeev Thakur, William Loewe et al. · 2006

Effectively using I/O resources on HPC machines is a black art. The purpose of this tutorial is to shed light on the state-of-the-art in parallel I/O and to provide the knowledge necessary for attendees to best leverage the I/O resources available to them. We cover the entire I/O software stack from parallel file systems at the lowest layer, to intermediate layers (such as MPI-IO), and finally high-level I/O libraries (such as HDF-5). The emphasis is not just on how to use these layers, but ways to use them that result in high performance.This tutorial first discusses parallel file systems (PFSs). We cover general concepts and examine three examples: GPFS, Lustre, and PVFS2. Next we examine the upper layers of the I/O stack, covering four interfaces: POSIX I/O, MPI-IO, Parallel netCDF, and HDF5. We discuss interface features, show code examples, and describe how application calls translate into PFS operations.

Cited by 18

20 Parallel I/O and the Parallel Virtual File System

Walt Ligon, Rob Ross · 2013

Ever more frequently users of clusters find themselves in an interesting situation: it isn’t the processors, communication network, or memory that is limiting their application; it is the storage system. This might force the users to checkpoint less frequently than they would like, might limit the resolution of output visualization data, or might prevent the use of out-of-core solutions needed for the largest of problems. What’s worse, the I/O hardware in the system may indeed be adequate for the user’s needs but may be being used ineffectively by one of the many software layers involved. A lot of mystery surrounds I/O solutions in clusters today. For this reason we have rewritten this chapter in the second edition. We begin by covering what we believe are some of the most important issues in parallel I/O systems. These include parallel access patterns, parallel I/O system components and architectures, and consistency semantics. Knowing how parallel I/O systems operate and the issues involved can be useful when performance tuning an application for a particular system or choosing an I/O solution to match expected workloads. This material

Cited by 0

Parallel simulation of parallel file systems and I/O programs

Rajive Bagrodia, Stephen Docy, Andy Kahn · 1997

Efficient I/O implementations can have a significant impact on the performance of parallel applications. This paper describes the design and implementation of PIOSIM, a parallel simulation library for MPI-IO programs. The simulator can be used to predict the performance of existing MPI-IO programs as a function of architectural characteristics, caching algorithms, and alternative implementations of collective I/O operations. This paper describes the simulator and presents the results of a number of performance studies to evaluate the impact of the preceding factors on a set of MPI-IO benchmarks, including programs from the NAS benchmark suite.

Cited by 24Open access

Massively parallel I/O: Building an infrastructure for parallel computing

David E. Womble, David S. Greenberg · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · 1997

The solution of Grand Challenge Problems will require computations that are too large to fit in the memories of even the largest machines. Inevitably, new designs of I/O systems will be necessary to support them. This report describes the work in investigating I/O subsystems for massively parallel computers. Specifically, the authors investigated out-of-core algorithms for common scientific calculations present several theoretical results. They also describe several approaches to parallel I/O, including partitioned secondary storage and choreographed I/O, and the implications of each to massively parallel computing.

Cited by 0Open access

Iteration Based Collective I/O Strategy for Parallel I/O Systems

Zhixiang Wang, Xuanhua Shi, Hai Jin et al. · 2014

MPI collective I/O is a widely used I/O method that helps data-intensive scientific applications gain better I/O performance. However, it has been observed that existing collective I/O strategies do not perform well due to the access contention problem. Existing collective I/O optimization strategies mainly focus on the I/O phase efficiency and ignore the shuffle cost that may limit the potential of their performance improvement. We observe that as the size of I/O becomes larger, one I/O operation from the upper application would be separated into several iterations to complete. So, I/O requests in each file domain do not necessarily issue to the parallel file system simultaneously unless they are carried out within the same iteration step. Based on that observation, this paper proposes a new collective I/O strategy that reorganizes I/O requests within each file domain instead of coordinating requests across file domains, such that we can eliminate access contentions without introducing extra shuffle cost between aggregators and computing processes. Using benchmark workloads IOR, we evaluate our new strategy and compare with the conventional one. The proposed strategy achieves up to 47%-63% I/O bandwidth improvement compared to the existing ROMIO collective I/O strategy.

Cited by 17

Parallel I/O for High Performance Computing

John May · Medical Entomology and Zoology · 2000

Preface Chapter 1: Introduction Chapter 2: Storage Devices and Interconnects Chapter 3: File Systems Chapter 4: Access Patterns and Optimization Chapter 5: Low-Level I/O Interfaces Chapter 6: Scientific Data Libraries Chapter 7: Special-Purpose I/O Techniques Chapter 8: Data Management and Analysis

Cited by 119

Leveraging parallel I/O in OmpSs-2

Sanchis Ramírez, Aleix · RECERCAT (Consorci de Serveis Universitaris de Catalunya) · 2019

Aquest document mostra el desenvolupament portat a terme per a crear una capa dins del model de programació OmpSs-2 que permeti la interoperabilitat entre SPDK i el runtime Nanos6. Es començarà dissenyant una llibreria anomenada TASPDK (Task-Aware SPDK) que proporcionarà una interfície més còmoda per al programador oferint un accés a blocs de dispositius de forma transparent a SPDK. Després es crearan programes de prova, i s'avaluarà el rendiment mitjançant un External Mergesort, un K-Means clustering i un Convolutional Image Filter.

Cited by 0Open access

Tools for Parallel I/O and Compute

Intensive Applications Thse, Vincent Messerli, Prof David Kotz et al. · 1998

Most parallel file systems provide applications with conventional sequential views of files that stripe data transparently across multiple disks thus reducing the bottleneck of relatively slow disk access throughput. The Unix-like interface increases the ease of application portability, but conceals the underlying parallelism within the file system precluding any optimization of the I/O access pattern from different processes. Although recent multiprocessor file systems incorporate more sophisticated I/O interfaces, e.g. collective I/O interfaces, enabling the I/O access-pattern information to flow from an application program down to the data management system, they still lack flexibility and they do not enable application-specific codes to run on I/O nodes. Therefore application-specific processing operations cannot be directly executed where data resides. Moreover, with the conventional approach of developing parallel I/O- and compute- intensive applications on distributed memory architectures, programmers are faced with two different systems: a parallel file system for parallel I/O and a message passing system for parallel computation. This separation between storage and processing makes it difficult for application programmers to combine I/O and computation in an efficient manner, i.e. overlap I/O requests with computations.

Cited by 0

Understanding Parallel I/O Performance and Tuning

Suren Byna · 2022

Performance of parallel I/O is critical for large-scale scientific applications to store and access data from parallel file systems on high-performance computing (HPC) systems. These applications use HPC systems often to generate and analyze large amounts of data. They use the parallel I/O software stack for accessing and retrieving data. This stack includes several layers of software libraries - high-level I/O libraries such as HDF5, middleware (MPI-IO), and low-level I/O libraries (POSIX, STD-IO). Each of these layers have complex inter-dependencies among them that impact the I/O performance significantly. As a result, scientific applications frequently spend a large fraction of their execution time in reading and writing data on parallel file systems. These inter-dependencies also complicate tuning parallel I/O performance.

Cited by 0

Collective buffering: Improving parallel I/O performance

Bill Nitzberg, Virginia Lo · 2002

"Parallel I/O" is the support of a single parallel application run on many nodes; application data is distributed among the nodes, and is read or written to a single logical file, itself spread across nodes and disks. Parallel I/O is a mapping problem from the data layout in node memory to the file layout on disks. Since the mapping can be quite complicated and involve significant data movement, optimizing the mapping is critical for performance. We discuss our general model of the problem, describe four Collective Buffering algorithms we designed, and report experiments testing their performance on an Intel Paragon and an IBM SP2 both housed at NASA Ames Research Center. Our experiments show improvements of up to two order of magnitude over standard techniques and the potential to deliver peak performance with minimal hardware support.

Cited by 41

Parallel I/O: Benchmarking and common pitfalls

Lührs, Sebastian · JuSER (Forschungszentrum Jülich) · 2018

Beside the computational scalability of an HPC application, its I/O behaviour can significantly influence the overall performance. This behaviour is influenced by many factors, e.g. by the implementation inside the application, the file system, the network and data hardware and software, as well as, since storage is usually a shared resource, other jobs running on the same cluster. Within EoCoE an I/O benchmarking activity was established to investigate how certain I/O patterns can influence the overall performance and how this behaviour can be reflected within the real world HPC applications to optimize the general I/O bandwidth. The talk will present the benchmarking methodology, common parallel I/O pitfalls, and it should guide HPC application developers in the context of parallel I/O problems.

Cited by 0Open access

pGTIOL: A Parallel GeoTIFF I/O Library

HU Shujia · 2015

Data I/O has become one of the main bottlenecks for parallel geospatial computing. In this study, we firstly explore the data structure of a widely used GIS raster data format-Geo TIFF, particularly focusing on its storage modes(strip storage and tile storage). The transfer functions which map the logical structure of data to the physical storage structure were constructed for both storage modes.This article also designs a framework for parallel I/O of raster data and implementsa parallel Geo TIFF I/O library(p GTIOL) using the file-view technique of MPI-IO. Experimental results showed that p GTIOL effectively enhances the I/O performance in comparison with the master-worker I/O mode which uses the Geospatial Data Abstraction Library(GDAL). p GTIOL encapsulates the underlying parallel I/O routines, and provides easy-to-use interfaces for the parallel reading and writing of Geo TIFF data. Compared with other parallel raster I/O software packages, p GTIOL supports a wide range of data types, both the strip and tile data storage modes, and various domain decomposition methods. Most importantly, p GTIOL supports asynchronous parallel I/O, which allows multiple processes to read and write sub-domains of data on demand.Hence,it could facilitate dynamic load-balancing in application.

Cited by 1

Stream*: Fast, Flexible Data-Parallel I/O

Jason Moore, Philip J. Hatcher, Michael J. Quinn · 1994

Although hardware supporting parallel file I/O has improved greatly since the introduction of firstgeneration parallel computers, the programming interface has not. Each vendor provides a different logical view of parallel files as well as nonportable operations for manipulating files. Neither do parallel languages provide standards for performing I/O. In this paper, we describe a view of parallel files for data-parallel languages, dubbed Stream*, in which each virtual processor writes to and reads from its own stream. In this scheme each virtual processor's I/O operations have the same familiar, unambiguous meaning as in a sequential C program. We demonstrate how I/O operations in Stream* can run as fast as those of vendor-specific parallel file systems on the operations most often encountered in data-parallel programs. We show how this system supports general virtual processor operations for debugging and elemental functions. Finally, we present empirical results from a prot...

Cited by 3