Nešković, Gvozden · Publication Server of Goethe University Frankfurt am Main (Goethe University Frankfurt) · 2018
We live in age of data ubiquity. Even the most conservative estimates predict exponential growth in produced, transmitted and stored data. Big data is used to power business analytics as well as to foster scientific discoveries. In many cases, explosion of produced data exceeds capabilities of digital storage systems. Scientific high-performance computing environments cope with this problem by utilizing large, distributed, storage systems. These complex systems can only provide a high degree of reliability and durability by means of data redundancy. The most straight-forward way of doing that is by replicating the data over different physical devices. However, more elaborate approaches, such as erasure coding, can provide similar data protection while utilizing less storage. Recently, software-defined reliability methods began to replace traditional, hardware- based, solutions. Complicated failure modes of storage system components also warrant checksums to guaranty long-term data integrity. To cope with ever increasing data volumes, flexible and efficient software implementation of error correction codes is of great importance. This thesis introduces a method for realizing a flexible Reed-Solomon erasure code using the âJust-In-Timeâ compilation technique. By exploiting intrinsic arithmetic redundancy in the algorithm, and by relying on modern optimizing compilers, we obtain a throughput-efficient erasure code implementation. Additionally, exploitation of data parallelism is achieved effortlessly by instructing the compiler to produce SIMD code for desired execution platform. We show results of codes implemented using SSE and AVX2 SIMD instruction sets for x86, and NEON instruction set for ARM platforms. Next, we introduce a framework for efficient vectorized RAID-Z redundancy operations of ZFS file system. Traditional, table-based Galois field multiplication algorithms are replaced with custom SSE and AVX2 parallel methods, providing significantly faster and more efficient parity operations. The implementation of this framework was made publicly available as a part of ZFS on Linux project, since version 0.7. Finally, we propose a new erasure scheme for use with existing, high performance, parallel filesystems. Described reliability middleware (ECCFS) allows definition of flexible, file-based, reliability policies, adapting to customized user needs. By utilizing the block erasure code, the ECCFS achieves optimal storage, computation, and network resource utilization, while providing a high level of reliability. The distributed nature of the middleware allows greater scalability and more efficient utilization of storage and network resources, in order to improve availability of the system.
Cited by 0Open access
Brian Atkinson · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · 2022
Abstract not provided.
Cited by 0Open access
Apratim Purakayastha, Carla Schlatter Ellis, David Kotz et al. · 1996
Many parallel scientific applications need high-performance I/O. Unfortunately, end-to-end parallel-I/O performance has not been able to keep up with substantial improvements in parallel-I/O hardware because of poor parallel file-system software. Many radical changes, both at the interface level and the implementation level, have recently been proposed. One such proposed interface is collective I/O, which allows parallel jobs to request transfer of large contiguous objects in a single request, thereby preserving useful semantic information that would otherwise be lost if the transfer were expressed as per-processor non-contiguous requests. Kotz has proposed disk-directed I/O as an efficient implementation technique for collective-I/O operations, where the compute processors make a single collective data-transfer request, and the I/O processors thereafter take full control of the actual data transfer, exploiting their detailed knowledge of the disk-layout to attain substantially improved performance. Recent parallel file-system usage studies show that writes to write-only files are a dominant part of the workload. Therefore, optimizing writes could have a significant impact on overall performance. In this paper, we propose EN-WRICH, a compute-processor write-caching scheme for write-only files in parallel file systems. ENWRICH combines low-overhead write caching at the compute processors with high performance disk-directed I/O at the I/O processors to achieve both low latency and high bandwidth. This combination facilitates the use of the powerful disk-directed I/O
Cited by 0
Juan Piernas, Jarek Nieplocha · Parallel Computing · 2009
Cited by 6
Ricardo Ryoiti Sugawara, Líria Matsumoto Sato · 2013
Many recent data intensive parallel systems builds with cost effective hardware and combine compute and storage facilities. Since bandwidth-bisecting networks are the norm, distributing jobs near data provides significant performance improvements. However, the data locality information is not easily available to the programmer. It requires interaction with file system internals, or the adoption of a custom programming and run-time frameworks that provide locality-aware job scheduling, such as Mapreduce and Hadoop. In this paper, we present a parallel file system implementation combined with virtual files presented as text that can be queried for locality data or written to control the placement of new data blocks. This simplifies how software that do not adhere to Mapreduce's model can benefit from computing near the data. To evaluate the proposed approach, a number of tests were run on an initial implementation using fast disks, with locality-aware cases showing from 2 to 9 times faster reads and higher processor utilization.
Cited by 1
Jianwei Liao, Guoqiang Xiao, Xiaoning Peng · The Scientific World JOURNAL · 2014
This paper presents a novel metadata management mechanism on the metadata server (MDS) for parallel and distributed file systems. In this technique, the client file system backs up the sent metadata requests, which have been handled by the metadata server, so that the MDS does not need to log metadata changes to nonvolatile storage for achieving highly available metadata service, as well as better performance improvement in metadata processing. As the client file system backs up certain sent metadata requests in its memory, the overhead for handling these backup requests is much smaller than that brought by the metadata server, while it adopts logging or journaling to yield highly available metadata service. The experimental results show that this newly proposed mechanism can significantly improve the speed of metadata processing and render a better I/O data throughput, in contrast to conventional metadata management schemes, that is, logging or journaling on MDS. Besides, a complete metadata recovery can be achieved by replaying the backup logs cached by all involved clients, when the metadata server has crashed or gone into nonoperational state exceptionally.
Cited by 2Open access
Xubin He, Li Ou, Christian Engelmann et al. · Journal of Parallel and Distributed Computing · 2009
Cited by 13
Ziqi Fan, Fenggang Wu, Jim Diehl et al. · 2017
For modern HPC systems, failures are treated as the norm instead of exceptions. To avoid rerunning applications from scratch, checkpoint/restart techniques are employed to periodically checkpoint intermediate data to parallel file systems. To increase HPC checkpointing speed, distributed burst buffers (DBB) have been proposed to use node-local NVRAM to absorb the bursty checkpoint data. However, without proper coordination, DBB is prone to suffer from low resource utilization. To solve this problem, we propose an NVRAM-based burst buffer coordination system, named collaborative distributed burst buffer (CDBB). CDBB coordinates all the available burst buffers, based on their priorities and states, to help overburdened burst buffers and maximize resource utilization. We built a proof-of-concept prototype and tested CDBB at the Minnesota Supercomputing Institute. Compared with a traditional DBB system, CDBB can speed up checkpointing by up to 8.4x under medium and heavy workloads and only introduces negligible overhead.
Cited by 1
Chen Chen, Liu Jianzhong, Yanliang Zou et al. · 2019
Improving I/O performance of large-scale computing systems has become increasingly dependent on the efficiency of parallel file systems. A common way to deploy parallel file systems is using the user space file system framework (e.g. FUSE), which introduces an extra I/O interposition layer that may cause considerable overhead due to excessive utilization of kernel crossings and system function calls. These slowdowns have been exacerbated due to the growing isolation of user and kernel space, increasing kernel crossing overhead significantly. In this paper, we present our findings on the evaluation of how FUSE affects the efficiency of a popular parallel file system, the Parallel Log-structured File System (PLFS). We then suggest a means to mitigate this issue by removing FUSE from the equation and demonstrate its viability with a proof of concept library.
Cited by 1
Hwa Jung Kim, Jiwoo Bang, Dong Kyu Sung et al. · 2021
Parallel file systems (PFS) are used to distribute data processing and establish shared access to large-scale data. Despite being able to provide high I/O bandwidth on each node, PFS has difficulty utilizing the I/O bandwidth due to a single connection between the client and server nodes. To mitigate the performance bottleneck, users increase the number of connections between the nodes by modifying PFS or applications. However, it is difficult to modify PFS itself due to its complicated internal structure. Thus, PFS users manually increase the number of connections between the nodes by employing several methods. In this paper, we propose a user-transparent I/O subsystem, MulConn, to make users exploit high I/O bandwidth between nodes. To avoid the modifications of PFS and user applications, we have developed a horizontal mount procedure and two I/O scheduling policies, TtoS and TtoM, in the virtual file system (VFS) layer. We expose a single mount point that has multiple connections by modifying the mount path of VFS from vertical hierarchy to horizontal hierarchy. We also introduce two I/O scheduling policies to distribute I/O requests evenly to multiple connections. The experimental results show that MulConn improves write and read performance by up to 2.6x and 2.8x, respectively, compared with those of PFS using the existing kernel. In addition, we provide the best I/O performance that PFS can provide in the given experimental environments.
Cited by 0
Frank Shorter · TigerPrints (Clemson University) · 2003
Cited by 18
Chen Wang, Kathryn Mohror, Marc Snir · IEEE Transactions on Parallel and Distributed Systems · 2024
The semantics of HPC storage systems are defined by the consistency models to which they abide. Storage consistency models have been less studied than their counterparts in memory systems, with the exception of the POSIX standard and its strict consistency model. The use of POSIX consistency imposes a performance penalty that becomes more significant as the scale of parallel file systems increases and the access time to storage devices, such as node-local solid storage devices, decreases. While some efforts have been made to adopt relaxed storage consistency models, these models are often defined informally and ambiguously as by-products of a particular implementation. In this work, we establish a connection between memory consistency models and storage consistency models and revisit the key design choices of storage consistency models from a high-level perspective. Further, we propose a formal and unified framework for defining storage consistency models and a layered implementation that can be used to easily evaluate their relative performance for different I/O workloads. Finally, we conduct a comprehensive performance comparison of two relaxed consistency models on a range of commonly seen parallel I/O workloads, such as checkpoint/restart of scientific applications and random reads of deep learning applications. We demonstrate that for certain I/O scenarios, a weaker consistency model can significantly improve the I/O performance. For instance, in small random reads that are typically found in deep learning applications, session consistency achieved a 5x improvement in I/O bandwidth compared to commit consistency, even at small scales.
Cited by 7Open access
Bharti Wadhwa, Arnab K. Paul, Sarah Neuwirth et al. · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · 2019
Parallel I/O performance is crucial to sustaining scientific applications on large-scale High-Performance Computing (HPC) systems. However, I/O load imbalance in the underlying distributed and shared storage systems can significantly reduce overall application performance. There are two conflicting challenges to mitigate this load imbalance: (i) optimizing systemwide data placement to maximize the bandwidth advantages of distributed storage servers, i.e., allocating I/O resources efficiently across applications and job runs; and (ii) optimizing client-centric data movement to minimize I/O load request latency between clients and servers, i.e., allocating I/O resources efficiently in service to a single application and job run. Moreover, existing approaches that require application changes limit wide-spread adoption in commercial or proprietary deployments. We propose iez, an “end-to-end control plane” where clients transparently and adaptively write to a set of selected I/O servers to achieve balanced data placement. Our control plane leverages realtime load information for distributed storage server global data placement while our design model leverages trace-based optimization techniques to minimize I/O load request latency between clients and servers. We evaluate our proposed system on an experimental cluster for two common use cases: synthetic I/O benchmark IOR for large sequential writes and a scientific application I/O kernel, HACC-I/O. Results show read and write performance improvements of up to 34% and 32%, respectively, compared to the state of the art.
Cited by 13Open access
2012
We present a method for parallelizing serial applications strictly dependent on sequential file output. We give a scheme for maintaining serial-append for parallel programs. Subsequently, we describe the implementation of serial-append for the Cilk multithreaded language. Empirically, we show that a single file is a bottleneck in multithreaded computations that use files. We propose using multiple files to represent the same logical file and gain performance. Our results show that the application of concurrent skip lists, as a concurrent order maintenance data structure to support serial-append, scales well with
Cited by 0
Md Hasanur Rashid, Nathan R. Tallent, Forrest Sheng Bao et al. · arXiv (Cornell University) · 2026
Tuning parallel file system in High-Performance Computing (HPC) systems remains challenging due to the complex I/O paths, diverse I/O patterns, and dynamic system conditions. While existing autotuning frameworks have shown promising results in tuning PFS parameters based on applications' I/O patterns, they lack scalability, adaptivity, and the ability to operate online. In this work, focusing on scalable online tuning, we present CARAT, an ML-guided framework to co-tune client-side RPC and caching parameters of PFS, leveraging only locally observable metrics. Unlike global or pattern-dependent approaches, CARAT enables each client to make independent and intelligent tuning decisions online, responding to real-time changes in both application I/O behaviors and system states. We then prototyped CARAT using Lustre and evaluated it extensively across dynamic I/O patterns, real-world HPC workloads, and multi-client deployments. The results demonstrated that CARAT can achieve up to 3x performance improvement over the default or static configurations, validating the effectiveness and generality of our approach. Due to its scalability and lightweight, we believe CARAT has the potential to be widely deployed into existing PFS and benefit various data-intensive applications.
Cited by 0Open access
Qiong Li, Shuaizhe Lv, Xuchao Xie et al. · Lecture notes in computer science · 2023
Cited by 0
Neda Tavakoli, Dong Dai, Yong Chen · arXiv (Cornell University) · 2018
Object-based parallel file systems have emerged as promising storage solutions for high-performance computing (HPC) systems. Despite the fact that object storage provides a flexible interface, scheduling highly concurrent I/O requests that access a large number of objects still remains as a challenging problem, especially in the case when stragglers (storage servers that are significantly slower than others) exist in the system. An efficient I/O scheduler needs to avoid possible stragglers to achieve low latency and high throughput. In this paper, we introduce a log-assisted straggler-aware I/O scheduling to mitigate the impact of storage server stragglers. The contribution of this study is threefold. First, we introduce a client-side, log-assisted, straggler-aware I/O scheduler architecture to tackle the storage straggler issue in HPC systems. Second, we present three scheduling algorithms that can make efficient decision for scheduling I/Os while avoiding stragglers based on such an architecture. Third, we evaluate the proposed I/O scheduler using simulations, and the simulation results have confirmed the promise of the newly introduced straggler-aware I/O scheduler.
Cited by 0Open access
V. Prathyusha1 | Aaruni Giriraj2 | Dr. Ramakrishna3 · Bulletin of Miscellaneous Information (Royal Gardens Kew)
In this work we studied the key establishment for secure many-to-many communications. The main problem is inspired by the rapid increase of large-scale distributed file systems supporting parallel access to multiple storage devices. The system focus on the current Internet standard for such file systems, i.e., parallel Network File System (pNFS), which makes use of Kerberos key exchange protocols to implement parallel session keys between clients and storage servers. Our study of the existing Kerberos protocol shows that it has a number of limitations: (i) a metadata server providing key exchange among the clients and the storage devices has heavy workload that limits the scalability of the protocol; (ii) the protocol cannot provide forward secrecy; (iii) the metadata server generates all the session keys for securing communication between clients and storage devices, and this inadvertently leads to key escrow. In this paper, we put forward three different authenticated key exchange protocols that are designed to address the above issues. We prove that our protocols are capable for minimizing up to almost50% of the workload of the metadata server and at the same time supporting forward secrecy and escrow-prevention. All this requires only a small fraction of increased computation overhead at the client.
Cited by 0
Md Hasanur Rashid, Nathan R. Tallent, Forrest Sheng Bao et al. · arXiv (Cornell University) · 2026
Tuning parallel file system in High-Performance Computing (HPC) systems remains challenging due to the complex I/O paths, diverse I/O patterns, and dynamic system conditions. While existing autotuning frameworks have shown promising results in tuning PFS parameters based on applications' I/O patterns, they lack scalability, adaptivity, and the ability to operate online. In this work, focusing on scalable online tuning, we present CARAT, an ML-guided framework to co-tune client-side RPC and caching parameters of PFS, leveraging only locally observable metrics. Unlike global or pattern-dependent approaches, CARAT enables each client to make independent and intelligent tuning decisions online, responding to real-time changes in both application I/O behaviors and system states. We then prototyped CARAT using Lustre and evaluated it extensively across dynamic I/O patterns, real-world HPC workloads, and multi-client deployments. The results demonstrated that CARAT can achieve up to 3x performance improvement over the default or static configurations, validating the effectiveness and generality of our approach. Due to its scalability and lightweight, we believe CARAT has the potential to be widely deployed into existing PFS and benefit various data-intensive applications.
Cited by 0Open access
Ting Wu, Linbo Long, Zhulin Ma et al. · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · 2025
Internet and IoT applications have generated increasing amounts of data that require efficient storage. Many persistent memory file systems have been designed to handle high-performance storage demands by fully exploiting the senior features of persistent memory (PM). However, PMs suffer from limited write endurance. Existing PM file systems achieve PM wear-leveling by designing wear-leveling-aware allocators. These allocators focus on providing higher-balanced writes to PMs while neglecting the overhead. Moreover, they cause serious request conflicts in parallel block requests by multiple threads in modern multiprocessor computer systems. In this paper, we propose an efficient wear-leveling-aware parallel allocator, WPAlloc, for persistent memory file systems to achieve wear-leveling of PM and high parallel performance. The essential idea of WPAlloc is to allocate blocks with lower write counters for each allocation request and to provide parallel block allocation and deallocation for multiple threads via one free list per logical processor. WPAlloc consists of two key techniques: the bucket sort-based range management scheme (BRMS) and the parallel allocation and deallocation scheme (PADS). First, we design the BRMS to obtain the less worn free blocks. Based on BRMS, an online wear range adjustment algorithm has been designed to adjust the wear range dynamically. Then, we present the PADS to avoid request conflicts by multiple threads. We implement WPAlloc based on PMFS. Experimental results show that WPAlloc can reduce the maximum write by 54.2%, 10.2%, and 55.7%, while achieving average performance improvements of 5.68%, 54.18%, and 11.28% compared to PMFS, DWARM, and WASA, respectively.
Cited by 0