Skip to content
Free planNo credit card requiredStart writing
AI for Research home
AI for Research home
Dashboard
Sign inGet started

Search 250 million research papers

Read the abstract, see who wrote it and who cites it, and get the citation in APA, MLA, Chicago or Harvard.

About 2,281 papers for “parallel file systems”

Authenticated Key Exchange Protocols for Parallel Network File Systems

Hoon Wei Lim, Guomin Yang · IEEE Transactions on Parallel and Distributed Systems · 2015

We study the problem of key establishment for secure many-to-many communications. The problem is inspired by the proliferation of large-scale distributed file systems supporting parallel access to multiple storage devices. Our work focuses on the current Internet standard for such file systems, i.e., parallel Network File System (pNFS), which makes use of Kerberos to establish parallel session keys between clients and storage devices. Our review of the existing Kerberos-based protocol shows that it has a number of limitations: (i) a metadata server facilitating key exchange between the clients and the storage devices has heavy workload that restricts the scalability of the protocol; (ii) the protocol does not provide forward secrecy; (iii) the metadata server generates itself all the session keys that are used between the clients and storage devices, and this inherently leads to key escrow. In this paper, we propose a variety of authenticated key exchange protocols that are designed to address the above issues. We show that our protocols are capable of reducing up to approximately 54 percent of the workload of the metadata server and concurrently supporting forward secrecy and escrow-freeness. All this requires only a small fraction of increased computation overhead at the client.

Cited by 7

Cooperative Caching And Prefetching In Parallel/Distributed File Systems

Cortés, Toni · LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) · 1997

If we examine the structure of the applications that run on parallel machines, we observe that their I/O needs increase tremendously every day. These applications work with very large data sets which, in most cases, do not fit in memory and have to be kept in the disk. The input and output data files are also very large and have to be accessed very fast. These large applications also want to be able to checkpoint themselves without wasting too much time. These facts constantly increase the expectations placed on parallel and distributed file systems. Thus, these file systems have to improve their performance to avoid becoming the bottleneck in parallel/distributed environments. On the other hand, while the performance of the new processors, interconnection networks and memory increases very rapidly, no such thing happens with the disk performance. This lack of improvement is due to the mechanical parts used to build the disks. These components are slow and limit both the latency and t...

Cited by 0Open access

Metadata-intensive I/O optimizations in parallel file systems

Limin Xiao, Ke Xie, Xiuqiao Li et al. · WIT transactions on engineering sciences · 2013

Welcome to the WIT Press eLibrary - the home of the Transactions of the Wessex Institute collection, providing on-line access to papers presented at the Institute's prestigious international conferences and from its State-of-the-Art in Science & Engineering publications.

Cited by 0

1 Motivation Scaling Security for Big, Parallel File Systems

Andrew Leung Ethan · 2014

The need for peta- and exabyte scale parallel file systems that support high-performance computing (HPC) has been rapidly increasing. These systems have unique demands, different from those of traditional distributed file systems. As a result, securing I/O in big, parallel file systems without significantly impacting performance has proven challenging. Parallel file systems are commonly composed of three components: clients, metadata servers, and network-attached storage devices, such as disks or object-storage devices. Security is commonly based on a capability model, where capabilities are ’tokens ’ which represent single block or object I/O authorization. Capabilities are generated by metadata servers, given to clients, and presented to storage devices with I/O requests. Peta- and exabyte scale systems may have tens of thousands of clients and storage devices. Files are very large, often gigabytes to terabytes, and are commonly striped across thousands of storage devices. HPC workloads are parallel and bursty, meaning I/O often comes from many clients at a time with very short inter-arrival times. This implies files are commonly accessed by thousands of clients within a few seconds. These factors are often worst-case scenarios for many existing solutions. While there are many security schemes for distributed file systems, few have addressed large-scale, demanding environments. In these environments existing solutions cannot sustain high performance or must weaken security to do so. HPC workloads can burden metadata and data servers with generating and verifying millions of single block or object capabilities. Reliance on shared-key cryptography introduces vulnerabilities when millions of storage devices are susceptible to attack. Also, many common security techniques, such as revocation, become difficult with so many clients and storage devices.

Cited by 0

Visual Cryptography Based Authentication For Parallel Network File Systems

R. Keerthivasan, K. S. S. Iyer, Vyshnav A.K et al. · International Journal of Computer Sciences and Engineering · 2017

International Journal of Computer Sciences and Engineering (A UGC Approved and indexed with DOI, ICI and Approved, DPI Digital Library) is one of the leading and growing open access, peer-reviewed, monthly, and scientific research journal for scientists, engineers, research scholars, and academicians, which gains a foothold in Asia and opens to the world, aims to publish original, theoretical and practical advances in Computer Science,Information Technology, Engineering (Software, Mechanical, Civil, Electronics & Electrical), and all interdisciplinary streams of Computing Sciences. It intends to disseminate original, scientific, theoretical or applied research in the field of Computer Sciences and allied fields. It provides a platform for publishing results and research with a strong empirical component. It aims to bridge the significant gap between research and practice by promoting the publication of original, novel, industry-relevant research.

Cited by 1Open access

Bridging The Gap Between Parallel File Systems And Local File Systems: A Case Study With Pvfs

Peng, Gu, Jun, Wang, Ross, Robert · Journal of International Crisis and Risk Communication Research · 2008

Parallel I/O plays an increasingly important wie in today's data intensive computing applications. While much attention has been paid to parallel read performance, most of this work has focused on the parallel file system, middleware, or application layers, ignoring the potential for improvement through more effective use of local storage. In this paper, we present the design and implementation of Segment-structured On-disk data Grouping and Prefetching (SOGP), a technique that leverages additional local storage to boost the local data read performance for parallel file systems, especially for those applications with partially overlapped access patterns. Parallel Virtual File System (PVFS) is chosen as an example. Our experiments show that an SOGP-enhancedPVFS prototype system can outperform a traditional Linux-Ext-based PVFS for many applications and benchmarks, in some tests by as much as 230% in terms of I/O bandwidth. © 2008 IEEE.

Cited by 0Open access

Parallel File Systems: alternatives to reduce the bottleneck in accessing the file system

Carvalho Junior, Roberto Ferreira de, 1968- · Digital Library of Theses and Dissertations of the University of São Paulo (Universidade de São Paulo) · 2005

Nos últimos anos, a evolução dos processadores e redes para computadores de baixo custo foi muito maior se comparada com o aumento do desempenho dos discos de armazenamento de dados. Com isso, muitas aplicações estão encontrando dificuldades em atingir o pleno uso dos processadores, pois estes têm de esperar até que os dados cheguem para serem utilizados. Uma forma popular para resolver esse tipo de empecílio é a adoção de sistemas de arquivos paralelos, que utilizam a velocidade da rede local, além dos recursos de cada máquina, para suprir a deficiência de desempenho no uso isolado de cada disco. Neste estudo, analisamos alguns sistemas de arquivos paralelos e distribuídos, detalhando aqueles mais interessantes e importantes. Por fim, mostramos que o uso de um sistema de arquivos paralelo pode ser mais eficiente e vantajoso que o uso de um sistema de arquivos usual, para apenas um cliente.

Cited by 0

A Parallel File Transfer Protocol for Clusters and Grid Systems

D. Bhardwaj, R. Satheesh Kumar · 2006

Parallel file transfer protocol (PFTP) introduces the concept of data transfer using multiple parallel data paths between clusters. It makes use of striped data distributed across the multiple IO-nodes of a parallel file system on clusters. The protocol uses the striping format of the parallel file system to push multiple stripes of the file over multiple TCP connections. The flexibility in PFTP allows user to use different file transfer applications over these TCP connections. In this paper we describe the design architecture and implementation of PFTP. Several experiments explain how it improves the data transfer rate. We also provide performance results of PFTP for a varying file data size of 100 MB to 4GB. The flexible, secure, fault tolerant and high performance design of PFTP can be useful for large data grids.

Cited by 10

Automatic I/O scheduling algorithm selection for parallel file systems

Francieli Zanon Boito, Rodrigo Virote Kassick, Philippe O. A. Navaux et al. · Concurrency and Computation Practice and Experience · 2015

Summary This article presents our approach to provide input/output (I/O) scheduling with double adaptivity: to applications and devices. In high‐performance computing environments, parallel file systems provide a shared storage infrastructure to applications. In the situation where multiple applications access this shared infrastructure concurrently, their performance can be impaired because of interference. Our work focuses on I/O scheduling as a tool to improve performance by alleviating interference effects. The role of the I/O scheduler is to decide the order in which applications' requests must be processed by the parallel file system's servers, applying optimizations to adjust the resulting access pattern for improved performance. Our approach to improve I/O scheduling results is based on using information from applications' access patterns and storage devices' sensitivity to access sequentiality. We have applied machine learning to provide the ability to automatically select the best scheduling algorithm for each situation. Our approach improves performance by up to 75 % over an approach that uses the same scheduling algorithm to all situations, without adaptability. Our results evidence that both aspects – applications and storage devices – are essential to make good scheduling decisions. Copyright © 2015 John Wiley & Sons, Ltd.

Cited by 22Open access

Malleability and fault tolerance in ad-hoc parallel file systems

Dario Muñoz-Muñoz, Felix Garcia-Carballeira, Diego Camarmas-Alonso et al. · Cluster Computing · 2025

Abstract In the last few years, there has been a significant rise in I/O demands across different applications, especially those related to Big Data and Artificial Intelligence. This has resulted in the requirement for improvements to I/O capabilities in order to prevent potential data access bottlenecks. The Expand Ad-Hoc parallel file system is being designed and developed to address this. Given the possibility of variations in I/O workloads throughout an application’s execution, it is essential that the file system employed possesses the requisite malleability to adapt accordingly to avoid wasting resources. Additionally, as these applications have long execution times, fault tolerance mechanisms within the file system are essential to ensure their continued operation in the presence of failures. The primary innovation of this work lies in the design of the support for malleability, taking into account the data replication of the fault tolerance. This approach results in a system that exhibits both expansion and reduction capability, made possible by the integration of malleability and fault tolerance. Therefore, this work presents a design for the Expand Ad-Hoc parallel file system and evaluates it on the HPC4AI Laboratory supercomputer in Turin. We have evaluated this design using benchmarks and real applications by comparing the result with the native file system used on the HPC4AI supercomputer. We have evaluated the fault tolerance support using benchmarks and real applications by comparing the result with the native file system used on the HPC4AI supercomputer. For the real application used, improvements of up to 100% have been obtained regarding the backend file system. For malleability, the time required to execute a malleability operation has been analyzed. In this case, the results show that it is possible to perform this malleability operation with very small overhead times. The results show that the malleability scales effectively with the resources, and despite the data replication in the fault tolerance, we outperform other parallel file systems without fault tolerance.

Cited by 0Open access

Comparison of leading parallel NAS file systems on commodity hardware

Richard Hedges, K Fitzgerald, Mark Gary et al. · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · 2010

High performance computing has experienced tremendous gains in system performance over the past 20 years. Unfortunately other system capabilities, such as file I/O, have not grown commensurately. In this activity, we present the results of our tests of two leading file systems (GPFS and Lustre) on the same physical hardware. This hardware is the standard commodity storage solution in use at LLNL and, while much smaller in size, is intended to enable us to learn about differences between the two systems in terms of performance, ease of use and resilience. This work represents the first hardware consistent study of the two leading file systems that the authors are aware of.

Cited by 7Open access

Direct-pnfs: scalable, transparent, and versatile access to parallel file systems

Dean Hildebrand · Deep Blue (University of Michigan) · 2007

Grid computations require global access to massive data stores. To meet this need, the GridNFS project aims to provide scalable, high-performance, transparent, and secure wide-area data management as well as a scalable and agile name space. While parallel file systems give high I/O throughput, they are highly specialized, have limited operating system and hardware platform support, and often lack strong security mechanisms. Remote data access tools such as NFS and GridFTP overcome some of these limitations, but fail to provide universal, transparent, and scalable remote data access. As part of GridNFS, this paper introduces Direct-pNFS, which builds on the NFSv4.1 protocol to meet a key challenge in accessing remote parallel file systems: high-performance and scalable data access without sacrificing transparency, security, or portability. Experiments with Direct-pNFS demonstrate I/O throughput that equals or outperforms the exported parallel file system across a range of workloads.

Cited by 0Open access

pWalrus: Towards better integration of parallel file systems into cloud storage

Yoshihisa Abe, Garth A. Gibson · 2010

Amazon S3-style storage is an attractive option for clouds that provides data access over HTTP/HTTPS. At the same time, parallel file systems are an essential component in privately owned clusters that enable highly scalable dataintensive computing. In this work, we take advantage of both of those storage options, and propose pWalrus, a storage service layer that integrates parallel file systems effectively into cloud storage. Essentially, it exposes the mapping between S3 objects and backing files stored in an underlying parallel file system, and allows users to selectively use the S3 interface and direct access to the files. We describe the architecture of pWalrus, and present preliminary results showing its potential to exploit the performance and scalability of parallel file systems.

Cited by 16