article · 2011
Notes on parallel file systems HDFS & GFS
Randal E. Bryant
Cited by 0Abstract and details only
Abstract
Both GFS & HDFS are designed with similar goals: * High throughput (latency less important)- Designed for batch processing jobs, e.g., MapReduce * High capacity (large block size). Typical files> 1GB * High scalability. Handle> 10^7 files * Reliability through replication. Treat failure as normal. Although GFS was developed first, HDFS is much simpler, and so will describe it first. HDFS Based on "write once, read many " model: * Each file has single writer * File fully written and closed before any reader given access Three components: * Clients. * NameNode. Single node containing all metadata about all files * DataNodes. Set of nodes that store actual file contents. * File represented as sequence of blocks of fixed size (64 or 128 MB). (Given byte offset for read, can immediately determine which block to read.) * Each block has unique block ID. * Blocks are distributed across multiple DataNodes to enable parallel access. * Blocks replicated (default 3X) to enable recovery when DataNode fails. * When block created, NameNode decides placements- Default: two within single rack, third on a different rack- Access time / safety tradeoff. NameNode * Metadata: Information about file + plus set of block IDs + location of all replicas of all blocks * Treats each FS operation as transaction- Maintains all information in memory- Logs to EditLog file- (Possibly outdated) backup copy stored on disk.- Can bring this copy up to date by replaying entries in EditLog * Periodic checkpoint:- Apply transactions in editlog to disk image- Delete old parts of editlog- Can do in background while other updates occuring.- Does not store locations of replicas * Tries to satisfy given read request with nearby DataNode
Cite this paper
Bryant, R. E. (2011). Notes on parallel file systems HDFS & GFS.
Bryant, Randal E. “Notes on parallel file systems HDFS & GFS.” 2011.
Bryant, Randal E. 2011. “Notes on parallel file systems HDFS & GFS.”
Bryant, R.E. (2011) ‘Notes on parallel file systems HDFS & GFS’.
@article{bryant2011notes,
title = {{Notes on parallel file systems HDFS & GFS}},
author = {Bryant, Randal E.},
year = {2011}
}Made from the index record. Check names, capitalisation and page numbers against the paper. Need another style? AI for Research has over 10,000.