Electron microscopy
  
PythonML
HDFS (Hadoop Distributed File System)
- Python Automation and Machine Learning for ICs -
- An Online Book: Python Automation and Machine Learning for ICs by Yougui Liao -
Python Automation and Machine Learning for ICs                                                           http://www.globalsino.com/ICs/        


Chapter/Index: Introduction | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z | Appendix

=================================================================================

HDFS (Hadoop Distributed File System) is a fundamental component of Apache Hadoop and is designed to store very large data sets reliably, and to stream those data sets at high bandwidth to user applications. Here are the key features and characteristics of HDFS:

  • Distributed Storage: HDFS stores data across multiple machines to ensure reliability and fault tolerance. Even if one or more machines fail, data is preserved.

  • Scalability: It can scale up from a single server to thousands of machines, each offering local computation and storage. 

  • Large Data Sets: HDFS is optimized to handle large data sets, typically in the range of gigabytes to terabytes in size.
  • High Throughput: It provides high data throughput, which is beneficial for applications that require data-intensive operations.
  • Data Replication: HDFS automatically replicates data blocks across different nodes in a Hadoop cluster to ensure data availability and durability. The default replication factor is three, but this can be configured.
  • Write Once, Read Many Times: The file system follows a write-once-read-many access model. Once written, the files in HDFS are not typically updated, but they can be read multiple times. This is ideal for batch processing tasks.
  • Block Structured: HDFS stores data as a sequence of blocks, typically of a fixed size (e.g., 128 MB by default). This block structure is suitable for distributed processing.
  • HDFS achieves fault tolerance by replicating data blocks across multiple DataNodes. This replication ensures that data is preserved even if one or more nodes fail.
  • For HDFS, if one machine crashes, the data needs to be rebuilt again.
  • In HDFS, the term "nodes" refers to the individual computers that are part of the entire Hadoop cluster where the data is stored and processed. Each node plays a specific role in the file system's architecture, primarily categorized into two types:
    • NameNode: The NameNode is the master node that manages the file system namespace and regulates access to files by clients. It maintains and manages the file system tree and the metadata for all the files and directories. This metadata is stored in memory, which makes the NameNode a critical component of HDFS. The NameNode also tracks where across the cluster the file data is kept, which it does by maintaining a map of file blocks to DataNodes.
    • DataNodes: DataNodes are the worker nodes that manage the storage of data on the machines on which they run. Each file in HDFS is split into blocks, and these blocks are stored in a set of DataNodes. The DataNodes take care of the read and write requests from the file system’s clients. They also take instructions from the NameNode on creating, deleting, and replicating data blocks.

Hadoop Distributed File System (HDFS) is used for storing large amounts of data in a distributed manner, effectively making it a good choice for data lakes which require storing vast quantities of raw, unstructured data.

===========================================

         
         
         
         
         
         
         
         
         
         
         
         
         
         
         
         
         
         

 

 

 

 

 



















































 

 

 

 

 
=================================================================================